Abstract
Electromyography (EMG) is essential for accurate assessment of motor function in rehabilitation, sports science, and robotics. However, its various time-consuming human operations (e.g., electromagnetic noise countermeasures) limit its widespread use. Meanwhile, motion capture technology has become more accessible, leading to increasing interest in musculoskeletal simulation models such as OpenSim. Although advances have been made in individualizing the model parameters, accurately estimating muscle activity remains a significant challenge. Previous efforts to optimize the parameters in musculoskeletal model simulators have yielded limited improvements in estimation accuracy. A key source of error that is identified in this study is the spatio-temporal distortion between the estimated and actual muscle activity, which has been inadequately addressed in previous research. To address this problem, this study proposes the Neural-Enhanced Motion-to-EMG (NEM2E) framework, which mitigates spatio-temporal distortions in simulated muscle activity using the Spatio-Temporal Distortion Refinement Network (STDR-Net). The STDR-Net is implemented via a Sequence-to-Sequence model with attention mechanisms to refine the estimates. Validation on two public datasets (walking and running motions) confirms significant accuracy improvements: enhanced estimations for all five muscles in the running dataset and for two of five muscles in the walking dataset. These findings demonstrate the potential of the NEM2E framework to refine OpenSim-generated muscle activity estimates and advance personalized applications in muscle activity analysis.
1 Introduction
Electromyography (EMG) is essential for accurate assessment of motor function in rehabilitation, sports science, and robotics. However, it is not suitable for routine measurement because it requires various time-consuming human operations, such as the application of electrodes by specialists and electromagnetic noise countermeasures, for correct measurement. In addition, it is affected by sweat and changes in skin conditions, making it unsuitable for long-term measurement. In contrast, markerless motion capture systems such as THEIA three-dimensional (3D), Azure Kinect, and mediapipe allow for easy motion measurement. Mobile force plates that can measure ground reaction forces simply by putting on shoes have also been investigated (; ) and commercialized by Tec Gihan Co., Ltd. (M3D Force Plate). In addition, VMOCAP (), which estimates motion and muscle activity using only an RGB camera, has been proposed. Owing to these technological advances, muscle activity estimation using a musculoskeletal model simulator with measurements of the motions and ground reaction forces has become a useful system for measuring muscle activity in the elderly and in athletes who wish to reduce their measurement burden.
In this context, musculoskeletal model simulators (; ; ; ) such as OpenSim (; ) have attracted increasing attention for muscle activity estimation. A key challenge in musculoskeletal simulation is the accurate modeling of individual human dynamics. Optimizing model parameters such as the bone length, muscle force characteristics, and joint stiffness is crucial for achieving more realistic analyses ().
Among these simulators, OpenSim is one of the most widely used open-source platforms for musculoskeletal simulation (; ; ). OpenSim facilitates the dissemination of research within the SimTK community and is used extensively in academic and clinical research (; ; ; ; ; ; ; ). It supports integration with skeletal models (; ; ) and optimization algorithms (; ) as add-on components. Recent advancements, such as Myosuite () combined with Multi-Joint dynamics with Contact (MuJoCo) () have further expanded the capabilities of musculoskeletal simulations.
OpenSim operates through two key layers: torque estimation and muscle activity estimation (see Supplementary Appendix SA for details on the muscle activation estimation procedure). The torque estimation layer accurately predicts joint torques by customizing musculoskeletal models based on user-specific physical characteristics and ground reaction forces (; ; ; ). Conversely, the muscle activity estimation layer solves an inverse problem to derive muscle activations from joint torques. However, this inverse problem lacks a unique solution owing to the redundancy of muscle actuators in the OpenSim musculoskeletal model. OpenSim addresses this challenge by minimizing the sum of muscle activity to achieve local optimal solutions (); however, this often leads to significant discrepancies between the estimated and measured muscle activities.
Several sources of error in the estimation of muscle activity estimation using OpenSim have been identified. Modeling errors, particularly in muscle dynamics, are one major cause. For example, prior research () observed a consistent time delay of approximately 75Â ms between the estimated and measured muscle activities in running datasets. In addition, other studies (; ; ) reported shifts in the peak of muscle activity. Another critical source of error arises from the non-unique solutions to the inverse problem of determining muscle activity from joint torque owing to the redundancy of muscle actuators, which are employed to reproduce the human musculoskeletal system. OpenSim uses Computed Muscle Control (CMC) () to solve this problem; however, its objective function, which minimizes activation while achieving the target torques, does not guarantee alignment with actual muscle activity patterns. To the best of our knowledge, no previous study has generically addressed the motion-to-EMG problem using a neural network, as musculoskeletal simulators do.
Based on these findings, this study hypothesizes that errors in muscle activity estimation stem from two primary factors: temporal mismatches and spatial redundancy in musculoskeletal models. As the results of previous studies have shown, the resolution of these errors is limited only by the conventional optimization of the model parameters. Therefore, as opposed to optimizing the musculoskeletal parameters, this study explores an alternative approach by introducing a compensation model to refine the OpenSim outputs.
The Neural-Enhanced Motion-to-EMG (NEM2E) framework (Figure 1) is introduced to address the above problems. It estimates realistic muscle activity from motion and ground reaction force data by incorporating the Spatio-Temporal Distortion Refinement Network (STDR-Net). Specifically, the STDR-Net refines the muscle activity outputs of OpenSim by compensating for spatio-temporal distortion. The network leverages a Sequence-to-Sequence (Seq2Seq) model (), which is a recent development in natural language and image processing, enhanced with attention mechanisms (). The framework implements Seq2Seq models with spatial and temporal attention mechanisms to investigate the spatial and temporal error contributions separately.
FIGURE 1
The analysis was performed using the walk and running motion of elderly people and athletes, as well as extractable discrete motions such as baseball batting and golf swinging, which are commonly applied in sports and rehabilitation science. This study used two publicly available datasets (; ) that pair the muscle activity estimations of OpenSim with actual EMG data for lower-limb walking and running motions. In addition, the accuracy was tested on unknown subjects through cross-validation between subjects based on open data to enable cross-sectional measurement.
The contributions of this study are summarized as follows.
• The NEM2E framework is developed to enhance the realism of musculoskeletal model simulators.
• The study hypothesizes that discrepancies between simulated muscle activity and measured EMG data are owing to spatio-temporal distortions that are inherent in conventional musculoskeletal models.
• The STDR-Net is proposed using Seq2Seq and attention mechanisms to refine the muscle activity estimations.
• Validation is performed using public datasets, demonstrating significant accuracy improvements for all five muscles in running data and two out of five muscles in walking data.
By addressing a significant error source in musculoskeletal modeling, this study opens pathways for further innovations in refining biomechanical simulations and integrating neural networks into computational modeling.
The remainder of this paper is organized as follows: Section 2 reviews related studies, Section 3 describes the NEM2E framework, Section 4 explains the model learning and statistical analysis, Section 5 presents the validation results, Section 6 discusses the findings, and Section 7 concludes the paper.
2 Related works
One of the key roles of OpenSim is to estimate muscle activity. demonstrated that integrating joint angular acceleration feedback into the muscle activity optimization routine (static optimization) of OpenSim yields muscle activity estimates with timing similar to EMG-based measurements. This method is implemented in OpenSim as CMC.
However, estimating muscle activity from joint torque remains an ill-posed problem owing to the redundancy of actuators in musculoskeletal models. Several studies have reported discrepancies between the simulated and experimental muscle activity, which have often been attributed to changes in muscle loading and the effects of assisted movement (; ).
Efforts to improve the accuracy of OpenSim musculoskeletal models have followed two main approaches:
2.1 Development of sophisticated models
The first approach involves creating more detailed models. For instance, studies have developed precise models of the lumbar spine, addressing areas that are not adequately represented by the base model of OpenSim (; ). Tools such as NMSBuilder developed by G. Valente et al. have also been introduced to aid researchers in developing customized OpenSim models (). However, there are inherent limitations to the accuracy that is achievable in simulations owing to challenges in replicating the complexity of the human body. In addition, measurements such as CT scans are required to capture individual differences accurately, which may not be feasible for all users.
2.2 Optimization of model parameters
The second approach focuses on optimizing the model parameters. OpenSim moco () introduces functions to calibrate parameters such as the muscle length and maximum tension using individual user data. Although this improves the personalization, discrepancies between the estimated muscle activity and measured EMG data persist. Valente et al. proposed robust optimization methods to address uncertainties in muscle models (). However, parameter optimization alone cannot resolve fundamental modeling errors in musculoskeletal models.
2.3 Our concept
In contrast to previous studies that focused only on optimizing the parameters or models within the OpenSim framework, this study introduces the NEM2E framework. By addressing temporal and spatial modeling errors separately, the STDR-Net compensates for these errors using a Seq2Seq model with an attention mechanism. This approach has the potential to mitigate the limitations of conventional methods and improve the accuracy of muscle activity estimation.
3 Materials and methods
This section presents an overview of the methods and experimental setup.
3.1 Methods
NEM2E framework and STDR-Net.
3.1.1 NEM2E framework
The NEM2E framework refines the muscle activity estimations generated by OpenSim (Figure 1). It starts with the standard workflow of OpenSim, which includes 1) scaling (skeletal model parameter adjustment), 2) inverse kinematics (joint angle calculation), 3) residual reduction algorithm (joint torque estimation), and 4) CMC (muscle activity estimation) (see Supplementary Appendix SA for a detailed description). The resulting muscle activity data are then refined by STDR-Net to compensate for spatio-temporal distortions that are introduced by the OpenSim musculoskeletal model.
3.1.2 STDR-Net
STDR-Net addresses spatio-temporal distortions in OpenSim musculoskeletal models. Several time-series modeling techniques (e.g., linear regression, recurrent neural networks, long short-term memory (LSTM), and the Transformer) can be applied; however, this study implements STDR-Net using a Seq2Seq model with an attention mechanism () because this approach leverages attention layers to analyze the spatio-temporal relationships during data refinement. A schematic is shown in Figure 2. Two distinct models, (a) and (b), are constructed to analyze the temporal and spatial distortion, respectively.
FIGURE 2
These models use the muscle activity estimated by OpenSim as the input and the measured real muscle activity as the output , and train a neural network (encoder , decoder , and attention ) that compensates for the spatio-temporal distortions between the input and output. By analyzing the attention mechanism that is embedded in the network, the information that is used to correct the distortion can be revealed, and the contribution of spatio-temporal information to refining the estimated muscle activity can be verified. See Supplementary Appendix SB for details on the Seq2Seq with attention model.
3.2 Experiment
The proposed NEM2E framework was validated using two publicly available datasets (): published in https://simtk.org/projects/mspeedwalksims and the datasets used in the literature () published in https://simtk.org/projects/nmbl_running. The contents of the dataset are shown in Table 1 These datasets contain the estimated muscle activation results for which individual body parameter tuning and other standard tuning has already been performed in the respected studies.
TABLE 1
| Dataset | Motion | Subject no. | Model | Muscle dim |
|---|---|---|---|---|
| Walking () | Four different speeds | 8 | gait2392 | 92 |
| Running () | Four different speeds | 10 | Original () | 102 |
Datasets.
3.2.1 Model learning
The parameters of the STDR-Net were trained using a dataset, where the simulated muscle activation from OpenSim served as the input and the muscle activation obtained from the measured EMG served as the output. The squared error was used as the loss function and the optimization process was carried out using the Adam algorithm ().
For all conditions, the STDR-Net was configured with 20 units corresponding to the number of muscle actuators in OpenSim for the encoder (input layer), 700 units for each intermediate layer, and 20 units for the decoder (output layer). Training was performed with a batch size of 1 and 700 epochs. The data were standardized by thinning the input and output data to 20 samples each. For the temporal attention models, the input dimension corresponded to the single dimension of the target muscle.
3.2.2 Statistical analysis
The performance of the proposed model was compared with the baseline muscle activity estimations of OpenSim. All EMG measurements in each dataset (running data: 16 muscles, walking data: 5 muscles) were verified. The names and abbreviations of each muscle are as follows: the soleus (soleus), biceps femoris long head (bi), semimembranosus (sem), tibialis anterior (tib), rectus femoris (rect), gluteus maximus (glmax1, glmax2, and glmax3), gluteus medius (glmed1, glmed2, and glmed3), gastrocnemius medial head (gasmed), gastrocnemius lateral head (gaslat), vastus lateralis (vaslat), and vastus medialis (vasmed) muscles. These names follow the column labels used in publicly available OpenSim muscle activity datasets and may differ from standard anatomical abbreviations. All data were segmented into individual gait cycles. A leave-one-out cross-validation approach was used, with data blocked by subject for both the walking and running datasets.
The accuracy of the model was assessed using the root mean squared error (RMSE) between the model output and measured EMG data , which is calculated using the following (Equation 1):where is the number of samples. Assuming that the error in the refined muscle activity obtained by the proposed method is and the error in the estimated muscle activity obtained from OpenSim is , the error improvement rate is calculated as
Because each subject has data for four different speeds, the average of the four RMSE values is treated as the representative value for each subject. The null hypothesis was tested using a T-test. The effect size is calculated using Cohen’s d in the following (Equation 3).where, and denote average and standard deviation of , respectively. is hypothesized population mean.
The 95% confidence interval (CI) is calculated using the following (
Equation 4).
where,
represents the two-tailed t-value of
in the t-distribution with n-1 degrees of freedom. n is sample number.
is 0.05. Three input data patterns were defined for validation.
• One muscle, quarter cycle (One-Quarter): Sequence data for a quarter cycle of one muscle of the target muscle.
• One muscle, entire-cycle (One-Entire): Sequence data for an entire cycle of one muscle of the target muscle.
• All muscles, entire cycle (All-Entire): Sequence data for an entire cycle of all muscle sets in the musculoskeletal model.
One-Entire tested the temporal distortion compensation. Because the proposed framework emphasizes the importance of utilizing the entire sequence data to address temporal distortion, the effect of using only a portion of the sequence as input was examined in the One-Quarter model. All-Entire assessed the spatial distortion compensation by incorporating all muscle activity data. Temporal attention (Figure 2a) was used for One-Entire and One-Quarter, whereas spatial attention (Figure 2b) was applied to All-Entire. Tukey’s multiple comparison tests were performed for each muscle to analyze significant differences among the three input patterns.
4 Results
4.1 Refinement results of NEM2E
Figure 3 shows representative results from the NEM2E framework. The results for the One-Entire condition in the walking dataset are shown in (a) and the results for the All-Entire condition in the running dataset are shown in (b). The blue lines indicate simulated muscle activity from OpenSim, the red lines show the refined output from the STDR-Net, and the black lines represent measured EMG data. The horizontal axis denotes the gait cycle (right heel strike to left heel strike), whereas the vertical axis represents muscle activation. In all cases, the refined muscle activation (red) closely matched the measured EMG (black), demonstrating the ability of the framework to improve upon the baseline OpenSim estimates (blue).
FIGURE 3
4.2 Statistical analysis
Tables 2 and 3 summarize the statistical results for the improvement rates in the estimation accuracy for the walking and running datasets, respectively. indicates that the null hypothesis is rejected in the estimation error improvement rate for each muscle calculated by Equation (2). In the walking data, significant improvements in accuracy were observed for the soleus, tib, and rect under the All-Entire condition. No significant improvements were observed for bi and sem under any conditions. In the running dataset, significant improvements were evident for all muscles in the All-Entire condition. Under the One-Quarter condition, only 4 out of the 16 muscles showed significant improvement, and under the One-Entire condition, 10 out of the 16 muscles demonstrated significant improvement.
TABLE 2
| Soleus | bi | Sem | tib | Rect | |
|---|---|---|---|---|---|
| One-Quarter error | −0.19 * (p = 0.036) | 0.30 (p = 0.15) | −0.034 (p = 0.57) | −0.09 * (p = 0.015) | 0.19 (p = 0.19) |
| 95% CI | [-0.37, −0.02] | [-0.13, 0.73] | [-0.17, 0.10] | [-0.16, 0.02] | [-0.13,0.52] |
| Effect size, | −0.92, 0.41 | 0.57, 0.12 | −0.21, 0.22 | −1.13, 0.32 | 0.51, 0.10 |
| One-Entire error | −0.33 ** (p = 0.0012) | 0.11 (p = 0.15) | −0.07 (p = 0.16) | −0.071 (p = 0.17) | 0.19 (p = 0.26) |
| 95% CI | [-0.48, −0.18] | [-0.05, 0.28] | [-0.18, 0.04] | [-0.18, 0.04] | [-0.17,0.55] |
| Effect size, R2 | −1.84, 0.56 | 0.57, 0.12 | −0.56, 0.19 | −0.55, 0.30 | 0.43, 0.16 |
| All-Entire error | −0.31 * (p = 0.035) | 0.15 (p = 0.062) | 0.015 (p = 0.82) | −0.13 * (p = 0.025) | −0.15 * (0.033) |
| 95% CI | [-0.60, −0.029] | [-0.0010, 0.30] | [-0.13, 0.16] | [-0.24, −0.021] | [-0.28, −0.015] |
| Effect size, | −0.92, 0.65 | 0.78, 0.18 | 0.084, 0.12 | −1.00, 0.33 | −0.93, 0.28 |
Statistical analysis for walking dataset.
p . Bold type indicates improved accuracy.
TABLE 3
| Soleus | Bi | Semim | Tib | |
|---|---|---|---|---|
| One-Quarter error | −0.39 *** | −0.04 (p = 0.523) | −0.35 *** | 0.16 (p = 0.097) |
| 95% CI | [-0.52, −0.26] | [-0.18, −0.1] | [-0.43, −0.28] | [-0.03, 0.35] |
| Effect size, | −2.17, 0.92 | −0.21, 0.77 | −3.33, 0.80 | 0.59, 0.70 |
| One-Entire error | −0.48 *** | −0.21 ** (p = 0.003) | −0.44 *** | −0.03 (p = 0.68) |
| 95% CI | [-0.55, −0.40] | [-0.33, −0.09] | [-0.51, −0.36] | [-0.20, 0.14] |
| Effect size, | −4.57, 0.80 | −1.26, 0.44 | −4.05, 0.52 | −0.14, 0.38 |
| All-Entire error | −0.59 *** | −0.42 *** | −0.47 *** | −0.14 * (p = 0.020) |
| 95% CI | [-0.68, −0.51] | [-0.49, −0.36] | [-0.56, −0.38] | [-0.25, −0.03] |
| Effect size, | −4.92, 0.87 | −4.62, 0.59 | −3.77, 0.58 | −0.89, 0.40 |
| Rect | Semit | Glut max1 | Glut max2 | |
|---|---|---|---|---|
| One-Quarter error | −0.023 (p = 0.717) | −0.35 *** | 0.052 (p = 0.63) | 0.038 (p = 0.33) |
| 95% CI | [-0.16, 0.13] | [-0.44, −0.26] | [-0.18, 0.29] | [-0.045, 0.12] |
| Effect size, | −0.12, 0.52 | −2.80, 0.39 | 0.16, 0.20 | 0.33, 0.24 |
| One-Entire error | −0.23 *** | −0.40 *** | 0.0062 (p = 0.96) | 0.0081 (p = 0.89) |
| 95% CI | [-0.39, −0.07] | [-0.48, −0.32] | [-0.28, 0.29] | [-0.11, 0.13] |
| Effect size, | −1.05, 0.42 | −3.59, 0.47 | 0.016, 0.36 | 0.047, 0.33 |
| All-Entire error | −0.37 *** | −0.49 *** | −0.23 *** | −0.12 * (p = 0.013) |
| 95% CI | [-0.48, −0.25] | [-0.58, −0.40] | [-0.34, −0.13] | [-0.21, −0.032] |
| Effect size, | −2.25, 0.52 | −3.93, 0.57 | −1.63, 0.39 | −0.97, 0.36 |
| Glut max3 | Glut med1 | Glut med2 | Glut med3 | |
|---|---|---|---|---|
| One-Quarter error | 0.12 (p = 0.18) | −0.030 (p = 0.70) | −0.12 (p = 0.065) | 0.044 (p = 0.58) |
| 95% CI | [-0.065, 0.30] | [-0.20, 0.14] | [-0.25, 0.0096] | [-0.13, 0.22] |
| Effect size, | 0.46, 0.0.16 | −0.13, 0.38 | −0.66, 0.41 | 0.18, 0.40 |
| One-Entire error | −0.051 (p = 0.49) | −0.23 * (p = 0.019) | −0.027 (p = 0.80) | −0.10 (p = 0.33) |
| 95% CI | [-0.21, 0.11] | [-0.41, −0.047] | [-0.26, 0.20] | [-0.31, 0.12] |
| Effect size, | −0.23, 0.30 | −0.90, 0.45 | −0.084, 0.43 | −0.33, 0.53 |
| All-Entire error | −0.18 *** | −0.35 *** | −0.29 *** | −0.23 *** |
| 95% CI | [-0.30, −0.054] | [-0.43, −0.28] | [-0.49, −0.36] | [-0.33, −0.13] |
| Effect size, | −1.030, 0.36 | −3.32, 0.57 | −0.97, 0.56 | −1.59, 0.57 |
| Gaslad | Gasmed | Vaslat | Vasmed | |
|---|---|---|---|---|
| One-Quarter error | −0.097 (p = 0.33) | −0.096 (p = 0.20) | −0.037 (p = 0.60) | −0.030 (p = 0.70) |
| 95% CI | [-0.31, 0.12] | [-0.045, 0.12] | [-0.20, 0.13] | [-0.20, 0.14] |
| Effect size, | −0.33, 0.54 | 0.33, 0.49 | −0.17, 0.53 | −0.13, 0.38 |
| One-Entire error | −0.32 *** | −0.41 *** | −0.29 ** (p = 0.0040) | −0.47 *** |
| 95% CI | [-0.45, −0.19] | [-0.53, −0.29] | [-0.46, −0.12] | [-0.51, −0.42] |
| Effect size, | −1.74, 0.61 | −2.41, 0.71 | −1.21, 0.72 | −3.59, 0.80 |
| All-Entire error | −0.50 *** | −0.52 *** | −0.39 *** | −0.58 *** |
| 95% CI | [-0.50, −0.29] | [-0.62, −0.42] | [-0.50, −0.29] | [-0.62, −0.42] |
| Effect size, | −3.44, 0.74 | −3.77, 0.78 | −2.75, 0.80 | −9.23, 0.85 |
Statistical analysis for running dataset.
*p . Bold type indicates improved accuracy.
Figure 4 show the improvement rates in the estimation error for the walking and running datasets. A value of 0 represents the baseline accuracy, whereas negative values indicate improvement. In the walking dataset, there were no significant differences between conditions. Conversely, in the running data, significant differences were observed between One-Quarter and All-Entire in 13 out of 16 muscles as well as between One-Entire and All-Entire in bi and glmax2.
FIGURE 4
In the running dataset, significant improvements were observed in all muscles under the All-Entire condition. However, under the One-Entire condition, no significant improvements were observed in 6 out of the 16 muscles. Furthermore, in the walking dataset, significant improvements in the accuracy of the gastrocnemius muscle were observed under all conditions. The rectus femoris muscle showed improvements only under the All-Entire condition, suggesting that the spatial attention mechanism in this condition plays a crucial role in enhancing muscle activity estimation. Furthermore, under the One-Quarter condition, significant improvements were observed in only 5 out of 21 muscles across both running and walking data, indicating the need to consider the entire sequence rather than applying window processing to the sequence data. Moreover, as the models in this study were trained only on running and walking data, shortcut learning may have occurred, in which average muscle activity patterns were learned that are dependent on the movement duration. Then, for the One-Entire model trained on the soleus walk data, which showed strong improvement results in the validation, the improvement rate of the refined results was tested with a T-test by entering 32 randomly frequency- and phase-changed data, which was the same number of test trials in the walk dataset of the original paper. The average improvement rate (- indicates improvement and + indicates degradation) was +0.48, with a p-value of 0.001, showing significant degradation. Thus, the results demonstrate that shortcut learning, which is dependent on the operating time, did not occur.
Furthermore, Figure 5 shows the loss trends for the training and validation data across each learning trial. In the running data, as shown in Figure 5a, the validation loss converged, whereas, in the walking data, as shown in Figure 5b, the loss temporarily increased without decreasing and then converged. For the running data, the validation loss demonstrates a stable relationship with the training loss, consistently decreasing alongside it. This trend suggests that overfitting is unlikely for this dataset, even with the relatively high number of epochs (700). The observed generalization across individuals further indicates that the model is successfully learning the underlying patterns for this specific sequence task without memorizing the training data.
FIGURE 5
Conversely, for the walk data, the validation loss exhibits an increasing trend from early epochs. However, the validation loss remains stagnant and elevated from the outset, suggesting that the intrinsic complexity or inherent noise within this dataset may hinder the model’s ability to achieve substantial learning improvements from the beginning. This observation implies that for the walk data, increasing the dataset size is considered more critical for improving model performance and generalization than simply extending the training duration or adjusting architectural parameters.
4.3 Attention
Figure 6 depicts two representative examples of temporal attention weights in the One-Entire condition. In (I), the attention mechanism primarily focuses on the central part of the input sequence to refine the initial portion of the output waveform of the model. Conversely (II) highlights attention on the latter part of the input sequence, corresponding to the refinement of the second half of the output waveform.
FIGURE 6
The temporal attention results (Figure 6) reveal that the model allocated higher weights to data near the center of the input sequence when refining the muscle activity. This suggests that the temporal distortions in the estimations of OpenSim can be effectively learned and corrected by the attention mechanism. However, the lack of improvement in the One-Quarter condition for certain muscles, such as the biceps femoris long head, underscores the need for longer temporal sequences to capture the full distortion. The fact that the attention was focused on the peak shift in the attention weight results suggests that the model learned as intended, correcting for temporal bias rather than movement duration.
Figure 7 depicts the spatial attention results for the tibialis anterior in the All-Entire condition. Each skeleton visualizes the gait cycle motions at 0%, 50%, and 100%, with the top 10 muscles having the highest attention weights. At 0% and 100%, corresponding to the heel strike phase, similar muscle groups with high attention weights were identified. At 50%, additional trunk and hip muscles were selected, reflecting a shift in the coordination required during the swing phase.
FIGURE 7
The spatial attention revealed that the top 10 muscles with the highest attention weights for the tibialis anterior, which exhibited improved accuracy only in the All-Entire condition, varied at 0%, 50%, and 100% of the gait cycle. This variation may be explained by differences in the active muscle groups during the stance and swing phases of gait.
In addition, the top 10 muscles with the highest attention weights included muscles from the trunk, hip, and lower back on the opposite leg, in addition to the peripheral muscles of the target muscle. The peripheral muscles may have complemented the information obtained through muscle synergy during gait. Regarding the involvement of the hip, pelvis, and trunk muscles of the opposite leg, previous studies have reported that the trailing limb angle is related to ankle moments during gait (). This suggests that ankle torque is not solely generated by muscle activity in the target leg, but also involves coordinated actions with the trunk and opposite leg. Owing to the significant number of muscles associated with the hip and trunk, these regions likely act as a counterweight for ankle movement. The results suggest that incorporating information from these muscles improved the accuracy of the muscle activity estimations by accounting for the broader coordination required during gait.
5 Discussion
We have proposed the NEM2E framework, which corrects the spatio-temporal distortion between OpenSim estimated and real muscle activity. Within this framework, a refined model for estimated muscle activity was trained using Seq2Seq with attention based on open data, and the improvement in estimation accuracy for new users was verified.
The temporal attention results shown in Figure 5a indicate that attention is focused on the 50% gait cycle muscle activity from OpenSim to estimate the 0%–40% gait cycle muscle activity. This result suggests that the proposed method refines the approximately 75 ms temporal shift observed in some muscles, as reported by a previous study () using the running dataset. Because the temporal attention results are generalized across individuals, the results support the previous study’s suggestion that some muscles exhibit a common time delay.
In this study, we specifically address the persistent issue of spatio-temporal distortion in OpenSim muscle activity estimations. Previous studies (; ; ; ) have focused on model tuning within the OpenSim environment to improve the accuracy of musculoskeletal simulators. These include tuning body parameters, aligning joint torque with floor reaction force data, and accounting for uncertainty in tuning parameters, such as muscle length. However, our proposed NEM2E framework introduces a novel external refinement approach. By treating OpenSim outputs as initial estimates and subsequently correcting their spatio-temporal properties with a dedicated refinement network, we provide a complementary solution that enhances the accuracy of existing musculoskeletal models without modifying OpenSim’s core parameters. In some cases, the improvement in accuracy could not be confirmed in certain walk data. One possible explanation is the increased variation in spatio-temporal distortions due to individual differences in balance and control strategies during the double support phase of gait. This variation may have reduced the generalization performance of the model owing to the limited dataset. Because the loss in the verification data in Figure 5b does not decrease, it is necessary to increase the size of the learning data and verify its effectiveness. Increasing the dataset size could address this issue and improve the effectiveness of the model.
A limitation of this work is the use of open data for walking and running as the validation dataset. Therefore, this method can primarily be applied to discrete motions, and the accuracy of the model for entirely unknown motions cannot be guaranteed. However, as generalization to unknown users is allowed, the method can be widely used in the analysis of movements related to walking and running, such as walking analysis of the elderly and athletics. The framework can also be applied to other discrete motions, such as golf swinging and baseball batting and pitching. However, verification regarding motions other than walking and running is crucial. In the future, it is necessary to build a benchmark dataset with a wide range of motions and large volume of data to validate the proposed framework and design appropriate refinement models.
The observed improvements in estimation accuracy for new users, particularly in generalization performance, highlights the importance of explicitly addressing spatio-temporal distortions. These findings directly support the hypothesis that spatio-temporal distortions contribute significantly to errors in OpenSim’s estimations. Furthermore, the effectiveness of the Seq2Seq with attention architecture in isolating and correcting these errors indicates that data-driven models offer a promising means of enhancing the fidelity of physiologically informed models. By externally addressing these distortions through a refinement network, the proposed framework provides a promising approach for enhancing the accuracy and applicability of musculoskeletal models. As a refined model in this study, the Seq2Seq with attention model was used as an example implementation to analyze spatio-temporal distortions; however, more suitable models may exist for this design. If a sufficient dataset can be constructed to train and compare models, the proposed framework could incorporate methods that more accuratey refine the spatio-temporal distortions discussed in this study. As a future direction, if a sufficient dataset can be constructed, research on direct estimation of muscle activity using neural network methods may become feasible.
The code to train the public dataset and perform CV between subjects under the All-Entier condition of this paper is available at SimTK (https://simtk.org/projects/nem2e).
Statements
Data availability statement
Publicly available datasets were analyzed in this study. This data can be found here: https://simtk.org/projects/mspeedwalksimshttps://simtk.org/projects/nmbl_running.
Author contributions
TT: Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Resources, Software, Validation, Visualization, Writing – original draft, Writing – review and editing. TM: Conceptualization, Methodology, Supervision, Writing – review and editing. TN: Conceptualization, Funding acquisition, Project administration, Writing – review and editing. JM: Conceptualization, Funding acquisition, Project administration, Writing – review and editing.
Funding
The author(s) declare that financial support was received for the research and/or publication of this article. This research was supported by JST Moonshot R&D (Grant Number JPMJMS2034), NEDO (Grant Number JPNP20006), and JSPS KAKENHI (Grant Number 24K21325, 23K24925).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declare that no Generative AI was used in the creation of this manuscript.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fbioe.2025.1611414/full#supplementary-material
References
1
AdachiW.TsujiuchiN.KoizumiT.ShiojimaK.TsuchiyaY.InoueY. (2011). Development of walking analysis system using by motion sensor with Mobile force plate. J. Syst. Des. Dyn.6, 655–664. 10.1299/jsdd.6.655
2
AnyBody (2025). AnyBody Modeling System. Aalborg, Denmark: AnyBody Technology. Available online at: https://www.anybodytech.com/.
3
ARMO (2025). Healthcare Tech Solves Corporate and Social Issues and Improves Business Productivity. Tokyo, Japan: G-Sport Inc. Available online at: https://www.gsport.co.jp/products-2/armo/.
4
ArnoldE. M.WardS. R.LieberR. L.DelpS. L. (2010). A model of the lower limb for analysis of human movement. Ann. Biomed. Eng.38, 269–279. 10.1007/s10439-009-9852-5
5
BedoB. L. S.MantoanA.CatelliD. S.CruaudW.ReggianiM.LamontagneM. (2021). Bops: a matlab toolbox to batch musculoskeletal data processing for opensim. Comput. Methods Biomechanics Biomed. Eng.24, 1104–1114. 10.1080/10255842.2020.1867978
6
BlacheY.BegonM.MichaudB.DesmoulinsL.AllardP.Dal MasoF. (2017). Muscle function in glenohumeral joint stability during lifting task. PLos One12, e0189406. 10.1371/journal.pone.0189406
7
CaggianoV.WangH.DurandauG.SartoriM.KumarV. (2022). Myosuite: a contact-rich simulation suite for musculoskeletal motor control. In: FirooziR.MehrN.YelE.AntonovaR.BohgJ.SchwagerM., editors. Proceedings of the 4th annual learning for dynamics and control conference. New York, NY: Proceedings of Machine Learning Research, Vol. 168. p. 492–507.
8
ChristophyM.Faruk SenanN. A.LotzJ. C.O’ReillyO. M. (2012). A musculoskeletal model for the lumbar spine. Biomechanics Model. Mechanobiol.11, 19–34. 10.1007/s10237-011-0290-6
9
DelpS. L.AndersonF. C.ArnoldA. S.LoanP.HabibA.JohnC. T.et al (2007). Opensim: open-Source software to create and analyze dynamic simulations of movement. IEEE Trans. Biomed. Eng.54, 1940–1950. 10.1109/tbme.2007.901024
10
DembiaC. L.BiancoN. A.FalisseA.HicksJ. L.DelpS. L. (2021). Opensim moco: musculoskeletal optimal control. PLOS Comput. Biol.16, e1008493–21. 10.1371/journal.pcbi.1008493
11
Dhaibaworks (2025). Preface. Tokyo, Japan: DhaibaWorks. Available online at: https://www.dhaibaworks.com/.
12
GaoX.XuD.BakerJ. S.Ee-ChonT.LiangM.GuY. (2024). Exploring biomechanical variations in ankle joint injuries among Latin dancers with different stance patterns: utilizing opensim musculoskeletal models. Front. Bioeng. Biotechnol.12, 1359337. 10.3389/fbioe.2024.1359337
13
GastaldiL.PaneroE.RossoV.PastorelliS.VieiraT.BotterA. (2021). Upper limbs musculoskeletal opensim model: customization and assessment. In: NiolaV.GasparettoA., editors. Advances in Italian mechanism science. Cham: Springer International Publishing. p. 162–170.
14
HamnerS. R.DelpS. L. (2013). Muscle contributions to fore-aft and vertical body mass center accelerations over a range of running speeds. J. Biomechanics46, 780–787. 10.1016/j.jbiomech.2012.11.024
15
HamnerS. R.SethA.DelpS. L. (2010). Muscle contributions to propulsion and support during running. J. Biomech.43, 2709–2716. 10.1016/j.jbiomech.2010.06.025
16
HsiaoH.KnarrB. A.PohligR. T.HigginsonJ. S.Binder-MacleodS. A. (2016). Mechanisms used to increase peak propulsive force following 12-weeks of gait training in individuals poststroke. J. Biomechanics49, 388–395. 10.1016/j.jbiomech.2015.12.040
17
KingmaD.BaJ. (2015). Adam: a method for stochastic optimization. In: International conference on learning representations (ICLR);2015 May 7–9; San Diega, CA, USA. Ithaca, NY: ArXiv. p. 1–15.
18
KollerJ. R.RemyC. D.FerrisD. P. (2018). Biomechanics and energetics of walking in powered ankle exoskeletons using myoelectric control versus mechanically intrinsic control. J. Neuroeng Rehabil.15, 42. 10.1186/s12984-018-0379-6
19
LeeL.-F.UmbergerB. R. (2016). Generating optimal control simulations of musculoskeletal movement using opensim and matlab. PeerJ4, e1638. 10.7717/peerj.1638
20
LiuM. Q.AndersonF. C.SchwartzM. H.DelpS. L. (2008). Muscle contributions to support and progression over a range of walking speeds. J. Biomechanics41, 3243–3252. 10.1016/j.jbiomech.2008.07.031
21
LiuT.InoueY.ShibataK. (2010). A wearable force plate system for the continuous measurement of triaxial ground reaction force in biomechanical applications. Meas. Sci. Technol.21, 085804. 10.1088/0957-0233/21/8/085804
22
LuongT.PhamH.ManningC. D. (2015). Effective approaches to attention-based neural machine translation. In: Mà rquezL.Callison-BurchC.SuJ., editors. Proceedings of the 2015 conference on empirical methods in natural language processing. Lisbon, Portugal: Association for Computational Linguistics. p. 1412–1421.
23
MansouriM.ReinboltJ. A. (2012). A platform for dynamic simulation and control of movement based on opensim and matlab. J. biomechanics45, 1517–1521. 10.1016/j.jbiomech.2012.03.016
24
MokhtarzadehH.JiangF.ZhaoS.MalekipourF. (2023). Opencolab project: Opensim in google colaboratory to explore biomechanics on the web. Comput. Methods Biomechanics Biomed. Eng.26, 1055–1063. 10.1080/10255842.2022.2104607
25
NasiriR.AftabiH.AhmadabadiM. N. (2022). Human-in-the-loop weight compensation in upper limb wearable robots towards total muscles’ effort minimization. IEEE Robotics Automation Lett.7, 3273–3278. 10.1109/lra.2022.3144519
26
OhashiT.IkegamiY.NakamuraY. (2020). Synergetic reconstruction from 2D pose and 3D motion for wide-space multi-person video motion capture in the wild. Image Vis. Comput.104, 104028. 10.1016/j.imavis.2020.104028
27
Opensim (2025). OpenSim Software Ecosystem. Houston, Texas: Opensim. Available online at: https://opensim.stanford.edu/.
28
RaabeM. E.ChaudhariA. M. (2016). An investigation of jogging biomechanics using the full-body lumbar spine model: model development and validation. J. Biomechanics49, 1238–1243. 10.1016/j.jbiomech.2016.02.046
29
ReinboltJ. A.SethA.DelpS. L. (2011). Simulation of human movement: applications using opensim. Procedia Iutam2, 186–198. 10.1016/j.piutam.2011.04.019
30
SaulK.MurrayW.DelpS. (2005). A model of the upper extremity for simulating musculoskeletal surgery and analyzing neuromuscular control. Ann. Biomed. Eng.33, 829–840. 10.1007/s10439-005-3320-7
31
SethA.HicksJ. L.UchidaT. K.HabibA.DembiaC. L.DunneJ. J.et al (2018). Opensim: simulating musculoskeletal dynamics and neuromuscular control to study human and animal movement. PLOS Comput. Biol.14, e1006223–20. 10.1371/journal.pcbi.1006223
32
SIMM (2025). Singapore: SIMM. Available online at: https://www.motionanalysis.com/videos/simm/.
33
SutskeverI.VinyalsO.LeQ. V. (2014). Sequence to sequence learning with neural networks. Adv. neural Inf. Process. Syst.27. 10.48550/arXiv.1409.3215
34
ThelenD. G.AndersonF. C.DelpS. L. (2003). Generating dynamic simulations of movement using computed muscle control. J. Biomechanics36, 321–328. 10.1016/s0021-9290(02)00432-3
35
TodorovE.ErezT.TassaY. (2012). Mujoco: a physics engine for model-based control. In: 2012 IEEE/RSJ international conference on intelligent robots and systems; 2012 October 07–12; Vilamoura-Algarve, Portugal: IEEE. p. 5026–5033. 10.1109/IROS.2012.6386109
36
ValenteG.CrimiG.VanellaN.SchileoE.TaddeiF. (2017). Nmsbuilder: freeware to create subject-specific musculoskeletal models for opensim. Comput. Methods. Prog. Biomed.152, 85–92. 10.1016/j.cmpb.2017.09.012
37
ValenteG.PittoL.TestiD.SethA.DelpS. L.StagniR.et al (2014). Are subject-specific musculoskeletal models robust to the uncertainties in parameter identification?PLos One9, e112625–10. 10.1371/journal.pone.0112625
38
WillsonA. M.AndersonA. J.RichburgC. A.MuirB. C.CzernieckiJ.SteeleK. M.et al (2023). Full body musculoskeletal model for simulations of gait in persons with transtibial amputation. Comput. Methods Biomechanics Biomed. Eng.26, 412–423. 10.1080/10255842.2022.2065630
39
ZhangZ.ZouJ.LuP.HuJ.CaiY.XiaoC.et al (2024). Analysis of lumbar spine loading during walking in patients with chronic low back pain and healthy controls: an opensim-based study. Front. Bioeng. Biotechnol.12, 1377767. 10.3389/fbioe.2024.1377767
Summary
Keywords
musculoskeletal simulation, OpenSim, muscle activity estimation, Seq2Seq with attention, spatio-temporal distortion
Citation
Teramae T, Matsubara T, Noda T and Morimoto J (2025) Neural-enhanced motion-to-EMG: refining simulated muscle activity from musculoskeletal models using a Seq2Seq approach. Front. Bioeng. Biotechnol. 13:1611414. doi: 10.3389/fbioe.2025.1611414
Received
14 April 2025
Accepted
02 July 2025
Published
25 July 2025
Volume
13 - 2025
Edited by
Fabiano Bini, Sapienza University of Rome, Italy
Updates
Copyright
© 2025 Teramae, Matsubara, Noda and Morimoto.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Tatsuya Teramae, t-teramae@atr.jp
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.