Augmented feedback modes during functional grasp training with an intelligent glove and virtual reality for persons with traumatic brain injury

1 Introduction

Trauma to the brain can severely impair motor function to perform activities of daily living (Colantonio et al., 2004). For affected individuals, rehabilitation of hand function, especially reaching and grasping, is critical for environmental access (Chaabani et al., 2014). Physical therapy is a primary option to rehabilitate hand function; however, traditional therapy involves intensive and repetitive movement training (Connell et al., 2018). Feelings of rigor during training are naturally detrimental to efficient gains in function; thus, methods fostering greater engagement are needed to overcome the monotony of physical practice (Lohse et al., 2013). Newer approaches to physical therapy are seeking to utilize advanced technologies, such as virtual reality (VR) (Howard, 2017) and instrumented wearables (Simone et al., 2007), to motivate participation in therapy.

Computerized technology, especially virtual reality, is increasingly employed in motor rehabilitation to facilitate greater motivation and to provide customizable training options, including enhanced feedback (Merians et al., 2006). Computerized interfaces can provide robust movement guidance (Gorgey, 2018) and leverage cognitive elements of physical training that can accelerate motor learning after neurological traumas (Mulder and Hochstenbach, 2005). Given their vast programmable features, virtual reality environments are well suited to personalize rehabilitative training that maximizes user engagement and functional outcomes based on neural processes (Holden and Todorov, 2002). Integrating advanced technologies with motor rehabilitation creates a user-computer interface that can motivate with colorful and immersive environments while also providing real-time guidance using enhanced sensory-driven feedback to facilitate motor recovery (Mulder and Hochstenbach, 2005). Thus, virtual environments can optimize motor learning by manipulating training conditions, e.g., guidance cues, for a given user profile, e.g., pathological features, to broadly affect motivational, cognitive, motor, and sensory learning mechanisms (Levin et al., 2015).

Augmented feedback with sensory cues informing individuals about performance achievements or errors during training is proven to enable motor learning (Sigrist et al., 2013). Augmented feedback activates sensory modalities (e.g., visual, audio, haptic) to guide performance during training (Sigrist et al., 2013). With “multimodal” augmented feedback, more than one sensory modality is activated concurrently to hasten motor learning trajectories by broadening the areas of neural activation and exceeding neural activation thresholds earlier during repeated practice (Sigrist et al., 2013; Seitz and Dinse, 2007). Thus, multimodal feedback in VR motor rehabilitation training is a promising approach to recovering motor function after neurological traumas. Our lab has shown how motor performance is sensitive to features in augmented feedback (Sanford et al., 2020) using computerized interfaces for either motion (Sanford et al., 2021) or myoelectric control tasks (Sanford et al., 2022; Walsh et al., 2021).

Still, it remains unclear if persons with neurological damage, such as traumatic brain injury (TBI), respond similarly to augmented feedback approaches as neurotypicals. Given disturbed brain connectivity after TBI (Hayes et al., 2016), the ability to process sensory cues (Folmer et al., 2011) and subsequently apply them with functional capabilities (Ciccarelli et al., 2020) can be compromised. Another potential challenge in utilizing augmented sensory feedback with TBI is a possible deficiency in synchronizing cues with the functional task being practiced (Ghajar and Ivry, 2008). Accurately inferring times of cues relative to task actions is especially critical to ensure motor training with augmented feedback will be effective.

Our lab has previously developed and verified the potential of training with an intelligent glove system capable of providing augmented sensory cues for a functional grasp task while also inducing a sense of agency (Liu et al., 2021). Sense of agency, or perception of control, is a cognitive measure highly associated with motor function (Moore, 2016). Intentional binding is an implicit measure of agency (Moore and Obhi, 2012), which manifests from the compression of one’s perception of the time between a voluntary action and an expected outcome. Our lab has shown positive relationships between implicit measures of agency and movement performance (Nataraj et al., 2020a; Nataraj et al., 2020b; Nataraj and Sanford, 2021; Nataraj et al., 2022) and seeks to leverage such connections for better rehabilitation approaches.

In our training paradigm with the glove system, we facilitate a sense of agency through intentional binding by progressively reducing the delay between the user’s action of a “secure” grasp and the outcome of sensory cues from the onboard modules. The glove system includes onboard force and flex sensors and a processor for an artificial neural network to identify secure grasp, as detailed in (Liu et al., 2021). Participants are cued about their action of securely grasping an object based on sensory-activation modules (visual: LED light, audio: beeper) onboard the glove and then proceed to complete the grasp-and-place task. During training, there is a progressive reduction of the delay between the action and consequential sensory cue to stimulate a perception of greater binding and, therefore, stronger feelings of agency. In our previous study with the intelligent glove system (Liu et al., 2021), we reported that neurotypicals demonstrated improved performance of a grasp-and-place task using “binding” feedback during training compared to no feedback or immediate (no delay) feedback. However, it was unclear if participants with neurological impairment may respond similarly, given potential challenges with discerning timing or processing augmented sensory feedback in VR.

The current study seeks to establish how participants with neurological dysfunction (i.e., TBI) will respond using this glove system when augmented sensory feedback is provided in the following ways: 1) progressively binding feedback to actions during training as done in (Liu et al., 2021), 2) further enhancing the sensory cues through VR, and 3) comparing the effects between providing unimodal (audio only) and multimodal (audio plus visual) cues. Responses in the presence of TBI will be characterized along domains of neural activation (electroencephalography, EEG), functional motor performance, and muscular engagement (electromyography, EMG) and compared against neurotypical responses. We hypothesized that multimodal feedback in VR will support greater neural (EEG) and muscular (EMG) activation and improve performance (reduced motion pathlengths, reduced completion times) of a grasp-and-place task for persons with TBI.

2 Materials and methods

2.1 Participants

Persons with TBI (n = 7) were recruited for a funded study (New Jersey Health Foundation, Research Grant PC 53-19) and tested at Kessler Foundation. These participants signed an informed consent form approved by the Institutional Review Board (IRB) at Kessler. The committee, composed of persons not associated with a given study, reviews and approves all human research studies at Kessler annually. They assure the safety of study participants, patients and healthy volunteers, including the use of clear language in the consent form. These participants were diagnosed as having moderate-to-severe TBI with upper extremity deficits.

Participants with TBI were classified based on the TBI Model Systems National Database (Dijkers et al., 2010), where one of the following criteria must be met: (a) loss of consciousness for 30 min or more; (b) posttraumatic anterograde amnesia for 24 h or more; (c) lowest Glasgow Coma Score (GCS) (Sternbach, 2000) in the first 24 h ≤ 15 (unless due to intubation, sedation, or intoxication); or (d) evidence of significant neurological injury on CT/MRI (e.g., subdural hematoma, cerebral contusion, subarachnoid hemorrhage). Severity was further defined using the following GCS score criteria: mild (14–15), moderate (9–13), or severe (3–8). Injury severity was confirmed from medical records when possible; in the absence of medical records, severity was determined by family member attestations of the length of loss of consciousness/coma.

Another group of neurotypical participants (n = 13) was recruited among a pool of students at Stevens Institute of Technology and compensated using funds from the Charles V. Shaefer, Jr. School of Engineering and Science at Stevens. These participants were tested at Stevens after signing an informed consent form approved by the Stevens IRB. The Stevens IRB is composed of members internal and external to the institution, and it reviews and approves all human research studies at Stevens annually. Neurotypical participants did not report nor indicate complications involving cognition or upper extremity function.

Study enrollment did not require participants to undergo clinical function assessments; thus, limited data were available to infer the degree of motor impairment for TBI participants. However, two participants were sampled from a participant pool having undergone timed tasks for the Wolf Motor Function tests (Lin et al., 2009). The average time score was 31 ± 13 s, which correlates to an upper-extremity Fugl-Meyer score of approximately 40 according to (Hodics et al., 2012), which denotes mild-to-moderate motor impairment (Woytowicz et al., 2017). Furthermore, the average maximum voluntary contraction (MVC) for EMG-recorded muscles of the TBI group was 72% ± 40% for the respective muscles of the neurotypical group. MVC exercises included index-thumb gripping (close- and open-grip directions) and wrist flexion-extension. Overall, we presume that TBI participants for this study have relatively high motor function.

2.2 Instrumented glove system to detect secure grasp

The glove system hardware (Figure 1) included a compression glove embedded with force (Interlink Electronics) and flex (Spectra Symbol) sensors across each digit, aligned on the palmar dorsal side, respectively. The sensors were connected to an instrumentation board (Teensy) programmed with Arduino. The board and wired connections were housed in a custom 3D-printed enclosure with a wrist-strapped mount. Sensory modules onboard the glove included an LED and sound beeper used for visual and audio cues in our previous study (Liu et al., 2021). The function of these sensor modules is now replaced (and enhanced) in this study using VR (details described in Section 2.3). The glove with onboard instrumentation has a mass of under 100 g. API code in MATLAB^® (Mathworks) read sensor data via serial communication at 40 Hz and was processed on an Intel desktop computer (Xeon^® 3.20 GHz, 32 GB RAM, Windows 10 Pro). The board is programmed to run a trained two-layer feedforward artificial neural network (Neural Network Toolbox, MATLAB^®, Mathworks) to compute (predict) whether hand grasp upon an object is secure (or not) based on inputs from the onboard force and flex sensors. The network creation and training procedures are detailed in (Liu et al., 2021). During each training trial, the glove 1) identifies the achievement of a “secure” grasp onto an object, 2) informs the user by activating a feedback module, and 3) facilitates agency via greater “binding” by progressively reducing the delay (from 1 to 0 s across all training trials) between grasp action and feedback cue. A surgical glove was placed over the sensor glove to ensure a better fit to the hand.

FIGURE 1

FIGURE 1. Instrumented glove includes force and flex sensors providing inputs to artificial neural network that predicts when secure grasp on object is achieved and triggers augmented sensory feedback cue. Note: Glove is right-handed shown from dorsal side with thumb inverted to show force-sensitive resistor on palmar side (i.e., thumb-pad).

2.3 Experimental protocol

All participants donned our custom-built instrumented glove on their self-selected dominant side (left- and right-hand versions available) to perform a functional (grasp-and-place) task for all trials. At Stevens, neurotypical participants wore a 32-channel scalp-surface cap for EEG recording (USBamp, g. tec) and skin-surface EMG electrodes (Delsys Trigno) at hand and forearm muscles. EMG recordings were taken at the following seven muscle sites: flexor carpi radialis (proximal flexor), extensor carpi radialis brevis (proximal extensor), flexor digitorum superficialis (distal flexor), extensor digitorum communis (distal extensor), abductor pollicis brevis (thumb abductor, palmar-side recording), adductor pollicis (thumb adductor, dorsal-side recording). At Kessler, TBI participants wore a 64-channel scalp-surface cap for EEG recording (actiCHamp Plus, BrainVision) and seven EMG electrodes (Power Lab/30 Series) at the same locations as neurotypical participants. Protocols at Kessler and Stevens were identical except for the number of trials collected in each block (explained below).

The motor task for each trial entailed reaching and grasping a small cubic object, lifting it from an “Initial” location, and then moving and placing the object onto a “Target” location (Figure 2). Participants were asked to grasp the object with a precision pinch, i.e., using index finger and thumb (Nataraj et al., 2014). Participants with TBI were encouraged to adapt their grasp strategy as needed to perform the task successfully. However, all TBI participants could achieve a precision pinch grasp without discernible adaptation. Participants were informed that they were assessed for performance primarily on minimizing the object’s motion pathlength and secondarily on placing it accurately on the designated target and completing the task promptly.

FIGURE 2

FIGURE 2. Flow diagram shown for experimental procedure for mixed-mode reality grasp-and-place task. Participant wears instrumented glove (under surgical glove) in grasping and moving cubic object while receiving augmented sensory feedback during training.

In adding VR feedback to our glove system, participants experienced mixed-mode reality. They manipulated a real object while viewing a VR environment (Unity) through a headset (HTC Vive) displaying virtual representations of the object and the gloved hand. These representations were identified and translated into VR using a motion controller (LEAP). Calibration procedures were performed to synchronize the positions of the virtual and real cubic objects and have them coincide with the participant’s perspective at the start of each trial. Secure grasp was still detected based on glove sensor inputs to the onboard neural network processor.

In VR, the audio feedback was naturally enhanced when provided through the headset’s earpiece. In addition, the visual cues were enhanced by having the entire virtual object change color (red to green) during secure grasp. Augmented feedback cues about secure grasp were only provided during training trials. During training, participants received augmented sensory feedback upon and during secure grasp in VR. Augmented feedback was delayed upon detecting a secure grasp. However, the delay progressively reduced from 1 to 0 s overall training trials to induce agency through binding (Moore and Obhi, 2012). The earpiece provided unimodal feedback as a singular beep (“audio cue”). The beep was short (100 m sec duration) with moderate tone and pitch. For multimodal feedback, the virtual object additionally changed color (“visual cue”) from red to green. The color change activated concurrently with the audio cue, but it was persistently active during secure grasp and would inactivate (i.e., the virtual object turned red again) upon release of the object. Providing multimodal feedback in this way, i.e., persistent visual cue and audio with a single beep, was most effective (least distracting) to participants based on a series of pilot experiments to validate this training approach with neurotypicals initially (Liu et al., 2021).

In each session, a participant executed three blocks of trials: 1) an initial block of trials without feedback to establish baseline performance (i.e., “pre” training), 2) a block of trials to train with augmented feedback at progressively shorter time delay intervals (1–0 s) after “secure” grasp to induce binding, 3) a block of trials without feedback to determine effects after (i.e., “post”) training. For pre/train/post blocks, neurotypical participants underwent 15/25/15 trials, respectively, and TBI participants underwent 25/50/25 trials, respectively. More trials were undertaken for TBI participants since clinical collaborators had suggested more trials would be better elicit an effect in this population. For neurotypical participants, we followed trial-level procedures according to our previous work with this glove system (Liu et al., 2021). The three blocks of trials were repeated for each of two different feedback conditions during the training block: 1) unimodal feedback (audio cues only), 2) multimodal feedback (concurrent audio and visual cues). These two training conditions were presented in random order for each participant session completed within a single day.

2.4 Data analysis

All metrics were computed as trial averages for each participant before determining the effects of the participant group (TBI versus neurotypical) or feedback condition (unimodal or multimodal). Metrics for performance included the 3-D motion pathlength of the cubic object being transported and the task completion time (i.e., the time the object is being moved from initial to target positions). In both cases, performance is better when the metric is lower. Participants consistently placed the object at the target location; thus, accuracy measures were not evaluated. Instead, the primary performance metric was computed as the object’s motion pathlength while transported from initial to target locations, with completion time serving as a supplementary performance metric.

Neural activity was computed as EEG power in the alpha (8–12 Hz) and beta (13–30 Hz) bands. Additionally, EMG metrics were calculated as the overall mean amplitude across all seven muscles recorded and EMG-EEG coherence. EMG-EEG coherence was computed between an intrinsic hand muscle with the highest EMG amplitude for that participant group (abductor pollicis brevis for neurotypicals, abductor pollicis longus for TBI) and the EEG electrode corresponding to the M1 motor area. Different muscles expressing, on average, maximum EMG amplitude suggests some variation in grip strategy between the neurotypical and TBI groups. However, the variations in grip are likely negligible given abductor pollicis muscles are still highly recruited and likely will reflect changes in EMG-EEG coherence primarily based on feedback training conditions, as designed.

The mean EEG power overall (across all channels) and within the alpha and beta frequency bands were analyzed using “EEGLAB” in MATLAB^®. Mean values for EEG and EMG were computed within a time window that spanned one second before the achievement of secure grasp to one second after the release of the object. All metrics were evaluated as a percentage change from “pre” to “post” blocks to assess the effects of training.

The Kolmogorov–Smirnov test was applied to confirm sufficient normality of each data set to be analyzed by a parametric test. A two-way ANOVA was applied on each measure to determine the effects of the two main experimental factors: feedback condition (i.e., unimodal: audio only; multimodal: audio + visual) and participant group (i.e., neurotypical, TBI). A paired two-sample t-test was used for assessing the simple effects of feedback conditions on each measure within participant groups since identifying the potential impact of feedback conditions for clinical populations is of primary interest in this study. In addition, a one-sample t-test was applied for each pairing of group and feedback condition to determine whether a significant post-training change occurred from baseline (i.e., a non-zero % change). Finally, linear regressions were applied to verify dependent relationships between performance and EEG and whether significant linear trends existed in trial-by-trial changes of each measure within (during) the training block.

3 Results

Examples of brain (alpha power) activation plots are shown for each group (neurotypical, TBI) paired with a training feedback condition (unimodal, multimodal) in Figure 3. Relatively higher alpha power is grossly observable with multimodal feedback for both groups; however, the activation regions appear more diffuse with neurotypicals. For multimodal feedback in TBI, two areas of concentrated activation are evident, including one near the primary motor cortex (M1).

FIGURE 3

FIGURE 3. Representative relative change in regional brain activity (EEG alpha power) after feedback training of a grasp-and-place task. Results shown for neurotypicals and participants with traumatic brain injury (TBI) under two conditions of training with augmented sensory feedback (unimodal: audio only; multimodal: audio plus visual).

The mean values of the percentage changes (i.e., from pre-to post-training) for each metric and the overall mean value at baseline (i.e., pre-training) are provided in Table 1. It should be noted that baseline values were not significantly different between neurotypical and TBI groups for any metric. In addition, for each measure, the specific p-values for individual comparisons between feedback conditions within each group (2-sample t-test), non-zero change from baseline (1-sample t-test), and aggregate factor-level effects for group and training feedback condition (2-way ANOVA) are shown in Table 2. From 2-way ANOVA, only the EEG and EMG-EEG coherence metrics showed a significant factor-level difference and only for the factor of training feedback condition. This result further highlighted the need to examine the simple effects of feedback on each metric within each group to affirm the critical hypothesis of this study (i.e., the presence of neurological dysfunction will alter how multimodal versus unimodal feedback impacts brain activity, muscle engagement, and functional performance of a motor rehabilitation task). Thus, individual metric results are discussed further within the context of feedback conditions with each participant group in the bar plots shown in Figure 4 through 6.

TABLE 1

TABLE 1. Mean baseline (pre-training) value for each metric of interest and mean relative percentage change from baseline after (post) training.

TABLE 2

TABLE 2. p-values indicating if a post-training change in metric is significant (p < 0.05) in comparison to zero (1-sample t-test), between group-condition pairs (2-sample t-test), and within levels for each factor (2-way ANOVA, factors: participant group, feedback condition).

FIGURE 4

FIGURE 4. The relative (percentage) change in EEG power (LEFT = alpha band, RIGHT = beta band) in the performing grasp-and-place task is shown from before training (baseline) to after training with augmented sensory feedback. Results are compared between feedback conditions (unimodal: audio only; multimodal: audio plus visual) per participant group (neurotypicals, TBI). Note: *p < 0.05, **p < 0.01, ***p < 0.001 in comparing effects of feedback condition within each group.

Significant post-training changes were observed in neurological activity (EEG power) within the alpha and beta bands after training with multimodal versus unimodal feedback (Figure 4). Again, results are expressed as the percentage change in each measure after training compared to before, i.e., at baseline. Neurological activity was significantly (p < 0.05) increased in both groups with multimodal feedback for both alpha and beta bands. This increase in activity with multimodal feedback is demonstrated as significant compared to unimodal feedback and from baseline. Unimodal feedback did not produce significant changes from the zero baseline (Table 2).

For performance, both metrics showed improvement (i.e., shorter pathlengths, shorter completion time) with multimodal feedback, compared to unimodal feedback (Figure 5), for the TBI group. However, these performance trends were reversed (i.e., performance worsened with multimodal feedback) for neurotypicals. Of further note, the change in motion pathlength was significantly different from zero (baseline) for every pairing of group and training condition. The change in completion time was significantly non-zero only for neurotypicals with unimodal feedback with significantly improved (reduced) completion time compared to baseline.

FIGURE 5

FIGURE 5. The relative (percentage) change in performance (LEFT = motion pathlength, RIGHT = completion time) in performing the grasp-and-place task is shown from before training (baseline) to after training with augmented sensory feedback. Results are compared between feedback conditions (unimodal: audio only; multimodal: audio plus visual) per participant group (neurotypicals, TBI). Note: *p < 0.05, **p < 0.01, ***p < 0.001 in comparing effects of feedback condition within each group.

EMG metrics did not demonstrate significant differences in any case for persons with TBI (Figure 6). However, there was a significant difference in EMG-EEG coherence between training feedback conditions for neurotypicals. Furthermore, for neurotypicals, the unimodal feedback condition produced a significant increase in EMG coherence from baseline after training, but multimodal feedback produced a significant decrease in coherence.

FIGURE 6

FIGURE 6. The relative (percentage) change in EMG metrics (LEFT = average EMG amplitude across all muscles recorded, RIGHT = EMG-EEG coherence between intrinsic hand muscle with highest amplitude and M1 brain area) in performing the grasp-and-place task is shown from before training (baseline) to after training with augmented sensory feedback. Results are compared between feedback conditions (unimodal: audio only; multimodal: audio plus visual) per participant group (neurotypicals, TBI). Note: *p < 0.05, **p < 0.01, ***p < 0.001 in comparing effects of feedback condition within each group.

When attempting to discover a correlation between performance and neural activity, a significant non-zero slope parameter with linear regression was observed in relating motion pathlength to EEG alpha activity separately for each participant group across both conditions (Figure 7). Notably, the TBI group demonstrated improved performance (reduced motion pathlength) with increased EEG alpha power. However, the neurotypical group showed worsened performance with increased EEG activity.

FIGURE 7

FIGURE 7. Correlations between participant-level mean values of performance metric (motion pathlength) and neural activity (EEG alpha power) within each participant group (LEFT = neurotypical, RIGHT = TBI). Results are shown for after training and pooled across both feedback conditions. Correlation represented through slope parameter for linear regression fitted to data. Slope magnitudes were assessed to be significantly non-zero (p < 0.05).

When examining trends in each metric during training trials, at least one significant difference was observed for each metric pending the specific group or training condition. For both feedback conditions, EEG metrics significantly increased across training trials in the TBI group (Figure 8). For performance metrics (Figure 9), significant improvements (reductions) were observed in completion time for all four group-condition pairs. Significant improvements were observed in motion pathlength only for TBI but with both feedback conditions, leaving non-conclusive trends in motion pathlength for neurotypicals with both conditions. For EMG (Figure 10), a significant reduction in EMG amplitude was observed for TBI and audio-only feedback. In contrast, a significant increase in EMG-EEG coherence was observed for TBI, but with multimodal feedback. Training trends for all remaining EMG cases were inconclusive. The specific slope and associated p-values are presented in Table 3.

FIGURE 8

FIGURE 8. Mean (across participants) of EEG activity (alpha, beta) across sequence of training trials during training with augmented sensory feedback (audio or audio + visual) for each participant group (neurotypical or TBI). Linear regression fitted to indicate global trend within training block in each case pairing feedback condition and participant group. Note: p-values indicate non-zero value for slope coefficient (i.e., significant trend present) of linear regression; *p < 0.05, **p < 0.01, ***p < 0.001.

FIGURE 9

FIGURE 9. Mean (across participants) in performance metrics (motion pathlength, task completion time) across sequence of training trials with augmented sensory feedback (audio or audio + visual) for each participant group (neurotypical or TBI). Linear regression fitted to indicate global trend within training block in each case pairing feedback condition and participant group. Note: p-values indicate non-zero value for slope coefficient (i.e., significant trend present) of linear regression; *p < 0.05, **p < 0.01, ***p < 0.001.

FIGURE 10

FIGURE 10. Mean (across participants) in EMG-related metrics (EMG amplitude, EMG-EEG coherence) across sequence of training trials with augmented sensory feedback (audio or audio + visual) for each participant group (neurotypical or TBI). Linear regression fitted to indicate global trend within training block in each case pairing feedback condition and participant group. Note: p-values indicate non-zero value for slope coefficient (i.e., significant trend present) of linear regression; *p < 0.05, **p < 0.01, ***p < 0.001.

TABLE 3

TABLE 3. The mean slope value (percentage change per trial repetition) during training block per group-condition pair and p-values indicating non-zero slope.

4 Discussion

This study evaluated how varying the nature of augmented sensory feedback used for motor training with virtual reality can impact post-training changes in neurological activity, motor performance, and muscular engagement for a grasp-and-place task. The central experimental factor was cueing neurotypical and TBI participants about secure grasp with either unimodal (audio cue only) or multimodal (audio cue plus visual cue) feedback during each training repetition. Ultimately, we examined the effects of training with each feedback condition by comparing EEG power, motor task performance, and EMG measures immediately after (post) training compared to before (pre) training, serving as the comparative baseline for each participant. Our primary finding was that the effects broadly observed on these measures were unique depending on whether participants were neurotypical or had moderate-to-severe TBI.

Both groups exhibited increased EEG activity, in both alpha and beta bands, after training with multimodal feedback compared to unimodal feedback. More robust EEG responses are generally expected following more exposure to sensory stimulation (Teplan et al., 2006). However, the relative increases with TBI were larger than neurotypicals and may have impacted respective motor outputs accordingly. With multimodal feedback, there were contradictory findings in performance as TBI participants significantly improved (reduced) their average motion pathlength and completion time; however, the additional visual cueing with multimodal feedback worsened performance in both metrics for neurotypicals. This seemingly paradoxical outcome across groups suggests a difference in how the added visual feedback is processed and leveraged for motor performance pending functional neurological states. In particular, multimodal feedback may support the expedited crossing of neural thresholds to improve learning as intended with multimodal feedback (Seitz and Dinse, 2007) for persons having TBI. Yet, for neurotypicals, the additional cueing may be excessive stimuli interpreted as confounding during task training and ultimately interferes with performance and learning progression (Spruit et al., 2016).

Such a finding suggests the need to optimize a computerized rehabilitation interface for users with neurological dysfunction. More specifically, guidance feedback may need to be delivered with greater sensory stimulation. The disturbed brain networks, such as after TBI, can alter how sensory feedback is processed for motor function (Nudo, 2013). Although not analyzed for significant differences, the brain plots in Figure 3 suggest the disparities in regional activation between TBI and neurotypical participants. Thus, increasing sensory stimulation with guidance feedback, even if redundantly encoding the same performance information, may partially compensate for processing dysfunctions with TBI. In any case, assessing the responses to feedback by neurotypicals independently from TBI participants is warranted. However, we still conducted a 2-way ANOVA to determine if each measure is broadly affected by each of the two main factors of group and feedback condition. Only the EEG measures (i.e., alpha power, beta power, and EMG-EEG coherence) demonstrated significant factor-level effects and only for feedback conditions.

While alpha activity is typically suppressed with active movements, it can reflect greater motor preparation (Deiber et al., 2012) and be enhanced by motor training paradigms that increase cognitive flexibility (Lasaponara et al., 2017). The post-training increase in alpha-band activity may suggest the foundation for early and robust consolidation of motor learning features from a pre-learning state (Henz and Schöllhorn, 2016). This phenomenon is readily shown with differential learning, characterized by practice variability to facilitate faster learning rates (Tassignon et al., 2021). The grasp-and-place task for this study was repetitive as it did not vary between trials; however, participants may perceive more variability when training with multimodal feedback. In the case of TBI, this perception of variability may be more effectively leveraged to improve potential motor learning.

On the other hand, beta activity can indicate increased alertness, including by visual stimuli (Kamiński et al., 2012), as done in this study. Regarding motor function, beta waves, especially over the motor cortex, are associated with strengthened sensory feedback during movement changes (Lalo et al., 2007). Thus, the long-term implications of increasing beta activity after each training session may facilitate learning through higher sensory-guided attention during movement training. While EMG metrics in this study were relatively insensitive to changes in feedback training, EMG-EEG coherence was significantly reduced for neurotypicals when receiving multimodal feedback. Since corticomuscular drive, especially in the beta band, indicates a change in muscle coordination strategy (Reyes et al., 2017), the attenuation of corticomuscular coherence may suggest that neurotypicals experienced divided attention (Johnson et al., 2011) in perceiving the added visual cue.

Although assessing post-training effects across measures of EEG, performance, and EMG from the pre-training baseline was the primary objective of this study, we also examined trial-by-trial trends during training for each measure. This analysis provides insight into how these measures may be actively manipulated with each training condition before participants return to independent (unguided) task performance. Although both neurotypicals and TBI demonstrated increased post-training neurological activity, only TBI demonstrated a linear trend towards increased neurological activity within the training block. There was a significant non-zero slope towards increased power across sequential training trials in both the alpha and beta bands.

These findings indicate that TBI participants may have a more immediate tendency to reformulate neural connections during training with augmented feedback. Neural reorganization to facilitate motor recovery is a crucial objective with motor rehabilitation training, and it is primarily expected with visually guided actions (Kantak et al., 2012). Comparatively, neurotypicals may be more limited in their capacity for neural plasticity for a relatively simple motor task. Furthermore, these training trends in neural activation were mirrored with the key performance metric of motion pathlength for both groups. Only TBI participants exhibited a significant trend in reduced pathlength (better performance) during training with more trial repetitions. Such correlates between brain activity and performance can be expected during motor sequence learning (Orban et al., 2010). While the difference in the number of training trials for each group may have impacted the magnitude of the post-training effect, the same trends, i.e., progressive changes in metrics (Figures 8–10), for TBI are readily apparent even halfway through the block of training trials.

For the secondary performance metric of completion time, both groups demonstrated significant trends in reduction across training trials with both feedback conditions. This finding suggests that augmented sensory feedback naturally incentivizes faster movements with more training repetitions. This finding is consistent with another study demonstrating that augmented feedback can impact movement times of reaching movements, irrespective of fixed task parameters (e.g., movement amplitude) (de Grosbois et al., 2015). In comparison, training trends with EMG activity were not as evident. Still, TBI participants did demonstrate a significant training trend in reduced EMG with unimodal feedback and increased EMG-EEG coherence with multimodal feedback. Thus, only the TBI participants appeared to progressively re-organize neural and motor activity during single-session training with augmented sensory feedback.

Furthermore, this study revealed a positive linear-level dependence between higher alpha activity and improved motor performance for TBI, despite a relatively small sample size. This finding suggests that designing rehabilitation paradigms to target increases in alpha activity during the training of persons with TBI may support better motor performance. Identifying and understanding such correlations open new pathways to optimize computerized rehabilitation. For example, control systems can be developed to adapt (personalize) more intelligently specific VR design elements, including feedback features and enhancement levels (e.g., the brightness of color and pitch of sound). The objective of such control systems would be to modulate neural rhythms in ways that are more likely to induce targeted plasticity and increased gains in function.

A presumed limitation of this study is the lack of a more fundamental control condition whereby participants would undergo no augmented feedback for an entire training block. However, the primary goal of this work was to examine the differential impact of multimodal feedback within VR. Thus, this study’s main limitation is that the scope of the evaluation is restricted to a single training session. Authentic learning, and gains in function, can only be ascertained with long-term assessments (e.g., tracking performance across multiple follow-up sessions). Furthermore, the margins of improvement with augmented training feedback we observed in the single session, although significant, likely would not produce a discernible change in performing activities of daily living. Still, short-term performance improvements (i.e., immediately after a single training session) can indicate this approach’s potential for motor learning (Reyes et al., 2017). Initially developed in (Liu et al., 2021), our training approach integrates the sense of agency with augmented sensory feedback cues. Since alpha power may be the primary neural oscillation in the sense of agency (Kang et al., 2015), our approach may leverage a cognitive-sensorimotor synergy in motor training. Furthermore, alpha activity at human M1 for task-specific involvement indicates the potential for rapid motor learning (Muellbacher et al., 2001). Thus, increased alpha activity and improved performance for the TBI group suggest the potency of multimodal VR feedback to promote neuroplasticity for more effective neuromotor rehabilitation.

5 Conclusion

This study demonstrates that training with agency-inspired augmented feedback in VR can significantly impact post-training neural activity, motor performance, and muscular engagement, depending on if the feedback is unimodal or multimodal. Furthermore, these effects can also depend on whether the person has a cognitive impairment (e.g., traumatic brain injury). A notably higher increase in alpha- and beta-band EEG activity after training, especially in brain regions associated with motor planning and execution, may offer an underlying neural explanation for improving motor performance. Thus, augmented feedback, particularly multimodal feedback, provided with VR is a promising approach for rehabilitating motor function after brain injury. Results from this study should motivate future investigations into optimizing the delivery of sensory-driven feedback from computerized rehabilitation interfaces aiming to maximize functional outcomes.

Data availability statement

The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by the Stevens IRB and the Kessler Foundation IRB. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in these studies.

Author contributions

ML wrote the first draft and is primarily accountable for accuracy and integrity of data SW was the primary developer of the mixed-mode reality platform SW, SoS, and RN (lead) contributed to conception of study ML, SoS, and RN (lead) contributed to design of study ML (lead), SeS, MG, SoS, and RN contributed to execution of study ML (lead), SeS, MG, and SD contributed to data collections ML and RN contributed equally to statistical and data analysis ML and RN (lead) interpreted data and revised the manuscript All authors contributed to the article and approved the submitted version.

Funding

This work was made possible by support from the New Jersey Health Foundation (Research Grant, PC 53-19) and the Charles V. Schaefer, Jr. School of Engineering and Science at the Stevens Institute of Technology.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Chaabani, A., Bellamine, M. S., and Gasmi, M. (2014). Motion planning and controlling algorithm for grasping and manipulating moving objects in the presence of obstacles. Int. J. Soft Comput. Artif. Intell. Appl. 3, 01–17. doi:10.5121/ijscai.2014.3401

CrossRef Full Text | Google Scholar

Ciccarelli, G. A., Nolan, M., Rao, H. M., Talkar, T., O'Brien, A., and Vergara, G. (2020). “Human balance models optimized using a large-scale, parallel architecture with applications to mild traumatic brain injury,” in 2020 IEEE High Performance Extreme Computing Conference (HPEC), Waltham, MA, USA, 22-24 September 2020, 1–8.

CrossRef Full Text | Google Scholar

Colantonio, A., Ratcliff, G., Chase, S., Kelsey, S., Escobar, M., and Vernich, L. (2004). Long term outcomes after moderate to severe traumatic brain injury. Disabil. Rehabilitation 26 (5), 253–261. doi:10.1080/09638280310001639722

PubMed Abstract | CrossRef Full Text | Google Scholar

Connell, L. A., Klassen, T. K., Janssen, J., Thetford, C., and Eng, J. J. (2018). Delivering intensive rehabilitation in stroke: factors influencing implementation. Phys. Ther. 98 (4), 243–250. doi:10.1093/ptj/pzy018

PubMed Abstract | CrossRef Full Text | Google Scholar

de Grosbois, J., Heath, M., and Tremblay, L. (2015). Augmented feedback influences upper limb reaching movement times but does not explain violations of Fitts’ Law. Front. Psychol. 6, 800. doi:10.3389/fpsyg.2015.00800

PubMed Abstract | CrossRef Full Text | Google Scholar

Deiber, M.-P., Sallard, E., Ludwig, C., Ghezzi, C., Barral, J., and Ibañez, V. (2012). EEG alpha activity reflects motor preparation rather than the mode of action selection. Front. Integr. Neurosci. 6, 59. doi:10.3389/fnint.2012.00059

PubMed Abstract | CrossRef Full Text | Google Scholar

Dijkers, M. P., Harrison-Felix, C., and Marwitz, J. H. (2010). The traumatic brain injury model systems: history and contributions to clinical service and research. J. head trauma rehabilitation 25 (2), 81–91. doi:10.1097/htr.0b013e3181cd3528

PubMed Abstract | CrossRef Full Text | Google Scholar

Folmer, R. L., Billings, C. J., Diedesch-Rouse, A. C., Gallun, F. J., and Lew, H. L. (2011). Electrophysiological assessments of cognition and sensory processing in TBI: applications for diagnosis, prognosis and rehabilitation. Int. J. Psychophysiol. 82 (1), 4–15. doi:10.1016/j.ijpsycho.2011.03.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Ghajar, J., and Ivry, R. B. (2008). The predictive brain state: timing deficiency in traumatic brain injury? Neurorehabilitation neural repair 22 (3), 217–227. doi:10.1177/1545968308315600

PubMed Abstract | CrossRef Full Text | Google Scholar

Gorgey, A. S. (2018). Robotic exoskeletons: the current pros and cons. World J. Orthop. 9 (9), 112–119. doi:10.5312/wjo.v9.i9.112

PubMed Abstract | CrossRef Full Text | Google Scholar

Hayes, J. P., Bigler, E. D., and Verfaellie, M. (2016). Traumatic brain injury as a disorder of brain connectivity. J. Int. Neuropsychological Soc. 22 (2), 120–137. doi:10.1017/s1355617715000740

CrossRef Full Text | Google Scholar

Henz, D., and Schöllhorn, W. I. (2016). Differential training facilitates early consolidation in motor learning. Front. Behav. Neurosci. 10, 199. doi:10.3389/fnbeh.2016.00199

PubMed Abstract | CrossRef Full Text | Google Scholar

Hodics, T. M., Nakatsuka, K., Upreti, B., Alex, A., Smith, P. S., and Pezzullo, J. C. (2012). Wolf Motor Function Test for characterizing moderate to severe hemiparesis in stroke patients. Archives Phys. Med. rehabilitation 93 (11), 1963–1967. doi:10.1016/j.apmr.2012.05.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Holden, M. K., and Todorov, E. (2002). “Use of virtual environments in motor learning and rehabilitation,” in Handbook of virtual environments (United State: CRC Press), 1039–1066.

Google Scholar

Howard, M. C. (2017). A meta-analysis and systematic literature review of virtual reality rehabilitation programs. Comput. Hum. Behav. 70, 317–327. doi:10.1016/j.chb.2017.01.013

CrossRef Full Text | Google Scholar

Johnson, A. N., Wheaton, L. A., and Shinohara, M. (2011). Attenuation of corticomuscular coherence with additional motor or non-motor task. Clin. Neurophysiol. 122 (2), 356–363. doi:10.1016/j.clinph.2010.06.021

PubMed Abstract | CrossRef Full Text | Google Scholar

Kamiński, J., Brzezicka, A., Gola, M., and Wróbel, A. (2012). Beta band oscillations engagement in human alertness process. Int. J. Psychophysiol. 85 (1), 125–128. doi:10.1016/j.ijpsycho.2011.11.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Kang, S. Y., Im, C. H., Shim, M., Nahab, F. B., Park, J., Kim, D. W., et al. (2015). Brain networks responsible for sense of agency: an EEG study. PLoS ONE 10 (8), e0135261. doi:10.1371/journal.pone.0135261

PubMed Abstract | CrossRef Full Text | Google Scholar

Kantak, S. S., Stinear, J. W., Buch, E. R., and Cohen, L. G. (2012). Rewiring the brain: potential role of the premotor cortex in motor control, learning, and recovery of function following brain injury. Neurorehabilitation neural repair 26 (3), 282–292. doi:10.1177/1545968311420845

PubMed Abstract | CrossRef Full Text | Google Scholar

Lalo, E., Gilbertson, T., Doyle, L., Lazzaro, V. D., Cioni, B., and Brown, P. (2007). Phasic increases in cortical beta activity are associated with alterations in sensory processing in the human. Exp Brain Res. 177, 146–145. doi:10.1007/s00221-006-0828-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Lasaponara, S., Mauro, F., Carducci, F., Paoletti, P., Tombini, M., Quattrocchi, C. C., et al. (2017). Increased alpha band functional connectivity following the Quadrato Motor Training: a longitudinal study. Front. Hum. Neurosci. 11, 282. doi:10.3389/fnhum.2017.00282

PubMed Abstract | CrossRef Full Text | Google Scholar

Levin, M. F., Weiss, P. L., and Keshner, E. A. (2015). Emergence of virtual reality as a tool for upper limb rehabilitation: incorporation of motor control and motor learning principles. Phys. Ther. 95 (3), 415–425. doi:10.2522/ptj.20130579

PubMed Abstract | CrossRef Full Text | Google Scholar

Lin, K., Hsieh, Y., Wu, C., Chen, C., Jang, Y., and Liu, J. (2009). Minimal detectable change and clinically important difference of the Wolf motor function test in stroke patients. Neurorehabil Neural Repair 23 (5), 429–434. doi:10.1177/1545968308331144

PubMed Abstract | CrossRef Full Text | Google Scholar

Liu, M., Wilder, S., Sanford, S., Saleh, S., Harel, N. Y., and Nataraj, R. (2021). Training with agency-inspired feedback from an instrumented glove to improve functional grasp performance. Sensors 21 (4), 1173. doi:10.3390/s21041173

PubMed Abstract | CrossRef Full Text | Google Scholar

Lohse, K., Shirzad, N., Verster, A., Hodges, N., and Van der Loos, H. F. M. (2013). Video games and rehabilitation: using design principles to enhance engagement in physical therapy. J. Neurologic Phys. Ther. 37 (4), 166–175. Art. no. 4. doi:10.1097/NPT.0000000000000017

PubMed Abstract | CrossRef Full Text | Google Scholar

Merians, A. S., Poizner, H., Boian, R., Burdea, G., and Adamovich, S. (2006). Sensorimotor training in a virtual reality environment: does it improve functional recovery poststroke? Neurorehabilitation Neural Repair 20, 252–267. doi:10.1177/1545968306286914

PubMed Abstract | CrossRef Full Text | Google Scholar

Moore, J. W. (2016). What is the sense of agency and why does it matter? Front. Psychol. 7 (Aug), 1272. doi:10.3389/fpsyg.2016.01272

PubMed Abstract | CrossRef Full Text | Google Scholar

Moore, J. W., and Obhi, S. S. (2012). Intentional binding and the sense of agency: a review. Conscious. Cognition 21 (1), 546–561. doi:10.1016/j.concog.2011.12.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Muellbacher, W., Ziemann, U., Boroojerdi, B., Cohen, L., and Hallett, M. (2001). Role of the human motor cortex in rapid motor learning. Exp Brain Res. 136 (4), 431–438. doi:10.1007/s002210000614

PubMed Abstract | CrossRef Full Text | Google Scholar

Mulder, T., and Hochstenbach, J. (2005). “Motor control and learning: implications for neurological rehabilitation,” in Handbook of neurological rehabilitation (London: Psychology Press), 159–168.

Google Scholar

Nataraj, R., Evans, P. J., Seitz, W. H., and Li, Z.-M. (2014). Effects of carpal tunnel syndrome on reach-to-pinch performance. PloS one 9 (3), e92063. doi:10.1371/journal.pone.0092063

PubMed Abstract | CrossRef Full Text | Google Scholar

Nataraj, R., Hollinger, D., Liu, M., and Shah, A. (2020b). Disproportionate positive feedback facilitates sense of agency and performance for a reaching movement task with a virtual hand. PLoS One 15 (5), e0233175. doi:10.1371/journal.pone.0233175

PubMed Abstract | CrossRef Full Text | Google Scholar

Nataraj, R., and Sanford, S. (2021). Control modification of grasp force covaries agency and performance on rigid and compliant surfaces. Front. Bioeng. Biotechnol. 8, 574006. doi:10.3389/fbioe.2020.574006

PubMed Abstract | CrossRef Full Text | Google Scholar

Nataraj, R., Sanford, S., Liu, M., and Harel, N. Y. (2022). Hand dominance in the performance and perceptions of virtual reach control. Acta Psychol. 223, 103494. doi:10.1016/j.actpsy.2022.103494

CrossRef Full Text | Google Scholar

Nataraj, R., Sanford, S., Shah, A., and Liu, M. (2020a). Agency and performance of reach-to-grasp with modified control of a virtual hand: implications for rehabilitation. Front. Hum. Neurosci. 14, 126. doi:10.3389/fnhum.2020.00126

PubMed Abstract | CrossRef Full Text | Google Scholar

Nudo, R. J. (2013). Recovery after brain injury: mechanisms and principles. Front. Hum. Neurosci. 7, 887. doi:10.3389/fnhum.2013.00887

PubMed Abstract | CrossRef Full Text | Google Scholar

Orban, P., Peigneux, P., Lungu, O., Albouy, G., Breton, E., Laberenne, F., et al. (2010). The multifaceted nature of the relationship between performance and brain activity in motor sequence learning. Neuroimage 49 (1), 694–702. doi:10.1016/j.neuroimage.2009.08.055

PubMed Abstract | CrossRef Full Text | Google Scholar

Reyes, A., Laine, C. M., Kutch, J. J., and Valero-Cuevas, F. J. (2017). Beta band corticomuscular drive reflects muscle coordination strategies. Front. Comput. Neurosci. 11, 17. doi:10.3389/fncom.2017.00017

PubMed Abstract | CrossRef Full Text | Google Scholar

Sanford, S., Collins, B., Liu, M., Dewil, S., and Nataraj, R. (2022). Investigating features in augmented visual feedback for virtual reality rehabilitation of upper-extremity function through isometric muscle control. Front. Virtual Real. 159. doi:10.3389/frvir.2022.943693

CrossRef Full Text | Google Scholar

Sanford, S., Liu, M., and Nataraj, R. (2021). Concurrent continuous versus bandwidth visual feedback with varying body representation for the 2-legged squat exercise. J. Sport Rehabil. 30, 794–803. doi:10.1123/jsr.2020-0234

PubMed Abstract | CrossRef Full Text | Google Scholar

Sanford, S., Liu, M., Selvaggi, T., and Nataraj, R. (2020). Effects of visual feedback complexity on the performance of a movement task for rehabilitation. J. Mot. Behav. 53, 243–257. doi:10.1080/00222895.2020.1770670

PubMed Abstract | CrossRef Full Text | Google Scholar

Seitz, A. R., and Dinse, H. R. (2007). A common framework for perceptual learning. Curr. Opin. Neurobiol. 17 (2), 148–153. doi:10.1016/j.conb.2007.02.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Sigrist, R., Rauter, G., Riener, R., and Wolf, P. (2013). Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review. Psychon. Bull. Rev. 20 (1), 21–53. doi:10.3758/s13423-012-0333-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Simone, L. K., Sundarrajan, N., Luo, X., Jia, Y., and Kamper, D. G. (2007). A low cost instrumented glove for extended monitoring and functional hand assessment. J. Neurosci. methods 160 (2), 335–348. doi:10.1016/j.jneumeth.2006.09.021

PubMed Abstract | CrossRef Full Text | Google Scholar

Spruit, E. N., Kleijweg, L., Band, G. P., and Hamming, J. F. (2016). Varied practice in laparoscopy training: beneficial learning stimulation or cognitive overload? Front. Psychol. 7, 685. doi:10.3389/fpsyg.2016.00685

PubMed Abstract | CrossRef Full Text | Google Scholar

Sternbach, G. L. (2000). The Glasgow coma scale. J. Emerg. Med. 19 (1), 67–71. doi:10.1016/s0736-4679(00)00182-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Tassignon, B., Verschueren, J., Baeyens, J. P., Benjaminse, A., Gokeler, A., Serrien, B., et al. (2021). An exploratory meta-analytic review on the empirical evidence of differential learning as an enhanced motor learning method. Front. Psychol. 12, 533033. doi:10.3389/fpsyg.2021.533033

PubMed Abstract | CrossRef Full Text | Google Scholar

Teplan, M., Krakovska, A., and Štolc, S. (2006). EEG responses to long-term audio–visual stimulation. Int. J. Psychophysiol. 59 (2), 81–90. doi:10.1016/j.ijpsycho.2005.02.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Walsh, K. A., Sanford, S. P., Collins, B. D., Harel, N. Y., and Nataraj, R. (2021). Performance potential of classical machine learning and deep learning classifiers for isometric upper-body myoelectric control of direction in virtual reality with reduced muscle inputs. Biomed. Signal Process. Control 66, 102487. doi:10.1016/j.bspc.2021.102487

CrossRef Full Text | Google Scholar

Woytowicz, E. J., Rietschel, J. C., Goodman, R. N., Conroy, S. S., Sorkin, J. D., Whitall, J., et al. (2017). Determining levels of upper extremity movement impairment by applying a cluster analysis to the Fugl-Meyer assessment of the upper extremity in chronic stroke. Archives Phys. Med. rehabilitation 98 (3), 456–462. doi:10.1016/j.apmr.2016.06.023

CrossRef Full Text | Google Scholar