Visual assistance may impede sensorimotor learning during gamified rehabilitation exercises

Regular exercise is critical to post-stroke recovery, but can be frustrating and difficult for patients with limited motor capabilities. Robotic assistance devices are being used to support this process, but due to cost and accessibility concerns some researchers have drawn inspiration from virtual reality and proposed the use of what we call “visual-only assistance”, in which a patient’s physical movements are mapped into game-oriented visual feedback modified towards greater success in the gameplay objectives. Our concern is that the motivational benefits it provides may come at the cost of reduced sensorimotor learning, which could ultimately be counterproductive to the recovery process. To explore these concerns, we conducted a between-subjects study with 24 participants to examine how two types of visual-only assistance affect short-term proprioceptive skill learning in a motor training game involving airplane steering with wrist rotation. One group experienced “attractor assistance”, in which the airplane was continuously displaced toward an ideal position and orientation. The other experienced “tunnel assistance”, in which direct user control was maintained unless straying too far, triggering an invisible barrier. We hypothesized that motor learning would be more impeded with attractor assistance due to the constant mapping variation between physical movements and visual feedback, but our experiment found that motor learning did not materialize in either condition, suggesting that substituting intermittent visual assistance for constant visual assistance is insufficient to guarantee superior motor learning outcomes and that, pending further investigation, the use of visual assistance for proprioceptive training should continue to be regarded with caution.

1 Introduction

Motor rehabilitation is essential for the functional recovery and independence in people after a stroke (Langhorne et al., 2011; Dobkin, 2004). Research has shown that the repetition of movements can effectively assist in the re-establishment of the motor control pathway (Bütefisch et al., 1995; Kawahira et al., 2004) through reinforcement of sensorimotor feedback and proprioception, which makes it a core component of motor rehabilitation therapies.

However, the plain repetition of movements that are out of context can be tedious and unengaging. Many existing studies suggest that nonadherence to rehabilitation therapy is one of the main factors limiting the effectiveness of rehabilitation training (Dishman and Ickes, 1981; Sluijs et al., 1993). Therefore, there has been a growing trend to incorporate gamification elements to rehabilitation therapies to improve engagement and adherence (Burke et al., 2009; Mubin et al., 2020), usually with adjustable difficulty to accommodate the varying ability of stroke patients (Karime et al., 2013). Nevertheless, acute stroke patients can still find it too challenging to complete the gamified tasks unassisted due to their very limited motor capabilities, and making the tasks too simple can defeat the purpose of an engaging and motivating experience.

To account for this, dedicated robotic devices are designed and developed to assist the patients in executing the movements they intend to make by sensing and magnifying their own efforts towards a static or dynamic goal state, defined in the context of an appropriately designed gamified task (Marini et al., 2017; Veerbeek et al., 2017; Hussain et al., 2020). The use of robotic assistance is proved to be effective and reliable in facilitating motor recovery (Veerbeek et al., 2017; Hussain et al., 2020) but many complicated factors remain to be addressed, most notably the access to these devices being limited to onsite therapy sessions.

Since regular motor exercise is key to motor rehabilitation, researchers have been considering the deployment of gamified recovery support software (Rand et al., 2014; Baranyi et al., 2014), including games that provide what we call “visual assistance” as a potential compliment to the use of robotic assistance so that patients can perform motor exercises from home while allowing the therapists to track gameplay duration (adherence) and performance (skills improvement) remotely to support them in optimally steering the recovery process (Dukes et al., 2013; Ferdous et al., 2022). With visual assistance, a patient’s physical movements are mapped into game-oriented visual feedback in a way that is either overtly or covertly modified towards greater success in the gameplay objectives, which achieves motivational benefits similar to robotic assistance but without requiring actual mechanical devices.

While such visual manipulation can be implemented in different forms and is commonly used in virtual reality (VR) applications for various purposes (Poupyrev et al., 1996; Frees and Kessler, 2005; Wentzel et al., 2020) to improve usability, in rehabilitation exercises the ultimate goal is sensorimotor learning, which requires the patients to process sensory information and learn how to control their movements based on the sensory feedback. Our concern is that a method of visual assistance that applies a constantly-varying manipulation to the mapping between a user’s physical movement and the visual feedback they receive about that movement could be counter-productive to sensorimotor learning, because the same movement of the patient can lead to different outcomes depending on what the assistance is doing, which, in spite of motivational benefits, makes it harder for them to appreciate how they are controlling their physical movements. Therefore, in this study we hypothesize that a visual assistance method that preserves the movement-to-feedback mapping to greater extent can better accomplish the goal of avoiding patient frustration while enabling better motor training outcomes by affording more robust sensorimotor learning opportunities.

2 Related work

2.1 Proprioceptive training and sensorimotor rehabilitaion

Neurological conditions such as stroke can cause motor impairment as well as proprioceptive dysfunction (Carey et al., 1996; Kenzie et al., 2014). It is well-established that training of repetitive movements with volitional effort is an effective intervention to aid in stroke rehabilitation (Bütefisch et al., 1995; Kawahira et al., 2004; Tacchino et al., 2016), and sensorimotor training can improve proprioceptive functions, which further leads to improved motor performance (Elangovan et al., 2017; Elangovan et al., 2018). The rationale is that the relevant neuro-connections can be reorganized through the proprioceptive training and the brain can relearn the mapping between the motor inputs and outcomes. It is worth noting that measurable improvement in proprioceptive performance can not only be observed in stroke patients but also in healthy populations even after short-term training (Aman et al., 2015; Ostry and Gribble, 2016; Cuppone et al., 2018), so as a matter of ethics, researchers avoid using clinical populations for preliminary investigations, where results from healthy participants can be sufficient to inform the next steps in the work (Elangovan et al., 2017; 2018; Wang et al., 2021).

2.2 Robotic assistance for rehabilitation

In cases where the participants are unable to complete the training tasks due to limited motor capability, the standard solution is to assist their movement using exoskeletons or other robotic devices (Veerbeek et al., 2017; Hussain et al., 2020). Many different robotic systems have been developed for this purpose and proved to be effective. Masiero et al. created a simple 3-DOF wire-based robot which uses motors to control the lengths of the wires and pull the patients’ upper limbs to assist their motion (Masiero et al., 2007). Studies have demonstrated that early sensorimotor training using this device can speed up the recovery. Montagner et al. developed a force feedback exoskeleton for the right arm, and achieved positive results in terms of rehabilitation outcomes and user feedback in a pilot study consisting of various motor training tasks (Montagner et al., 2007). A wrist rehabilitation robot developed by Masia et al. can apply torques to either assist or resist the rotation of the wrist and is used in both motor control studies and rehabilitation (Masia et al., 2009). In Borboni et al. (2016), designed a hand rehabilitation glove capable of assisting personalized combinations of finger movements and showed its efficacy with experimental results on 126 stroke patients. However, a major constraint of robotic assistance is that the robotic devices are only available in hospitals or clinics, which significantly limits their accessibility.

2.3 Visual feedback manipulation

In recent years, many elements of VR have also been incorporated into motor training (Laver et al., 2017; Hao et al., 2022; Demeco et al., 2023; Khokale et al., 2023), including the introduction of visual manipulation in an assistive manner to support motor training when robotic assistance is unavailable.

In Virtual Reality, things are not constrained by the laws of physics, and many interaction techniques have been proposed that manipulate the visual feedback of users’ spatial input in a way that assists with the completion of specific tasks in the virtual environment. For example, “go-go” (Poupyrev et al., 1996) and its variants (Wentzel et al., 2020) allow users to work with objects out of reach by scaling up the displayed movements when they exceed $2/3$ of the user’s arm length. Precise and rapid interaction through scaled manipulation (PRISM) is a method that scales down the displayed movement to improve stability when performing fine movements (Frees and Kessler, 2005; Frees et al., 2007). There are also studies that implicitly modify users’ task performance in VR towards greater success, such as in the ball throwing task described in (Murakami et al., 2024), where the trajectory of the ball is pushed towards the target. The methods discussed in these studies share a similar goal to robotic assistance in that they aim to improve engagement and motivation by enabling users to perform certain tasks more easily or accurately than what they are actually capable of.

Some work has also incorporated the idea of visual assistance into the context of stroke rehabilitation and motor training. In the “duck duck punch” rehabilitation game developed by Dukes et al., the movements of stroke patients are visually amplified to account for their limited range of motion (Dukes et al., 2013). Similarly, Ferdous et al. used scaled movement to improve the reach of Parkinson patients in a VR rehabilitation game of smashing balloons (Ferdous et al., 2022). The same idea has also been used in lower-extremity rehabilitation training, such as in (Chowdhury et al., 2019), where rotational gain is applied to patients’ leg movement to assist them in a rehabilitation game involving kicking a ball. However, all the above-mentioned studies focused on task completion and motivational benefits without explicitly considering the introduction of visual assistance affects training outcomes. In Ross et al. (2023), demonstrated that the “duck duck punch game” combined with aerobic exercise can improve the upper-extremity function of stroke patients, but that training was targeting range-of-motion, i.e., how far one can move, rather than sensorimotor learning, i.e., how accurate one can perceive and execute movements. To the best of our knowledge no study has yet specifically considered the impact of visual assistance on the acquisition of sensorimotor skills.

Therefore, in this study we aim to examine the sensorimotor learning in a motor training game when visual assistance is present, and explore whether a variant of visual assistance that introduces manipulation in an “as-needed” fashion could effectively reduce the potential disruption of sensorimotor learning and achieve a superior training outcome compared to traditional visual assistance that is “always on” and employs a constantly varying mapping between action and outcome.

3 Methods

We conducted a between-subject study to compared the proprioceptive training outcomes and user experience between two implementations of visual assistance, one that constantly manipulates the visual feedback towards greater in-game success, and one that becomes active only when the user is doing exceptionally poorly. Based on our concern that visual feedback manipulation may disrupt proprioceptive learning, we hypothesized that an “as-needed” variant of visual assistance might enable better motor training improvement due to the reduced amount of visual manipulation being introduced, but might also potentially sacrifice usability due to the users performing more poorly. We did not consider the need for a control condition in which no assistance was applied, because we were focused on addressing practical situations in which some sort of assistance, either visual or robotic, is needed. However, we recognize in retrospect that including such a condition would have been wise.

3.1 Apparatus and materials

3.1.1 Experimental equipment

We used a wrist rehabilitation robot built by the Italian Institute of Technology (Masia et al., 2009) to track the user’s wrist movements in our study. This device has three degrees of freedom in movement, corresponding to pronation/supination, flexion/extension, and abduction/adduction of the human wrist. To use this device, participants sat on a stool, rested their arm on the device’s armrest, and grabbed the stick of the wrist robot with their dominant hand. In this study, the robot did not provide any mechanical assistance. The reason why we used a wrist robot as the tracker, as opposed to a smartphone or VR controllers was because in a future study we wanted to compare visual and robotic assistance with the hardware as a controlled variable, and this study served as a preliminary study to explore possible visual assistance design. The robot was connected to a desktop computer running Linux Mint (Version 18) and visual feedback was displayed on a monitor that was mounted at eye height on a narrow table approximately 40 cm in front of the participants. See Figure 1 for an illustration of the experimental setup.

Figure 1

Figure 1. A person playing the motor training game using the wrist robot and collage of the different scene variations in the game.

3.1.2 Motor training game

We developed a motor training game using Unity3D. The game was inspired by air racing, in which the player controls a virtual plane flying forward on its own at constant speed. The plane’s rotation is linked to the user’s wrist rotation. The goals are to steer the plane to fly through all gates and match the rotations visualized as outlines of plane’s rotational pose naturally maps to the 3-DOF of human wrist, and by setting the location of the gates and the orientation of the outlines we can define the required wrist rotations in an intuitive manner. The game has 20 procedurally generated levels, each consisting of 15 gates, where the forward displacement between adjacent gates is 15 units and the horizontal and vertical displacements are randomized within $\pm [\mathrm{1,3}]$ and $\pm [0.5,1]$ units, respectively. The target rotation is randomized within [-45°, 45°]. The purpose of the randomization to ensure non-repetitive gameplay, yet consistency in difficulty across levels. The forward speed of the plane is 4.5 units/second, meaning that each level takes 50 s to complete. We also created five scene variations shown every four levels to reduce visual fatigue (see Figure 1).

Whenever the plane reaches a gate, a score is calculated based on the accuracy of its position and orientation with respect to the target outline. Table 1 shows the scoring system. The players were shown how many additional points they earned after passing each gate, and the total score for each level was shown after the level’s completion.

Table 1

Table 1. Table of all possible situations when the plane is passing a gate and their corresponding scores.

3.1.3 Visual assistance

We implemented visual-only counterparts of two commonly used mechanical assistance schemes (Basteris et al., 2014): 1) The moving attractor (Chang et al., 2007; Lambercy et al., 2011; Squeri et al., 2013), in which an assistive force constantly attracts the end-effector towards a target trajectory and the amount of assistance is modulated by varying a stiffness parameter, e.g., proportional to the distance between the end-effector and the target, and 2) the virtual tunnel (Casadio et al., 2009), in which assistive forces are introduced only after the error exceeds a certain threshold.

Specifically, the two types of visual assistance we implemented for the game are as follows,

• Attractor: in which the plane is constantly attracted towards a predefined optimal trajectory and its roll is adjusted toward the required roll for the upcoming gate.

• Tunnel: in which the plane is prevented from deviating beyond a certain distance from a predefined optimal trajectory or beyond a certain angular displacement from the target orientation at the next gate

To avoid over-assistance, the assisted movement for the attractor is proportional to: 1) the distance from the plane location to the closest point on the trajectory $(d1)$, and 2) the distance in the forward direction from the plane to the next gate $(d2)$. Specifically, the assisted movement increases as the plane deviates farther away from the optimal trajectory i.e., when the player is not doing well, and as the plane approaches the gates, i.e., when there is less time left to adjust. See Figure 2 for a visual illustration. Specifically, the attractive force $f$, i.e., how much the position/rotation of the plane is adjusted towards the target position/rotation in one unit of time is calculated as $f=k\times (d1+(d-d2))$, where $d$ is the distance between adjacent gates, and $k$ is the scaling factor. Similarly, for tunnel assistance the threshold of allowable positional and rotational error gets smaller at a specific rate $(s)$ until reaching a minimum value $(minR)$ as the plane approaches the gate, allowing more freedom when the gate is far ahead and less room for error when the gate is near. The allowable error threshold at any given distance $d2$ to the next gate can be calculated as $thres=minR+s\times d2$. See Figure 2 for visual illustration of tunnel assistance. In both cases of assistance, the optimal trajectory is displayed as a semi-transparent curve in the game for reference.

Figure 2

Figure 2. Illustration of “Attractor” and “Tunnel” visual assistance. The attractive force increases as the plane deviates from optimal trajectory (d1 increases) and approaches the gates (d2 decreases). Similarly, the radius of the tunnel decreases as the plane approaches the gates.

In order to optimize the comparison between the two assistance methods, we conducted a pilot study with four participants from our research group to tune the parameters of the positional and rotational assistance as they played the game. The participants were guided to adjust the strength of the assistance so that the game felt moderately challenging and rewarding enough that they would be willing to play it for a long time. We asked them to tune one parameter at a time, and they went through each possible permutation of parameter tuning order 3 times with different random starting values. During the process, the participants used their dominant hand to control the airplane’s position and orientation via the wrist robot, and used their non-dominant hand to press the up/down arrow keys on the keyboard to increase/decrease the amount of assistance being provided. For our formal study, we used the median of the final values chosen by the four participants in the pilot study.

3.2 Participants

We recruited 24 participants (13M, 11F) between the ages of 18–34 from our University community, using posters and online platforms. Half (6M, 6F) were randomly assigned to the attractor condition, and half (7M, 5F) to the tunnel condition. While consistent with the number of participants used in prior related work Elangovan et al. (2017), Elangovan et al. (2018) we note that this is significantly less than the 128 participants recommended by the power analysis conducted on G*Power 3.1.9.7 for an independent t-test with a medium expected effect size (0.5) and alpha = 0.05 to achieve a power of 0.8. Due to the preliminary nature of our work, we focused our investigations on healthy participants with no known impairment in upper limb motion and normal or corrected-to-normal vision, similar to prior related work (Elangovan et al., 2017; Elangovan et al., 2018; Wang et al., 2021). After the study participants were compensated with a $\$30$ gift card from an online retailer. The study was approved by the University of Minnesota Institutional Review Board (IRB).

3.3 Proprioceptive Measurements

The goal of sensorimotor learning is to improve people’s ability ability to perceive and precisely control their physical movements. To measure the sensorimotor learning outcomes, we used the same instruments as in (Elangovan et al., 2017; Elangovan et al., 2018; Wang et al., 2021) were adopted, which consist of a proprioceptive acuity test and a motor performance test.

The proprioceptive acuity test measured the sensitivity with which a participant can discriminate the position of their wrist, i.e., the proprioceptive threshold $(PT)$. Across 30 different trials, the participant’s dominant hand was passively moved from a starting angle, first to one one angle and back, and then to a second angle and back. One of the two angles always represented a 15° displacement because that was the maximum rotation required in the training game. The other angle was always less than 15°. The participant was asked to indicate verbally which displacement was greater. A correct response would cause the difference between the two angles to decrease for the next trial, and an incorrect response would cause it to increase. The final measurement of proprioceptive acuity was computed using the psi-marginal method (Prins, 2013).

The motor performance test measured the accuracy with which a participant could actively replicate a passively demonstrated wrist pose, called the movement accuracy error $(MAE)$. Lower $MAE$ corresponds to better motor performance. To obtain wrist $MAE$, the wrist robot rotated the participant’s wrist to a flexion angle of 10°, held that pose for 2s and then returned to the starting position. subsequently, participants were instructed to actively rotate their wrist to match the angle they just experienced. When they felt they had assumed the correct position, they held still and gave a verbal signal. The test consisted of 10 trials and $MAE$ was computed as the mean of the errors across all trials.

3.4 Procedure

After obtaining informed consent, we collected baseline (pre-training) measurements of participants’ proprioceptive threshold $(PT)$ and movement accuracy error $(MAE)$. After this, participants performed the Proprioceptive Training task, which consisted of playing the airplane flying game for 20–30 min. They then filled out several short surveys regarding their user experience.

After a 10-min break, we asked participants to complete the Proprioceptive Measurements again. Following this, participants filled out the a demographics questionnaire.

Last came our behavioral test of participants’ gameplay enjoyment. We told participants that they would need to wait for an additional 5 min for us to “upload the data” from the experiment, and that they could feel free to continue playing the game while they waited. We then left the area, but surreptitiously monitored whether the participant chose to play the game for their own pleasure. Finally, we debriefed the participants and concluded the study. See Figure 3 for a flowchart of the study procedure.

Figure 3

Figure 3. Flowchart of the study procedure.

3.5 Outcome measurements

We quantified proprioceptive improvement as the relative difference $(prescore-postscore)/prescore$ for both $PT$ and $MAE$ was measured pre and post training. We also assessed participants’ in-game performance improvement by assessing the change in their in-game scores between successive levels.

We used the raw “NASA Task Load Index” (NASA TLX) survey (Hart and Staveland, 1988) to assess perceived effort and a customized usability survey to assess enjoyment and awareness of assistance on a five-point-likert scale. Specifically, enjoyment was measured as the extent of agreement with the statement “I enjoyed playing this game.” and awareness was measured with another three statements: (1) “Sometimes I felt that there were unseen forces affecting how the plane flew.” which aimed to capture participants’ awareness of manipulation; (2) “Sometimes I felt as if the plane drifted off-course for no reason.” which was intended as a decoy question; and (3) “Sometimes I felt as if the plane was staying on course all by itself.” which sought to assess the awareness of assistance. The usability survey is attached in Supplementary Material. Persistence (the number of participants who chose to keep playing the game for their own pleasure while waiting for the data upload) and total in-game score were also recorded as secondary measures.

We also quantified the extent of users’ reliance on the assistance to check if they were reducing their own effort over time, as it would be a concern if, in the presence of assistance, people developed a tendency to stop trying and just let the game play itself. We used different assistance metrics for the Attractor and Tunnel conditions due to the difference in their mechanisms. For the tunnel assistance, reliance was quantified as the proportion of the time that assistance was active during the total time spent in a level. For the attractor assistance, which was always active, reliance on positional and rotational assistance was quantified as the total distance and the total angle manipulated by the assistance in each level.

3.6 Hypotheses

We hypothesized that proprioceptive learning outcomes would be facilitated to a greater extent when the assistance preserves a deterministic mapping from physical movement to visual feedback over a longer period of time. We also hypothesized that in-game performance would show greater improvement when users are assisted less, because there would be more opportunity to learn. At the same time, we hypothesized that users would have a better subjective experience, stronger in-game performance and greater persistence in gameplay when using a mode of assistance that helped them perform consistently better in a surreptitious way. Finally, we aimed to validate that visual assistance is not counterproductive to the elicitation of effort by demonstrating that users are not increasing their reliance on the assistance over time. These hypotheses led to the following predictions:

• H1: Use of tunnel assistance will be associated with superior learning outcomes compared to attractor assistance, specifically in terms of greater proprioceptive improvement (H1.A) and greater in-game improvement (H1.B).

• H2: Use of attractor assistance will be associated with better user experience compared to tunnel assistance, specifically in terms of more positive usability survey responses (H2.A), higher in-game scores (H2.B) and more frequent persistence (H2.C) in gameplay beyond the required amount.

• H3: In both conditions, user reliance on assistance will decrease over time.

4 Results

For all of our analyses, we used parametric tests (T-tests) when Shaprio-Wilk tests showed that the data were normal and non-parametric tests (Mann-Whitney, Wilcoxon) when the assumption of normality was violated.

4.1 Proprioceptive and in-game improvement

Proprioceptive improvement is visualized in Figure 4a. A Mann-Whiteney Test showed no significant difference in relative improvement in $PT$ between the two conditions $(U=89.0,\hspace{0.17em}p=0.34)$, and an independent T-test showed no significant difference in $MAE$ improvement $(t=0.63,\hspace{0.17em}p=0.53)$.

Figure 4

Figure 4. (a) The relative post-training performance improvement of proprioceptive threshold (PT) and movement accuracy error (MAE) in the two assistance conditions. Values above zero correspond to improvement. Pre-post training performance comparison of (b) $PT$ and (C) $MAE$ for each participant. Smaller values correspond to better performance. (a) Relative Improvement in proprioceptive threshold and movement accuracy (b) Pre-post training performance of proprioceptive threshold (c) Pre-post training performance of movement accuracy.

Figures 4b,c show the pre-post statistics of $PT$ and $MAE$ for each participant. Paired T-tests showed no statistically significant improvement in $PT$ after training in either the attractor $(t=0.58,\hspace{0.17em}p=0.57)$ or tunnel conditions $(t=-0.27,\hspace{0.17em}p=0.79)$. A Wilcoxon signed test also showed no significant improvement in $MAE$ after training in the attractor condition $(W=30.0,\hspace{0.17em}p=0.52)$. However, a paired T-test indicated a significant increase in $MAE$ after training in the tunnel condition $(t=-2.49,\hspace{0.17em}p=0.03)$, corresponding to a worsening movement accuracy. In both conditions, we found that post-training $MAE$ was strongly positively correlated with pre-training $MAE$. $PT$ on the other hand showed a strong tendency to regress towards the mean, i.e., users who started out with the poorest scores improved while those who started out with the highest scores showed a decrease. Overall, the analyses of the proprioceptive measures do not support H1.A.

To compare In-game improvement between our two conditions, we tested the parallelism of the nonparametric ANCOVA and found $h=1.64759,p<0.001$, which rejects the null hypothesis that the two curves are parallel. In Figure 5 it can be observed that in-game scores increased more steeply in the tunnel condition, which is consistent with H1.B. However, this finding should be interpreted with caution because of the ceiling effects that can be seen in the in-game scores in the attractor condition, i.e., because those scores were high from the beginning, there was little room for further improvement.

Figure 5

Figure 5. Test of parallelism in the in-game improvement between the two assistance. Overlapping samples are illustrated with increased opacity.

4.2 Factors affecting proprioceptive learning

The histograms of participants’ responses to the three awareness questions are shown in Figure 6. The responses from “strongly disagree” to “strongly agree” were mapped to one through five for the data analyses. Mann Whiteney U test found no significant difference between conditions for any of the three responses: $U=74.0,\hspace{0.17em}p=0.93$ for “unseen forces”, $U=78.5,\hspace{0.17em}p=0.69$ for “all by itself”, and $U=58.0,\hspace{0.17em}p=0.39$ for “drifting off-course”.

Figure 6

Figure 6. Participants’ response to the exit survey questions: “Sometimes I felt that there were unseen forces affecting how the plane flew”, “Sometimes I felt as if the plane drifted off-course for no reason.“, and “Sometimes I felt as if the plane was staying on course all by itself”.

Few participants in either condition indicated a strong awareness of the presence of visual assistance. However, a Pearson correlation analysis showed a significant negative correlation between awareness of assistance and $MAE$ improvement in the attractor condition $(p=0.04,r=-0.61)$, as well as a negative correlation between awareness of assistance and $PT$ improvement $(p=0.10,r=-0.50)$, although the significance of the latter did not reach the 0.05 threshold. This indicates that, in the attractor condition, users who were more likely to report the presence of attractor assistance were less likely to experience proprioceptive learning after gameplay. See Figure 7 for an illustration of this relationship.

Figure 7

Figure 7. Correlation between participant’s awareness of assistance and their relative improvement in (a) proprioceptive threshold and (b) movement accuracy. Overlapping samples are illustrated with increased opacity. (a) Correlation between assistance awareness and proprioceptive threshold improvement (b) Correlation between assistance awareness and movement accuracy improvement.

We also found a significant positive correlation in the tunnel condition between ratings of mental exhaustion on the NASA TLX and a sense of the presence of unseen forces affecting the airplane’s behavior $(p=0.01,\hspace{0.17em}r=0.70)$ and a perception of the airplane drifting off course $(p<0.01,\hspace{0.17em}r=0.82)$, as well as a significant positive correlation between sensing the presence of unseen forces and perceiving the plane to be drifting off course $(p=0.01,\hspace{0.17em}r=0.69)$. This suggests an overall negative impact of the invisible tunnel assistance.

4.3 In-game scores and usability

Figure 8 gives a statistical summary of the an illustration of usability, enjoyment and in-game performance scores. Mean overall NASA TLX scores were $51.8\pm 13.4$ in the attractor condition, and $57.2\pm 9.8$ in the tunnel condition, where a smaller value corresponds to lower perceived workload. Enjoyment ratings (higher is better) were $4.91\pm 0.3$ in the attractor condition, and $4.50\pm 0.5$ in the tunnel condition, after the removal of a single outlier with a score of two in the attractor condition. Average total in-game scores (out of a maximum possible 900) were $876.9\pm 30.5$ in the attractor condition, and $778.7\pm 57.3$ in the tunnel condition.

Figure 8

Figure 8. Statistical summary of participants’ (a) Total in-game scores, (b) NASA TLX scores, (c) and histogram of responses to “I enjoyed playing this game” in the two assistance conditions (a) Total in-game scores (b) NASA TLX Scores (c) “I enjoyed playing this game”.

A Mann-Whitney U Test showed significant differences in the total in-game scores $(U=134.0,p<0.001)$ and enjoyment $(U=93.0,\hspace{0.17em}p=0.04)$, between the two assistance conditions, with better in-game performance and enjoyment with the attractor assistance. These results support H2.A and H2.B. Despite the difference, the in-game performance and enjoyment were high in both conditions. The difference in TLX scores was not significant according to an independent T-Test $(t=-1.13,\hspace{0.17em}p=0.27)$.

Only one participant in each condition chose to play the game while waiting for their data to be uploaded, so there was no difference in persistence between the two assistance conditions. The result does not support H2.C. During debriefing, most participants reported that fatigue was the main reason why they did not want to keep playing, even though they did enjoy the game.

4.4 Reliance on assistance

We performed a Mann-Kendall test for each participant to see if there was an increasing/decreasing trend in reliance on assistance over the training period. The results are shown in Table 2. For the majority of the participants, there was either no trend in reliance or the trend was decreasing. This result supports H3.

Table 2

Table 2. Number of participants with decreasing/increasing or no trend in their reliance on the assistance.

5 Discussion

The most notable finding from our study is the failure to see proprioceptive improvement after training in either of our visual assistance conditions. Given that prior related studies (Elangovan et al., 2017; 2018; Wang et al., 2021) had consistently observed some proprioceptive learning after a similar training process, we feel that the most likely explanation for this finding is that both the attractor and tunnel assistance, as implemented, interfered with motor learning. While we would need to do a follow-up study directly comparing proprioceptive learning after training with and without visual assistance to robustly confirm this hypothesis, we advise in the meantime that the use of visual-only assistance in proprioceptive training for stroke rehabilitation should be treated with caution. We had hypothesized that an intermittent assistance model like our tunnel method could better support proprioceptive training by preserving a consistent mapping from motor input to visual feedback over a greater period of time, but our data show that this hope was not borne out. Given that the tunnel condition was associated with less enjoyment, and a higher awareness of this assistance was associated with stronger perception of the plane “drifting off-course” as well as worsening movement accuracy error, it is possible that the experience of encountering invisible guardrails was non-intuitive and therefore unhelpful to the proprioceptive training. In future training development efforts, it might be better to explicitly show the guardrails rather than rendering them invisibly, as well as exploring other possible approaches to providing assistance that are less likely to interfere with the proprioceptive learning process.

Both assistance methods resulted in satisfactory in-game performance and high subjective enjoyment, successfully fulfilling their assistive purpose, which confirms the positive findings from prior research using visual assistance (Dukes et al., 2013; Ferdous et al., 2022; Chowdhury et al., 2019). However, neither method led to improvements in proprioception. This result highlights an important implication: relying solely on task performance and subjective measures, as commonly done in prior studies, may be insufficient for evaluating the effectiveness of visual assistance techniques in stroke rehabilitation or motor training. It is essential to include motor skill improvement as a key evaluation metric.

Our failure to elicit proprioceptive learning might also be explained by an overall dampening in motor performance after gameplay due to fatigue. The wristbot was heavy and required a lot of physical effort to manipulate. Because of this, few participants opted to continue playing after the end of the experiment. In the attractor condition, users who reported higher TLX physical demand had worse in-game performance and lower post-game PT. However, in the tunnel condition, higher TLX physical demand was associated with a greater improvement in gameplay performance, which is inconsistent with a negative impact of exhaustion.

It is also possible that the game we designed was somehow fundamentally ill-suited to supporting improved sensorimotor learning, independent of the presence of visual assistance. Unfortunately this possibility cannot be ruled out, as we did not include a control condition in our study. However, given the consistency with which motor improvements have been observed in the past after similar periods of engagement with a wide variety of similarly-designed games, we find this unlikely. We also note that there is limited practical value in considering proprioceptive training without some form of assistance. Nevertheless, we recognize that more robust followup work, including a control condition of no assistance, is needed to definitively establish the potentially deleterious impact to sensorimotor learning from providing visual assistance that surreptitiously varies the mapping between physical action and visual feedback in a constantly-changing way.

6 Limitations

Because the experiment did not contain a control condition, we cannot be certain that the lack of motor learning outcomes can be wholly attributed to the use of visual assistance. While we had intended that our experiment would compare a “high” visual assistance condition to a “low” visual assistance condition, our data suggest that our two visual assistance conditions may have each been detrimental in its own way, but we cannot prove that with the data we have and further studies would be needed to explore that possibility.

Also, due to the preliminary nature of our investigations and in accordance with prior norms for working with vulnerable populations, we tested both of our systems on healthy participants. It is possible that the introduction of visual assistance might impact such populations differently. Experiments with vulnerable/protected populations will be necessary in the future, after further preliminary hypothesis testing with healthy participants.

7 Conclusion

In this paper, we set out to investigate the extent to which visual-only assistance, i.e., manipulating the mapping from motor input to visual feedback towards greater in-game success, might potentially hinder sensorimotor learning during gamified proprioceptive training. We developed a motor training game and compared two different implementations of visual assistance: one that operated continuously and used a constantly-varying mapping to persistently manipulate the player’s visual feedback toward showing greater success in gameplay, and one that used an invisible tunnel paradigm to prevent the player from straying too far off course, thereby preserving an unmodified mapping from motor input to visual feedback throughout a greater proportion of the training session. Our hypothesis was that the “tunnel” type of assistance would better support proprioceptive learning, despite providing a potentially inferior user experience. However, our results showed a failure to achieve significant proprioceptive improvement after training with either method. Our results suggest that the use of visual assistance during motor training should be treated with caution, and highlights the need for further investigations to explore the potential impact of visual assistance in proprioceptive training for stroke rehabilitation.

Data availability statement

The datasets presented in this article are not readily available because according to the IRB protocol the dataset will not be made public. Requests to access the datasets should be directed to aHVhbjIwNzZAdW1uLmVkdQ==.

Ethics statement

The studies involving humans were approved by University of Minnesota Institutional Review Board. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

YH: Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Software, Visualization, Writing – original draft. NE: Conceptualization, Methodology, Resources, Writing – review and editing. JK: Conceptualization, Resources, Supervision, Writing – review and editing. VI: Conceptualization, Formal Analysis, Methodology, Resources, Supervision, Validation, Writing – review and editing.

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. This project was supported in part by the Minnesota Robotics Institute (MnRI) and by a grant from the National Science Foundation (I-Corps: Robotic system for physical rehabilitation, Award Number: 1745306).

Acknowledgments

The authors would like to acknowledge the Italian Institute of Technology for providing the wrist robot.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Generative AI statement

The author(s) declare that no Generative AI was used in the creation of this manuscript.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2025.1526567/full#supplementary-material

References

Aman, J. E., Elangovan, N., Yeh, I.-L., and Konczak, J. (2015). The effectiveness of proprioceptive training for improving motor function: a systematic review. Front. Hum. Neurosci. 8, 1075. doi:10.3389/fnhum.2014.01075

PubMed Abstract | CrossRef Full Text | Google Scholar

Baranyi, R., Reisecker, F., Lederer, N., Gobber, M., and Grechenig, T. (2014). “Wristdroid-a serious game to support and motivate patients throughout their wrist rehabilitation,” in 2014 IEEE conference on biomedical engineering and sciences (IECBES) (IEEE), 786–791.

CrossRef Full Text | Google Scholar

Basteris, A., Nijenhuis, S. M., Stienen, A. H., Buurke, J. H., Prange, G. B., and Amirabdollahian, F. (2014). Training modalities in robot-mediated upper limb rehabilitation in stroke: a framework for classification based on a systematic review. J. neuroengineering rehabilitation 11, 111–115. doi:10.1186/1743-0003-11-111

PubMed Abstract | CrossRef Full Text | Google Scholar

Borboni, A., Mor, M., and Faglia, R. (2016). Gloreha—hand robotic rehabilitation: design, mechanical model, and experiments. J. Dyn. Syst. Meas. Control 138, 138. doi:10.1115/1.4033831

CrossRef Full Text | Google Scholar

Burke, J. W., McNeill, M., Charles, D. K., Morrow, P. J., Crosbie, J. H., and McDonough, S. M. (2009). Optimising engagement for stroke rehabilitation using serious games. Vis. Comput. 25, 1085–1099. doi:10.1007/s00371-009-0387-4

CrossRef Full Text | Google Scholar

Bütefisch, C., Hummelsheim, H., Denzler, P., and Mauritz, K.-H. (1995). Repetitive training of isolated movements improves the outcome of motor rehabilitation of the centrally paretic hand. J. neurological Sci. 130, 59–68. doi:10.1016/0022-510x(95)00003-k

PubMed Abstract | CrossRef Full Text | Google Scholar

Carey, L. M., Oke, L. E., and Matyas, T. A. (1996). Impaired limb position sense after stroke: a quantitative test for clinical use. Archives Phys. Med. rehabilitation 77, 1271–1278. doi:10.1016/s0003-9993(96)90192-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Casadio, M., Giannoni, P., Morasso, P., and Sanguineti, V. (2009). A proof of concept study for the integration of robot therapy with physiotherapy in the treatment of stroke patients. Clin. Rehabil. 23, 217–228. doi:10.1177/0269215508096759

PubMed Abstract | CrossRef Full Text | Google Scholar

Chang, J.-J., Tung, W.-L., Wu, W.-L., Huang, M.-H., and Su, F.-C. (2007). Effects of robot-aided bilateral force-induced isokinetic arm training combined with conventional rehabilitation on arm motor function in patients with chronic stroke. Archives Phys. Med. rehabilitation 88, 1332–1338. doi:10.1016/j.apmr.2007.07.016

PubMed Abstract | CrossRef Full Text | Google Scholar

Chowdhury, T., Ferdous, S., Peck, T., and Quarles, J. (2019). “Virtual ability simulation: applying rotational gain to the leg to increase confidence during physical rehabilitation,” in ICAT-EGVE 2019-international conference on artificial reality and telexistence and eurographics symposium on virtual environments.

Google Scholar

Cuppone, A. V., Semprini, M., and Konczak, J. (2018). Consolidation of human somatosensory memory during motor learning. Behav. brain Res. 347, 184–192. doi:10.1016/j.bbr.2018.03.013

PubMed Abstract | CrossRef Full Text | Google Scholar

Demeco, A., Zola, L., Frizziero, A., Martini, C., Palumbo, A., Foresti, R., et al. (2023). Immersive virtual reality in post-stroke rehabilitation: a systematic review. Sensors 23, 1712. doi:10.3390/s23031712

PubMed Abstract | CrossRef Full Text | Google Scholar

Dishman, R. K., and Ickes, W. (1981). Self-motivation and adherence to therapeutic exercise. J. Behav. Med. 4, 421–438. doi:10.1007/bf00846151

PubMed Abstract | CrossRef Full Text | Google Scholar

Dobkin, B. H. (2004). Strategies for stroke rehabilitation. Lancet Neurology 3, 528–536. doi:10.1016/s1474-4422(04)00851-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Dukes, P. S., Hayes, A., Hodges, L. F., and Woodbury, M. (2013). “Punching ducks for post-stroke neurorehabilitation: system design and initial exploratory feasibility study,” in 2013 IEEE symposium on 3D user interfaces (3DUI) (IEEE), 47–54.

CrossRef Full Text | Google Scholar

Elangovan, N., Cappello, L., Masia, L., Aman, J., and Konczak, J. (2017). A robot-aided visuo-motor training that improves proprioception and spatial accuracy of untrained movement. Sci. Rep. 7 (1), 17054. doi:10.1038/s41598-017-16704-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Elangovan, N., Tuite, P. J., and Konczak, J. (2018). Somatosensory training improves proprioception and untrained motor function in Parkinson’s disease. Front. neurology 9, 1053. doi:10.3389/fneur.2018.01053

PubMed Abstract | CrossRef Full Text | Google Scholar

Ferdous, S. M. S., Michael, A., Chowdhury, T. I., and Quarles, J. (2022). “Use of scaling to improve reach in virtual reality for people with Parkinson’s disease,” in 2022 IEEE 10th international conference on serious games and applications for health (SeGAH). IEEE, 1–7.

CrossRef Full Text | Google Scholar

Frees, S., and Kessler, G. D. (2005). “Precise and rapid interaction through scaled manipulation in immersive virtual environments,” in IEEE proceedings. VR 2005. Virtual reality, 2005 (IEEE), 99–106.

Google Scholar

Frees, S., Kessler, G. D., and Kay, E. (2007). Prism interaction for enhancing control in immersive virtual environments. ACM Trans. Computer-Human Interact. (TOCHI) 14, 2–es. doi:10.1145/1229855.1229857

CrossRef Full Text | Google Scholar

Hao, J., Xie, H., Harp, K., Chen, Z., and Siu, K.-C. (2022). Effects of virtual reality intervention on neural plasticity in stroke rehabilitation: a systematic review. Archives Phys. Med. Rehabilitation 103, 523–541. doi:10.1016/j.apmr.2021.06.024

PubMed Abstract | CrossRef Full Text | Google Scholar

Hart, S. G., and Staveland, L. E. (1988). Development of nasa-tlx (task load index): results of empirical and theoretical research. Adv. Psychol. 52, 139–183. doi:10.1016/s0166-4115(08)62386-9

CrossRef Full Text | Google Scholar

Hussain, S., Jamwal, P. K., Van Vliet, P., and Ghayesh, M. H. (2020). State-of-the-art robotic devices for wrist rehabilitation: design and control aspects. IEEE Trans. Human-Machine Syst. 50, 361–372. doi:10.1109/thms.2020.2976905

CrossRef Full Text | Google Scholar

Karime, A., Eid, M., Alja’Am, J. M., El Saddik, A., and Gueaieb, W. (2013). A fuzzy-based adaptive rehabilitation framework for home-based wrist training. IEEE Trans. Instrum. Meas. 63, 135–144. doi:10.1109/tim.2013.2277536

CrossRef Full Text | Google Scholar

Kawahira, K., Shimodozono, M., Ogata, A., and Tanaka, N. (2004). Addition of intensive repetition of facilitation exercise to multidisciplinary rehabilitation promotes motor functional recovery of the hemiplegic lower limb. J. Rehabilitation Med. 36, 159–164. doi:10.1080/16501970410029753

PubMed Abstract | CrossRef Full Text | Google Scholar

Kenzie, J. M., Semrau, J. A., Findlater, S. E., Herter, T. M., Hill, M. D., Scott, S. H., et al. (2014). Anatomical correlates of proprioceptive impairments following acute stroke: a case series. J. neurological Sci. 342, 52–61. doi:10.1016/j.jns.2014.04.025

PubMed Abstract | CrossRef Full Text | Google Scholar

Khokale, R., Mathew, G. S., Ahmed, S., Maheen, S., Fawad, M., Bandaru, P., et al. (2023). Virtual and augmented reality in post-stroke rehabilitation: a narrative review. Cureus 15, e37559. doi:10.7759/cureus.37559

PubMed Abstract | CrossRef Full Text | Google Scholar

Lambercy, O., Dovat, L., Yun, H., Wee, S. K., Kuah, C. W., Chua, K. S., et al. (2011). Effects of a robot-assisted training of grasp and pronation/supination in chronic stroke: a pilot study. J. neuroengineering rehabilitation 8, 63–12. doi:10.1186/1743-0003-8-63

PubMed Abstract | CrossRef Full Text | Google Scholar

Langhorne, P., Bernhardt, J., and Kwakkel, G. (2011). Stroke rehabilitation. Lancet 377, 1693–1702. doi:10.1016/s0140-6736(11)60325-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Laver, K. E., Lange, B., George, S., Deutsch, J. E., Saposnik, G., and Crotty, M. (2017). Virtual reality for stroke rehabilitation. Cochrane database Syst. Rev. 2018. doi:10.1002/14651858.cd008349.pub4

PubMed Abstract | CrossRef Full Text | Google Scholar

Marini, F., Hughes, C. M., Squeri, V., Doglio, L., Moretti, P., Morasso, P., et al. (2017). Robotic wrist training after stroke: adaptive modulation of assistance in pediatric rehabilitation. Robotics Aut. Syst. 91, 169–178. doi:10.1016/j.robot.2017.01.006

CrossRef Full Text | Google Scholar

Masia, L., Casadio, M., Sandini, G., and Morasso, P. (2009). Eye-hand coordination during dynamic visuomotor rotations. PloS one 4, e7004. doi:10.1371/journal.pone.0007004

PubMed Abstract | CrossRef Full Text | Google Scholar

Masiero, S., Celia, A., Rosati, G., and Armani, M. (2007). Robotic-assisted rehabilitation of the upper limb after acute stroke. Archives Phys. Med. rehabilitation 88, 142–149. doi:10.1016/j.apmr.2006.10.032

PubMed Abstract | CrossRef Full Text | Google Scholar

Montagner, A., Frisoli, A., Borelli, L., Procopio, C., Bergamasco, M., Carboncini, M. C., et al. (2007). “A pilot clinical study on robotic assisted rehabilitation in vr with an arm exoskeleton device,” in 2007 virtual rehabilitation (IEEE), 57–64.

CrossRef Full Text | Google Scholar

Mubin, O., Alnajjar, F., Al Mahmud, A., Jishtu, N., and Alsinglawi, B. (2020). Exploring serious games for stroke rehabilitation: a scoping review. Disabil. Rehabilitation Assistive Technol. 17, 159–165. doi:10.1080/17483107.2020.1768309

PubMed Abstract | CrossRef Full Text | Google Scholar

Murakami, H., Fiscale, V., Soccini, A. M., and Inamura, T. (2024). “White lies in virtual reality: impact on enjoyment and fatigue,” in Proceedings of the 30th ACM symposium on virtual reality software and technology, 1–2.

Google Scholar

Ostry, D. J., and Gribble, P. L. (2016). Sensory plasticity in human motor learning. Trends Neurosci. 39, 114–123. doi:10.1016/j.tins.2015.12.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Poupyrev, I., Billinghurst, M., Weghorst, S., and Ichikawa, T. (1996). “The go-go interaction technique: non-linear mapping for direct manipulation in vr,” in Proceedings of the 9th annual ACM symposium on User interface software and technology, 79–80.

Google Scholar

Prins, N. (2013). The psi-marginal adaptive method: how to give nuisance parameters the attention they deserve (no more, no less). J. Vis. 13, 3. doi:10.1167/13.7.3

PubMed Abstract | CrossRef Full Text | Google Scholar

Rand, D., Givon, N., Weingarden, H., Nota, A., and Zeilig, G. (2014). Eliciting upper extremity purposeful movements using video games: a comparison with traditional therapy for stroke rehabilitation. Neurorehabilitation neural repair 28, 733–739. doi:10.1177/1545968314521008

PubMed Abstract | CrossRef Full Text | Google Scholar

Ross, R. E., Hart, E., Williams, E. R., Gregory, C. M., Flume, P. A., Mingora, C. M., et al. (2023). Combined aerobic exercise and virtual reality-based upper extremity rehabilitation intervention for chronic stroke: feasibility and preliminary effects on physical function and quality of life. Archives Rehabilitation Res. Clin. Transl. 5, 100244. doi:10.1016/j.arrct.2022.100244

PubMed Abstract | CrossRef Full Text | Google Scholar

Sluijs, E. M., Kok, G. J., and Van der Zee, J. (1993). Correlates of exercise compliance in physical therapy. Phys. Ther. 73, 771–782. doi:10.1093/ptj/73.11.771

PubMed Abstract | CrossRef Full Text | Google Scholar

Squeri, V., Masia, L., Giannoni, P., Sandini, G., and Morasso, P. (2013). Wrist rehabilitation in chronic stroke patients by means of adaptive, progressive robot-aided therapy. IEEE Trans. neural Syst. rehabilitation Eng. 22, 312–325. doi:10.1109/tnsre.2013.2250521

PubMed Abstract | CrossRef Full Text | Google Scholar

Tacchino, G., Gandolla, M., Coelli, S., Barbieri, R., Pedrocchi, A., and Bianchi, A. M. (2016). Eeg analysis during active and assisted repetitive movements: evidence for differences in neural engagement. IEEE Trans. Neural Syst. Rehabilitation Eng. 25, 761–771. doi:10.1109/tnsre.2016.2597157

PubMed Abstract | CrossRef Full Text | Google Scholar

Veerbeek, J. M., Langbroek-Amersfoort, A. C., Van Wegen, E. E., Meskers, C. G., and Kwakkel, G. (2017). Effects of robot-assisted therapy for the upper limb after stroke: a systematic review and meta-analysis. Neurorehabilitation neural repair 31, 107–121. doi:10.1177/1545968316666957

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, Y., Zhu, H., Elangovan, N., Cappello, L., Sandini, G., Masia, L., et al. (2021). A robot-aided visuomotor wrist training induces gains in proprioceptive and movement accuracy in the contralateral wrist. Sci. Rep. 11, 5281. doi:10.1038/s41598-021-84767-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Wentzel, J., d’Eon, G., and Vogel, D. (2020). “Improving virtual reality ergonomics through reach-bounded non-linear input amplification,” in Proceedings of the 2020 CHI conference on human factors in computing systems, 1–12.

Google Scholar