Visual assistance may impede sensorimotor learning during gamified rehabilitation exercises

Abstract

Regular exercise is critical to post-stroke recovery, but can be frustrating and difficult for patients with limited motor capabilities. Robotic assistance devices are being used to support this process, but due to cost and accessibility concerns some researchers have drawn inspiration from virtual reality and proposed the use of what we call “visual-only assistance”, in which a patient’s physical movements are mapped into game-oriented visual feedback modified towards greater success in the gameplay objectives. Our concern is that the motivational benefits it provides may come at the cost of reduced sensorimotor learning, which could ultimately be counterproductive to the recovery process. To explore these concerns, we conducted a between-subjects study with 24 participants to examine how two types of visual-only assistance affect short-term proprioceptive skill learning in a motor training game involving airplane steering with wrist rotation. One group experienced “attractor assistance”, in which the airplane was continuously displaced toward an ideal position and orientation. The other experienced “tunnel assistance”, in which direct user control was maintained unless straying too far, triggering an invisible barrier. We hypothesized that motor learning would be more impeded with attractor assistance due to the constant mapping variation between physical movements and visual feedback, but our experiment found that motor learning did not materialize in either condition, suggesting that substituting intermittent visual assistance for constant visual assistance is insufficient to guarantee superior motor learning outcomes and that, pending further investigation, the use of visual assistance for proprioceptive training should continue to be regarded with caution.

1 Introduction

Motor rehabilitation is essential for the functional recovery and independence in people after a stroke (Langhorne et al., 2011; Dobkin, 2004). Research has shown that the repetition of movements can effectively assist in the re-establishment of the motor control pathway (Bütefisch et al., 1995; Kawahira et al., 2004) through reinforcement of sensorimotor feedback and proprioception, which makes it a core component of motor rehabilitation therapies.

However, the plain repetition of movements that are out of context can be tedious and unengaging. Many existing studies suggest that nonadherence to rehabilitation therapy is one of the main factors limiting the effectiveness of rehabilitation training (Dishman and Ickes, 1981; Sluijs et al., 1993). Therefore, there has been a growing trend to incorporate gamification elements to rehabilitation therapies to improve engagement and adherence (Burke et al., 2009; Mubin et al., 2020), usually with adjustable difficulty to accommodate the varying ability of stroke patients (Karime et al., 2013). Nevertheless, acute stroke patients can still find it too challenging to complete the gamified tasks unassisted due to their very limited motor capabilities, and making the tasks too simple can defeat the purpose of an engaging and motivating experience.

To account for this, dedicated robotic devices are designed and developed to assist the patients in executing the movements they intend to make by sensing and magnifying their own efforts towards a static or dynamic goal state, defined in the context of an appropriately designed gamified task (Marini et al., 2017; Veerbeek et al., 2017; Hussain et al., 2020). The use of robotic assistance is proved to be effective and reliable in facilitating motor recovery (Veerbeek et al., 2017; Hussain et al., 2020) but many complicated factors remain to be addressed, most notably the access to these devices being limited to onsite therapy sessions.

Since regular motor exercise is key to motor rehabilitation, researchers have been considering the deployment of gamified recovery support software (Rand et al., 2014; Baranyi et al., 2014), including games that provide what we call “visual assistance” as a potential compliment to the use of robotic assistance so that patients can perform motor exercises from home while allowing the therapists to track gameplay duration (adherence) and performance (skills improvement) remotely to support them in optimally steering the recovery process (Dukes et al., 2013; Ferdous et al., 2022). With visual assistance, a patient’s physical movements are mapped into game-oriented visual feedback in a way that is either overtly or covertly modified towards greater success in the gameplay objectives, which achieves motivational benefits similar to robotic assistance but without requiring actual mechanical devices.

While such visual manipulation can be implemented in different forms and is commonly used in virtual reality (VR) applications for various purposes (Poupyrev et al., 1996; Frees and Kessler, 2005; Wentzel et al., 2020) to improve usability, in rehabilitation exercises the ultimate goal is sensorimotor learning, which requires the patients to process sensory information and learn how to control their movements based on the sensory feedback. Our concern is that a method of visual assistance that applies a constantly-varying manipulation to the mapping between a user’s physical movement and the visual feedback they receive about that movement could be counter-productive to sensorimotor learning, because the same movement of the patient can lead to different outcomes depending on what the assistance is doing, which, in spite of motivational benefits, makes it harder for them to appreciate how they are controlling their physical movements. Therefore, in this study we hypothesize that a visual assistance method that preserves the movement-to-feedback mapping to greater extent can better accomplish the goal of avoiding patient frustration while enabling better motor training outcomes by affording more robust sensorimotor learning opportunities.

2 Related work

2.1 Proprioceptive training and sensorimotor rehabilitaion

Neurological conditions such as stroke can cause motor impairment as well as proprioceptive dysfunction (Carey et al., 1996; Kenzie et al., 2014). It is well-established that training of repetitive movements with volitional effort is an effective intervention to aid in stroke rehabilitation (Bütefisch et al., 1995; Kawahira et al., 2004; Tacchino et al., 2016), and sensorimotor training can improve proprioceptive functions, which further leads to improved motor performance (Elangovan et al., 2017; Elangovan et al., 2018). The rationale is that the relevant neuro-connections can be reorganized through the proprioceptive training and the brain can relearn the mapping between the motor inputs and outcomes. It is worth noting that measurable improvement in proprioceptive performance can not only be observed in stroke patients but also in healthy populations even after short-term training (Aman et al., 2015; Ostry and Gribble, 2016; Cuppone et al., 2018), so as a matter of ethics, researchers avoid using clinical populations for preliminary investigations, where results from healthy participants can be sufficient to inform the next steps in the work (Elangovan et al., 2017; 2018; Wang et al., 2021).

2.2 Robotic assistance for rehabilitation

In cases where the participants are unable to complete the training tasks due to limited motor capability, the standard solution is to assist their movement using exoskeletons or other robotic devices (Veerbeek et al., 2017; Hussain et al., 2020). Many different robotic systems have been developed for this purpose and proved to be effective. Masiero et al. created a simple 3-DOF wire-based robot which uses motors to control the lengths of the wires and pull the patients’ upper limbs to assist their motion (Masiero et al., 2007). Studies have demonstrated that early sensorimotor training using this device can speed up the recovery. Montagner et al. developed a force feedback exoskeleton for the right arm, and achieved positive results in terms of rehabilitation outcomes and user feedback in a pilot study consisting of various motor training tasks (Montagner et al., 2007). A wrist rehabilitation robot developed by Masia et al. can apply torques to either assist or resist the rotation of the wrist and is used in both motor control studies and rehabilitation (Masia et al., 2009). In Borboni et al. (2016), designed a hand rehabilitation glove capable of assisting personalized combinations of finger movements and showed its efficacy with experimental results on 126 stroke patients. However, a major constraint of robotic assistance is that the robotic devices are only available in hospitals or clinics, which significantly limits their accessibility.

2.3 Visual feedback manipulation

In recent years, many elements of VR have also been incorporated into motor training (Laver et al., 2017; Hao et al., 2022; Demeco et al., 2023; Khokale et al., 2023), including the introduction of visual manipulation in an assistive manner to support motor training when robotic assistance is unavailable.

In Virtual Reality, things are not constrained by the laws of physics, and many interaction techniques have been proposed that manipulate the visual feedback of users’ spatial input in a way that assists with the completion of specific tasks in the virtual environment. For example, “go-go” (Poupyrev et al., 1996) and its variants (Wentzel et al., 2020) allow users to work with objects out of reach by scaling up the displayed movements when they exceed of the user’s arm length. Precise and rapid interaction through scaled manipulation (PRISM) is a method that scales down the displayed movement to improve stability when performing fine movements (Frees and Kessler, 2005; Frees et al., 2007). There are also studies that implicitly modify users’ task performance in VR towards greater success, such as in the ball throwing task described in (Murakami et al., 2024), where the trajectory of the ball is pushed towards the target. The methods discussed in these studies share a similar goal to robotic assistance in that they aim to improve engagement and motivation by enabling users to perform certain tasks more easily or accurately than what they are actually capable of.

Some work has also incorporated the idea of visual assistance into the context of stroke rehabilitation and motor training. In the “duck duck punch” rehabilitation game developed by Dukes et al., the movements of stroke patients are visually amplified to account for their limited range of motion (Dukes et al., 2013). Similarly, Ferdous et al. used scaled movement to improve the reach of Parkinson patients in a VR rehabilitation game of smashing balloons (Ferdous et al., 2022). The same idea has also been used in lower-extremity rehabilitation training, such as in (Chowdhury et al., 2019), where rotational gain is applied to patients’ leg movement to assist them in a rehabilitation game involving kicking a ball. However, all the above-mentioned studies focused on task completion and motivational benefits without explicitly considering the introduction of visual assistance affects training outcomes. In Ross et al. (2023), demonstrated that the “duck duck punch game” combined with aerobic exercise can improve the upper-extremity function of stroke patients, but that training was targeting range-of-motion, i.e., how far one can move, rather than sensorimotor learning, i.e., how accurate one can perceive and execute movements. To the best of our knowledge no study has yet specifically considered the impact of visual assistance on the acquisition of sensorimotor skills.

Therefore, in this study we aim to examine the sensorimotor learning in a motor training game when visual assistance is present, and explore whether a variant of visual assistance that introduces manipulation in an “as-needed” fashion could effectively reduce the potential disruption of sensorimotor learning and achieve a superior training outcome compared to traditional visual assistance that is “always on” and employs a constantly varying mapping between action and outcome.

3 Methods

We conducted a between-subject study to compared the proprioceptive training outcomes and user experience between two implementations of visual assistance, one that constantly manipulates the visual feedback towards greater in-game success, and one that becomes active only when the user is doing exceptionally poorly. Based on our concern that visual feedback manipulation may disrupt proprioceptive learning, we hypothesized that an “as-needed” variant of visual assistance might enable better motor training improvement due to the reduced amount of visual manipulation being introduced, but might also potentially sacrifice usability due to the users performing more poorly. We did not consider the need for a control condition in which no assistance was applied, because we were focused on addressing practical situations in which some sort of assistance, either visual or robotic, is needed. However, we recognize in retrospect that including such a condition would have been wise.

3.1 Apparatus and materials

3.1.1 Experimental equipment

We used a wrist rehabilitation robot built by the Italian Institute of Technology (Masia et al., 2009) to track the user’s wrist movements in our study. This device has three degrees of freedom in movement, corresponding to pronation/supination, flexion/extension, and abduction/adduction of the human wrist. To use this device, participants sat on a stool, rested their arm on the device’s armrest, and grabbed the stick of the wrist robot with their dominant hand. In this study, the robot did not provide any mechanical assistance. The reason why we used a wrist robot as the tracker, as opposed to a smartphone or VR controllers was because in a future study we wanted to compare visual and robotic assistance with the hardware as a controlled variable, and this study served as a preliminary study to explore possible visual assistance design. The robot was connected to a desktop computer running Linux Mint (Version 18) and visual feedback was displayed on a monitor that was mounted at eye height on a narrow table approximately 40 cm in front of the participants. See Figure 1 for an illustration of the experimental setup.

FIGURE 1

3.1.2 Motor training game

We developed a motor training game using Unity3D. The game was inspired by air racing, in which the player controls a virtual plane flying forward on its own at constant speed. The plane’s rotation is linked to the user’s wrist rotation. The goals are to steer the plane to fly through all gates and match the rotations visualized as outlines of plane’s rotational pose naturally maps to the 3-DOF of human wrist, and by setting the location of the gates and the orientation of the outlines we can define the required wrist rotations in an intuitive manner. The game has 20 procedurally generated levels, each consisting of 15 gates, where the forward displacement between adjacent gates is 15 units and the horizontal and vertical displacements are randomized within and units, respectively. The target rotation is randomized within [-45°, 45°]. The purpose of the randomization to ensure non-repetitive gameplay, yet consistency in difficulty across levels. The forward speed of the plane is 4.5 units/second, meaning that each level takes 50 s to complete. We also created five scene variations shown every four levels to reduce visual fatigue (see Figure 1).

Whenever the plane reaches a gate, a score is calculated based on the accuracy of its position and orientation with respect to the target outline. Table 1 shows the scoring system. The players were shown how many additional points they earned after passing each gate, and the total score for each level was shown after the level’s completion.

3.1.3 Visual assistance

We implemented visual-only counterparts of two commonly used mechanical assistance schemes (Basteris et al., 2014): 1) The moving attractor (Chang et al., 2007; Lambercy et al., 2011; Squeri et al., 2013), in which an assistive force constantly attracts the end-effector towards a target trajectory and the amount of assistance is modulated by varying a stiffness parameter, e.g., proportional to the distance between the end-effector and the target, and 2) the virtual tunnel (Casadio et al., 2009), in which assistive forces are introduced only after the error exceeds a certain threshold.

To avoid over-assistance, the assisted movement for the attractor is proportional to: 1) the distance from the plane location to the closest point on the trajectory , and 2) the distance in the forward direction from the plane to the next gate . Specifically, the assisted movement increases as the plane deviates farther away from the optimal trajectory i.e., when the player is not doing well, and as the plane approaches the gates, i.e., when there is less time left to adjust. See Figure 2 for a visual illustration. Specifically, the attractive force , i.e., how much the position/rotation of the plane is adjusted towards the target position/rotation in one unit of time is calculated as , where is the distance between adjacent gates, and is the scaling factor. Similarly, for tunnel assistance the threshold of allowable positional and rotational error gets smaller at a specific rate until reaching a minimum value as the plane approaches the gate, allowing more freedom when the gate is far ahead and less room for error when the gate is near. The allowable error threshold at any given distance to the next gate can be calculated as . See Figure 2 for visual illustration of tunnel assistance. In both cases of assistance, the optimal trajectory is displayed as a semi-transparent curve in the game for reference.

FIGURE 2

In order to optimize the comparison between the two assistance methods, we conducted a pilot study with four participants from our research group to tune the parameters of the positional and rotational assistance as they played the game. The participants were guided to adjust the strength of the assistance so that the game felt moderately challenging and rewarding enough that they would be willing to play it for a long time. We asked them to tune one parameter at a time, and they went through each possible permutation of parameter tuning order 3 times with different random starting values. During the process, the participants used their dominant hand to control the airplane’s position and orientation via the wrist robot, and used their non-dominant hand to press the up/down arrow keys on the keyboard to increase/decrease the amount of assistance being provided. For our formal study, we used the median of the final values chosen by the four participants in the pilot study.

3.2 Participants

We recruited 24 participants (13M, 11F) between the ages of 18–34 from our University community, using posters and online platforms. Half (6M, 6F) were randomly assigned to the attractor condition, and half (7M, 5F) to the tunnel condition. While consistent with the number of participants used in prior related work Elangovan et al. (2017), Elangovan et al. (2018) we note that this is significantly less than the 128 participants recommended by the power analysis conducted on G*Power 3.1.9.7 for an independent t-test with a medium expected effect size (0.5) and alpha = 0.05 to achieve a power of 0.8. Due to the preliminary nature of our work, we focused our investigations on healthy participants with no known impairment in upper limb motion and normal or corrected-to-normal vision, similar to prior related work (Elangovan et al., 2017; Elangovan et al., 2018; Wang et al., 2021). After the study participants were compensated with a gift card from an online retailer. The study was approved by the University of Minnesota Institutional Review Board (IRB).

3.3 Proprioceptive Measurements

The goal of sensorimotor learning is to improve people’s ability ability to perceive and precisely control their physical movements. To measure the sensorimotor learning outcomes, we used the same instruments as in (Elangovan et al., 2017; Elangovan et al., 2018; Wang et al., 2021) were adopted, which consist of a proprioceptive acuity test and a motor performance test.

The proprioceptive acuity test measured the sensitivity with which a participant can discriminate the position of their wrist, i.e., the proprioceptive threshold . Across 30 different trials, the participant’s dominant hand was passively moved from a starting angle, first to one one angle and back, and then to a second angle and back. One of the two angles always represented a 15° displacement because that was the maximum rotation required in the training game. The other angle was always less than 15°. The participant was asked to indicate verbally which displacement was greater. A correct response would cause the difference between the two angles to decrease for the next trial, and an incorrect response would cause it to increase. The final measurement of proprioceptive acuity was computed using the psi-marginal method (Prins, 2013).

The motor performance test measured the accuracy with which a participant could actively replicate a passively demonstrated wrist pose, called the movement accuracy error . Lower corresponds to better motor performance. To obtain wrist , the wrist robot rotated the participant’s wrist to a flexion angle of 10°, held that pose for 2s and then returned to the starting position. subsequently, participants were instructed to actively rotate their wrist to match the angle they just experienced. When they felt they had assumed the correct position, they held still and gave a verbal signal. The test consisted of 10 trials and was computed as the mean of the errors across all trials.

3.4 Procedure

After obtaining informed consent, we collected baseline (pre-training) measurements of participants’ proprioceptive threshold and movement accuracy error . After this, participants performed the Proprioceptive Training task, which consisted of playing the airplane flying game for 20–30 min. They then filled out several short surveys regarding their user experience.

After a 10-min break, we asked participants to complete the Proprioceptive Measurements again. Following this, participants filled out the a demographics questionnaire.

Last came our behavioral test of participants’ gameplay enjoyment. We told participants that they would need to wait for an additional 5 min for us to “upload the data” from the experiment, and that they could feel free to continue playing the game while they waited. We then left the area, but surreptitiously monitored whether the participant chose to play the game for their own pleasure. Finally, we debriefed the participants and concluded the study. See Figure 3 for a flowchart of the study procedure.

FIGURE 3

3.5 Outcome measurements

We quantified proprioceptive improvement as the relative difference for both and was measured pre and post training. We also assessed participants’ in-game performance improvement by assessing the change in their in-game scores between successive levels.

We used the raw “NASA Task Load Index” (NASA TLX) survey (Hart and Staveland, 1988) to assess perceived effort and a customized usability survey to assess enjoyment and awareness of assistance on a five-point-likert scale. Specifically, enjoyment was measured as the extent of agreement with the statement “I enjoyed playing this game.” and awareness was measured with another three statements: (1) “Sometimes I felt that there were unseen forces affecting how the plane flew.” which aimed to capture participants’ awareness of manipulation; (2) “Sometimes I felt as if the plane drifted off-course for no reason.” which was intended as a decoy question; and (3) “Sometimes I felt as if the plane was staying on course all by itself.” which sought to assess the awareness of assistance. The usability survey is attached in Supplementary Material. Persistence (the number of participants who chose to keep playing the game for their own pleasure while waiting for the data upload) and total in-game score were also recorded as secondary measures.

We also quantified the extent of users’ reliance on the assistance to check if they were reducing their own effort over time, as it would be a concern if, in the presence of assistance, people developed a tendency to stop trying and just let the game play itself. We used different assistance metrics for the Attractor and Tunnel conditions due to the difference in their mechanisms. For the tunnel assistance, reliance was quantified as the proportion of the time that assistance was active during the total time spent in a level. For the attractor assistance, which was always active, reliance on positional and rotational assistance was quantified as the total distance and the total angle manipulated by the assistance in each level.

3.6 Hypotheses

4 Results

For all of our analyses, we used parametric tests (T-tests) when Shaprio-Wilk tests showed that the data were normal and non-parametric tests (Mann-Whitney, Wilcoxon) when the assumption of normality was violated.

4.1 Proprioceptive and in-game improvement

Proprioceptive improvement is visualized in Figure 4a. A Mann-Whiteney Test showed no significant difference in relative improvement in between the two conditions , and an independent T-test showed no significant difference in improvement .

FIGURE 4

Figures 4b,c show the pre-post statistics of and for each participant. Paired T-tests showed no statistically significant improvement in after training in either the attractor or tunnel conditions . A Wilcoxon signed test also showed no significant improvement in after training in the attractor condition . However, a paired T-test indicated a significant increase in after training in the tunnel condition , corresponding to a worsening movement accuracy. In both conditions, we found that post-training was strongly positively correlated with pre-training . on the other hand showed a strong tendency to regress towards the mean, i.e., users who started out with the poorest scores improved while those who started out with the highest scores showed a decrease. Overall, the analyses of the proprioceptive measures do not support H1.A.

To compare In-game improvement between our two conditions, we tested the parallelism of the nonparametric ANCOVA and found , which rejects the null hypothesis that the two curves are parallel. In Figure 5 it can be observed that in-game scores increased more steeply in the tunnel condition, which is consistent with H1.B. However, this finding should be interpreted with caution because of the ceiling effects that can be seen in the in-game scores in the attractor condition, i.e., because those scores were high from the beginning, there was little room for further improvement.

FIGURE 5

4.2 Factors affecting proprioceptive learning

The histograms of participants’ responses to the three awareness questions are shown in Figure 6. The responses from “strongly disagree” to “strongly agree” were mapped to one through five for the data analyses. Mann Whiteney U test found no significant difference between conditions for any of the three responses: for “unseen forces”, for “all by itself”, and for “drifting off-course”.

FIGURE 6

Few participants in either condition indicated a strong awareness of the presence of visual assistance. However, a Pearson correlation analysis showed a significant negative correlation between awareness of assistance and improvement in the attractor condition , as well as a negative correlation between awareness of assistance and improvement , although the significance of the latter did not reach the 0.05 threshold. This indicates that, in the attractor condition, users who were more likely to report the presence of attractor assistance were less likely to experience proprioceptive learning after gameplay. See Figure 7 for an illustration of this relationship.

FIGURE 7

We also found a significant positive correlation in the tunnel condition between ratings of mental exhaustion on the NASA TLX and a sense of the presence of unseen forces affecting the airplane’s behavior and a perception of the airplane drifting off course , as well as a significant positive correlation between sensing the presence of unseen forces and perceiving the plane to be drifting off course . This suggests an overall negative impact of the invisible tunnel assistance.

4.3 In-game scores and usability

Figure 8 gives a statistical summary of the an illustration of usability, enjoyment and in-game performance scores. Mean overall NASA TLX scores were in the attractor condition, and in the tunnel condition, where a smaller value corresponds to lower perceived workload. Enjoyment ratings (higher is better) were in the attractor condition, and in the tunnel condition, after the removal of a single outlier with a score of two in the attractor condition. Average total in-game scores (out of a maximum possible 900) were in the attractor condition, and in the tunnel condition.

FIGURE 8

A Mann-Whitney U Test showed significant differences in the total in-game scores and enjoyment , between the two assistance conditions, with better in-game performance and enjoyment with the attractor assistance. These results support H2.A and H2.B. Despite the difference, the in-game performance and enjoyment were high in both conditions. The difference in TLX scores was not significant according to an independent T-Test .

Only one participant in each condition chose to play the game while waiting for their data to be uploaded, so there was no difference in persistence between the two assistance conditions. The result does not support H2.C. During debriefing, most participants reported that fatigue was the main reason why they did not want to keep playing, even though they did enjoy the game.

4.4 Reliance on assistance

We performed a Mann-Kendall test for each participant to see if there was an increasing/decreasing trend in reliance on assistance over the training period. The results are shown in Table 2. For the majority of the participants, there was either no trend in reliance or the trend was decreasing. This result supports H3.

5 Discussion

The most notable finding from our study is the failure to see proprioceptive improvement after training in either of our visual assistance conditions. Given that prior related studies (Elangovan et al., 2017; 2018; Wang et al., 2021) had consistently observed some proprioceptive learning after a similar training process, we feel that the most likely explanation for this finding is that both the attractor and tunnel assistance, as implemented, interfered with motor learning. While we would need to do a follow-up study directly comparing proprioceptive learning after training with and without visual assistance to robustly confirm this hypothesis, we advise in the meantime that the use of visual-only assistance in proprioceptive training for stroke rehabilitation should be treated with caution. We had hypothesized that an intermittent assistance model like our tunnel method could better support proprioceptive training by preserving a consistent mapping from motor input to visual feedback over a greater period of time, but our data show that this hope was not borne out. Given that the tunnel condition was associated with less enjoyment, and a higher awareness of this assistance was associated with stronger perception of the plane “drifting off-course” as well as worsening movement accuracy error, it is possible that the experience of encountering invisible guardrails was non-intuitive and therefore unhelpful to the proprioceptive training. In future training development efforts, it might be better to explicitly show the guardrails rather than rendering them invisibly, as well as exploring other possible approaches to providing assistance that are less likely to interfere with the proprioceptive learning process.

Both assistance methods resulted in satisfactory in-game performance and high subjective enjoyment, successfully fulfilling their assistive purpose, which confirms the positive findings from prior research using visual assistance (Dukes et al., 2013; Ferdous et al., 2022; Chowdhury et al., 2019). However, neither method led to improvements in proprioception. This result highlights an important implication: relying solely on task performance and subjective measures, as commonly done in prior studies, may be insufficient for evaluating the effectiveness of visual assistance techniques in stroke rehabilitation or motor training. It is essential to include motor skill improvement as a key evaluation metric.

Our failure to elicit proprioceptive learning might also be explained by an overall dampening in motor performance after gameplay due to fatigue. The wristbot was heavy and required a lot of physical effort to manipulate. Because of this, few participants opted to continue playing after the end of the experiment. In the attractor condition, users who reported higher TLX physical demand had worse in-game performance and lower post-game PT. However, in the tunnel condition, higher TLX physical demand was associated with a greater improvement in gameplay performance, which is inconsistent with a negative impact of exhaustion.

It is also possible that the game we designed was somehow fundamentally ill-suited to supporting improved sensorimotor learning, independent of the presence of visual assistance. Unfortunately this possibility cannot be ruled out, as we did not include a control condition in our study. However, given the consistency with which motor improvements have been observed in the past after similar periods of engagement with a wide variety of similarly-designed games, we find this unlikely. We also note that there is limited practical value in considering proprioceptive training without some form of assistance. Nevertheless, we recognize that more robust followup work, including a control condition of no assistance, is needed to definitively establish the potentially deleterious impact to sensorimotor learning from providing visual assistance that surreptitiously varies the mapping between physical action and visual feedback in a constantly-changing way.

6 Limitations

Because the experiment did not contain a control condition, we cannot be certain that the lack of motor learning outcomes can be wholly attributed to the use of visual assistance. While we had intended that our experiment would compare a “high” visual assistance condition to a “low” visual assistance condition, our data suggest that our two visual assistance conditions may have each been detrimental in its own way, but we cannot prove that with the data we have and further studies would be needed to explore that possibility.

Also, due to the preliminary nature of our investigations and in accordance with prior norms for working with vulnerable populations, we tested both of our systems on healthy participants. It is possible that the introduction of visual assistance might impact such populations differently. Experiments with vulnerable/protected populations will be necessary in the future, after further preliminary hypothesis testing with healthy participants.

7 Conclusion

In this paper, we set out to investigate the extent to which visual-only assistance, i.e., manipulating the mapping from motor input to visual feedback towards greater in-game success, might potentially hinder sensorimotor learning during gamified proprioceptive training. We developed a motor training game and compared two different implementations of visual assistance: one that operated continuously and used a constantly-varying mapping to persistently manipulate the player’s visual feedback toward showing greater success in gameplay, and one that used an invisible tunnel paradigm to prevent the player from straying too far off course, thereby preserving an unmodified mapping from motor input to visual feedback throughout a greater proportion of the training session. Our hypothesis was that the “tunnel” type of assistance would better support proprioceptive learning, despite providing a potentially inferior user experience. However, our results showed a failure to achieve significant proprioceptive improvement after training with either method. Our results suggest that the use of visual assistance during motor training should be treated with caution, and highlights the need for further investigations to explore the potential impact of visual assistance in proprioceptive training for stroke rehabilitation.

References

• 1

  AmanJ. E.ElangovanN.YehI.-L.KonczakJ. (2015). The effectiveness of proprioceptive training for improving motor function: a systematic review. Front. Hum. Neurosci.8, 1075. 10.3389/fnhum.2014.01075

  • CrossRef
  • Google Scholar

• 2

  BaranyiR.ReiseckerF.LedererN.GobberM.GrechenigT. (2014). “Wristdroid-a serious game to support and motivate patients throughout their wrist rehabilitation,” in 2014 IEEE conference on biomedical engineering and sciences (IECBES) (IEEE), 786–791.

  • Google Scholar

• 3

  BasterisA.NijenhuisS. M.StienenA. H.BuurkeJ. H.PrangeG. B.AmirabdollahianF. (2014). Training modalities in robot-mediated upper limb rehabilitation in stroke: a framework for classification based on a systematic review. J. neuroengineering rehabilitation11, 111–115. 10.1186/1743-0003-11-111

  • CrossRef
  • Google Scholar

• 4

  BorboniA.MorM.FagliaR. (2016). Gloreha—hand robotic rehabilitation: design, mechanical model, and experiments. J. Dyn. Syst. Meas. Control138, 138. 10.1115/1.4033831

  • CrossRef
  • Google Scholar

• 5

  BurkeJ. W.McNeillM.CharlesD. K.MorrowP. J.CrosbieJ. H.McDonoughS. M. (2009). Optimising engagement for stroke rehabilitation using serious games. Vis. Comput.25, 1085–1099. 10.1007/s00371-009-0387-4

  • CrossRef
  • Google Scholar

• 6

  BütefischC.HummelsheimH.DenzlerP.MauritzK.-H. (1995). Repetitive training of isolated movements improves the outcome of motor rehabilitation of the centrally paretic hand. J. neurological Sci.130, 59–68. 10.1016/0022-510x(95)00003-k

  • CrossRef
  • Google Scholar

• 7

  CareyL. M.OkeL. E.MatyasT. A. (1996). Impaired limb position sense after stroke: a quantitative test for clinical use. Archives Phys. Med. rehabilitation77, 1271–1278. 10.1016/s0003-9993(96)90192-6

  • CrossRef
  • Google Scholar

• 8

  CasadioM.GiannoniP.MorassoP.SanguinetiV. (2009). A proof of concept study for the integration of robot therapy with physiotherapy in the treatment of stroke patients. Clin. Rehabil.23, 217–228. 10.1177/0269215508096759

  • CrossRef
  • Google Scholar

• 9

  ChangJ.-J.TungW.-L.WuW.-L.HuangM.-H.SuF.-C. (2007). Effects of robot-aided bilateral force-induced isokinetic arm training combined with conventional rehabilitation on arm motor function in patients with chronic stroke. Archives Phys. Med. rehabilitation88, 1332–1338. 10.1016/j.apmr.2007.07.016

  • CrossRef
  • Google Scholar

• 10

  ChowdhuryT.FerdousS.PeckT.QuarlesJ. (2019). “Virtual ability simulation: applying rotational gain to the leg to increase confidence during physical rehabilitation,” in ICAT-EGVE 2019-international conference on artificial reality and telexistence and eurographics symposium on virtual environments.

  • Google Scholar

• 11

  CupponeA. V.SempriniM.KonczakJ. (2018). Consolidation of human somatosensory memory during motor learning. Behav. brain Res.347, 184–192. 10.1016/j.bbr.2018.03.013

  • CrossRef
  • Google Scholar

• 12

  DemecoA.ZolaL.FrizzieroA.MartiniC.PalumboA.ForestiR.et al (2023). Immersive virtual reality in post-stroke rehabilitation: a systematic review. Sensors23, 1712. 10.3390/s23031712

  • CrossRef
  • Google Scholar

• 13

  DishmanR. K.IckesW. (1981). Self-motivation and adherence to therapeutic exercise. J. Behav. Med.4, 421–438. 10.1007/bf00846151

  • CrossRef
  • Google Scholar

• 14

  DobkinB. H. (2004). Strategies for stroke rehabilitation. Lancet Neurology3, 528–536. 10.1016/s1474-4422(04)00851-8

  • CrossRef
  • Google Scholar

• 15

  DukesP. S.HayesA.HodgesL. F.WoodburyM. (2013). “Punching ducks for post-stroke neurorehabilitation: system design and initial exploratory feasibility study,” in2013 IEEE symposium on 3D user interfaces (3DUI) (IEEE), 47–54.

  • Google Scholar

• 16

  ElangovanN.CappelloL.MasiaL.AmanJ.KonczakJ. (2017). A robot-aided visuo-motor training that improves proprioception and spatial accuracy of untrained movement. Sci. Rep.7 (1), 17054. 10.1038/s41598-017-16704-8

  • CrossRef
  • Google Scholar

• 17

  ElangovanN.TuiteP. J.KonczakJ. (2018). Somatosensory training improves proprioception and untrained motor function in Parkinson’s disease. Front. neurology9, 1053. 10.3389/fneur.2018.01053

  • CrossRef
  • Google Scholar

• 18

  FerdousS. M. S.MichaelA.ChowdhuryT. I.QuarlesJ. (2022). “Use of scaling to improve reach in virtual reality for people with Parkinson’s disease,” in 2022 IEEE 10th international conference on serious games and applications for health (SeGAH). IEEE, 1–7.

  • Google Scholar

• 19

  FreesS.KesslerG. D. (2005). “Precise and rapid interaction through scaled manipulation in immersive virtual environments,” in IEEE proceedings. VR 2005. Virtual reality, 2005 (IEEE), 99–106.

  • Google Scholar

• 20

  FreesS.KesslerG. D.KayE. (2007). Prism interaction for enhancing control in immersive virtual environments. ACM Trans. Computer-Human Interact. (TOCHI)14, 2–es. 10.1145/1229855.1229857

  • CrossRef
  • Google Scholar

• 21

  HaoJ.XieH.HarpK.ChenZ.SiuK.-C. (2022). Effects of virtual reality intervention on neural plasticity in stroke rehabilitation: a systematic review. Archives Phys. Med. Rehabilitation103, 523–541. 10.1016/j.apmr.2021.06.024

  • CrossRef
  • Google Scholar

• 22

  HartS. G.StavelandL. E. (1988). Development of nasa-tlx (task load index): results of empirical and theoretical research. Adv. Psychol.52, 139–183. 10.1016/s0166-4115(08)62386-9

  • CrossRef
  • Google Scholar

• 23

  HussainS.JamwalP. K.Van VlietP.GhayeshM. H. (2020). State-of-the-art robotic devices for wrist rehabilitation: design and control aspects. IEEE Trans. Human-Machine Syst.50, 361–372. 10.1109/thms.2020.2976905

  • CrossRef
  • Google Scholar

• 24

  KarimeA.EidM.Alja’AmJ. M.El SaddikA.GueaiebW. (2013). A fuzzy-based adaptive rehabilitation framework for home-based wrist training. IEEE Trans. Instrum. Meas.63, 135–144. 10.1109/tim.2013.2277536

  • CrossRef
  • Google Scholar

• 25

  KawahiraK.ShimodozonoM.OgataA.TanakaN. (2004). Addition of intensive repetition of facilitation exercise to multidisciplinary rehabilitation promotes motor functional recovery of the hemiplegic lower limb. J. Rehabilitation Med.36, 159–164. 10.1080/16501970410029753

  • CrossRef
  • Google Scholar

• 26

  KenzieJ. M.SemrauJ. A.FindlaterS. E.HerterT. M.HillM. D.ScottS. H.et al (2014). Anatomical correlates of proprioceptive impairments following acute stroke: a case series. J. neurological Sci.342, 52–61. 10.1016/j.jns.2014.04.025

  • CrossRef
  • Google Scholar

• 27

  KhokaleR.MathewG. S.AhmedS.MaheenS.FawadM.BandaruP.et al (2023). Virtual and augmented reality in post-stroke rehabilitation: a narrative review. Cureus15, e37559. 10.7759/cureus.37559

  • CrossRef
  • Google Scholar

• 28

  LambercyO.DovatL.YunH.WeeS. K.KuahC. W.ChuaK. S.et al (2011). Effects of a robot-assisted training of grasp and pronation/supination in chronic stroke: a pilot study. J. neuroengineering rehabilitation8, 63–12. 10.1186/1743-0003-8-63

  • CrossRef
  • Google Scholar

• 29

  LanghorneP.BernhardtJ.KwakkelG. (2011). Stroke rehabilitation. Lancet377, 1693–1702. 10.1016/s0140-6736(11)60325-5

  • CrossRef
  • Google Scholar

• 30

  LaverK. E.LangeB.GeorgeS.DeutschJ. E.SaposnikG.CrottyM. (2017). Virtual reality for stroke rehabilitation. Cochrane database Syst. Rev.2018. 10.1002/14651858.cd008349.pub4

  • CrossRef
  • Google Scholar

• 31

  MariniF.HughesC. M.SqueriV.DoglioL.MorettiP.MorassoP.et al (2017). Robotic wrist training after stroke: adaptive modulation of assistance in pediatric rehabilitation. Robotics Aut. Syst.91, 169–178. 10.1016/j.robot.2017.01.006

  • CrossRef
  • Google Scholar

• 32

  MasiaL.CasadioM.SandiniG.MorassoP. (2009). Eye-hand coordination during dynamic visuomotor rotations. PloS one4, e7004. 10.1371/journal.pone.0007004

  • CrossRef
  • Google Scholar

• 33

  MasieroS.CeliaA.RosatiG.ArmaniM. (2007). Robotic-assisted rehabilitation of the upper limb after acute stroke. Archives Phys. Med. rehabilitation88, 142–149. 10.1016/j.apmr.2006.10.032

  • CrossRef
  • Google Scholar

• 34

  MontagnerA.FrisoliA.BorelliL.ProcopioC.BergamascoM.CarbonciniM. C.et al (2007). “A pilot clinical study on robotic assisted rehabilitation in vr with an arm exoskeleton device,” in 2007 virtual rehabilitation (IEEE), 57–64.

  • Google Scholar

• 35

  MubinO.AlnajjarF.Al MahmudA.JishtuN.AlsinglawiB. (2020). Exploring serious games for stroke rehabilitation: a scoping review. Disabil. Rehabilitation Assistive Technol.17, 159–165. 10.1080/17483107.2020.1768309

  • CrossRef
  • Google Scholar

• 36

  MurakamiH.FiscaleV.SocciniA. M.InamuraT. (2024). “White lies in virtual reality: impact on enjoyment and fatigue,” in Proceedings of the 30th ACM symposium on virtual reality software and technology, 1–2.

  • Google Scholar

• 37

  OstryD. J.GribbleP. L. (2016). Sensory plasticity in human motor learning. Trends Neurosci.39, 114–123. 10.1016/j.tins.2015.12.006

  • CrossRef
  • Google Scholar

• 38

  PoupyrevI.BillinghurstM.WeghorstS.IchikawaT. (1996). “The go-go interaction technique: non-linear mapping for direct manipulation in vr,” in Proceedings of the 9th annual ACM symposium on User interface software and technology, 79–80.

  • Google Scholar

• 39

  PrinsN. (2013). The psi-marginal adaptive method: how to give nuisance parameters the attention they deserve (no more, no less). J. Vis.13, 3. 10.1167/13.7.3

  • CrossRef
  • Google Scholar

• 40

  RandD.GivonN.WeingardenH.NotaA.ZeiligG. (2014). Eliciting upper extremity purposeful movements using video games: a comparison with traditional therapy for stroke rehabilitation. Neurorehabilitation neural repair28, 733–739. 10.1177/1545968314521008

  • CrossRef
  • Google Scholar

• 41

  RossR. E.HartE.WilliamsE. R.GregoryC. M.FlumeP. A.MingoraC. M.et al (2023). Combined aerobic exercise and virtual reality-based upper extremity rehabilitation intervention for chronic stroke: feasibility and preliminary effects on physical function and quality of life. Archives Rehabilitation Res. Clin. Transl.5, 100244. 10.1016/j.arrct.2022.100244

  • CrossRef
  • Google Scholar

• 42

  SluijsE. M.KokG. J.Van der ZeeJ. (1993). Correlates of exercise compliance in physical therapy. Phys. Ther.73, 771–782. 10.1093/ptj/73.11.771

  • CrossRef
  • Google Scholar

• 43

  SqueriV.MasiaL.GiannoniP.SandiniG.MorassoP. (2013). Wrist rehabilitation in chronic stroke patients by means of adaptive, progressive robot-aided therapy. IEEE Trans. neural Syst. rehabilitation Eng.22, 312–325. 10.1109/tnsre.2013.2250521

  • CrossRef
  • Google Scholar

• 44

  TacchinoG.GandollaM.CoelliS.BarbieriR.PedrocchiA.BianchiA. M. (2016). Eeg analysis during active and assisted repetitive movements: evidence for differences in neural engagement. IEEE Trans. Neural Syst. Rehabilitation Eng.25, 761–771. 10.1109/tnsre.2016.2597157

  • CrossRef
  • Google Scholar

• 45

  VeerbeekJ. M.Langbroek-AmersfoortA. C.Van WegenE. E.MeskersC. G.KwakkelG. (2017). Effects of robot-assisted therapy for the upper limb after stroke: a systematic review and meta-analysis. Neurorehabilitation neural repair31, 107–121. 10.1177/1545968316666957

  • CrossRef
  • Google Scholar

• 46

  WangY.ZhuH.ElangovanN.CappelloL.SandiniG.MasiaL.et al (2021). A robot-aided visuomotor wrist training induces gains in proprioceptive and movement accuracy in the contralateral wrist. Sci. Rep.11, 5281. 10.1038/s41598-021-84767-9

  • CrossRef
  • Google Scholar

• 47

  WentzelJ.d’EonG.VogelD. (2020). “Improving virtual reality ergonomics through reach-bounded non-linear input amplification,” in Proceedings of the 2020 CHI conference on human factors in computing systems, 1–12.

  • Google Scholar