Beyond the Joystick: Deep Learning Games for Hand Movement Recovery

Vrinda AcharyaHirakjyoti RoySurekha KamathAneesha Acharya K^*

• Manipal Academy of Higher Education, Manipal, India

Abstract—This research focuses on utilizing Deep Learning (DL) techniques for hand and gesture recognition, mainly targeting hand rehabilitation programs. The application aims to improve cognitive functions and hand-eye coordination through engaging and gamified mental exercises by accurately tracking hand movements and gestures. Pre-trained hand recognition models are used extensively, using Convolutional Neural Networks (CNNs), and several old arcade games are reimplemented as game exercises. This work also incorporates a score-based system for easy progress tracking, among other metrics. The MediaPipe Library, open-sourced by Google, for implementing hand and gesture recognition models, is used for hand tracking and gesture recognition. Four classic arcade games Pong, Tetris, Fruit Ninja, and a Virtual Keyboard have been reimagined and developed into gesture-controlled rehabilitation tools. The final web-based UI, the games, was built using the Phaser.js library. The system records metrics such as gesture recognition accuracy, with data from 15 sample participants showing consistent results. System Usability Scale (SUS) study was employed to evaluate the usability of virtual hand-tracking games. A one-sample t-test against the benchmark score was conducted to determine whether the system’s usability exceeded the accepted industry standard. The study documented the viability of monocular camera-based computer vision for dexterity recovery, offering a low-cost, accessible, and engaging therapeutic platform compared to alternative Low-Cost Hand Rehabilitation Tools. Implementing games and integrating the controls for these games with the gesture recognition module seems to prove the proof-of-concept feasibility study that mono camera-based computer vision techniques can be used for dexterity exercises in future.