Using Deep Learning to Detect Upper Limb Compensation in Individuals Post-Stroke Using Consumer-Grade Webcams - A Feasibility Study

Tim Unger^1*Benjamin Kühnis²Lena Sauerzopf³Martina Rebekka Spiess³Alexander De Spindler²Andreas R Luft^4,5,6Chris Awai Easthope^1*Josef G. Schönhammer^4,6Elena Gavagnin²

• ¹Data Analytics & Rehabilitation Technology (DART), Lake Lucerne Institute AG, Vitznau, Switzerland
• ²ZHAW Zurcher Hochschule fur Angewandte Wissenschaften School of Management and Law, Winterthur, Switzerland
• ³ZHAW Zurcher Hochschule fur Angewandte Wissenschaften Departement Gesundheit, Winterthur, Switzerland
• ⁴Department of Neurology and Clinical Neuroscience Center, UniversitatsSpital Zurich, Zürich, Switzerland
• ⁵cereneo Schweiz AG, Weggis, Switzerland
• ⁶Lake Lucerne Institute AG, Vitznau, Switzerland

As societies age, the number of individuals experiencing stroke increases, necessitating more effective rehabilitation strategies. Over half of stroke survivors suffer from upper limb impairments, making assessments of sensory-motor function crucial for both improving interventions and tracking progress. Ideally, such assessments could also be performed at home without requiring a therapist's presence. Advances in computer vision and human pose estimation allow for human movement analysis using consumer-grade cameras. This study investigates whether a single webcam, combined with human pose estimation and deep learning algorithms, can automatically detect compensatory movements in persons with stroke performing a drinking task. 20 participants with stroke with mild to moderate upper limb impairment were recruited. Each participant performed multiple repetitions of the drinking task while being recorded by multiple cameras and an optical motion capture system (OMC) for kinematic ground truth. The videos were labeled by therapists to indicate the presence or absence of compensatory movements. Human poses were extracted from the videos using MediaPipe, and deep learning models were trained to predict these compensatory movements based on MediaPipe keypoints. Several factors affecting compensation detection accuracy were evaluated. Models trained on raw MediaPipe keypoints for inter-person compensation detection failed to generalize, achieving accuracy around 50%. Using custom features instead of raw keypoints improved the accuracy to 70%. In contrast, intraperson classification achieved high accuracy, typically exceeding 90%. Using OMC data significantly improved classification accuracy compared to using MediaPipe keypoints. Camera angle had an effect on accuracy, and convolutional neural networks outperformed long short-term memory networks. Generalizing models remain limited by (1) the measurement uncertainty of human pose estimation and (2) insufficient data representing the full spectrum of compensatory strategies (3) accurate compensation labels. The results demonstrate that deep learning approaches can differentiate between compensatory and non-compensatory movements when movement representations are sufficiently accurate. Future work should improve pose estimation and expand labeled datasets to better reflect the stroke population. While general models are limited in accuracy, personalized models using consumer cameras can support home-based rehabilitation. This digitalized assessment approach has the potential to quantify recovery progress throughout the continuum of care.