Deep learning based markerless motion tracking as a clinical tool for movement disorders: Utility, feasibility and early experience

Clinical assessments of movement disorders currently rely on the administration of rating scales, which, while clinimetrically validated and reliable, rely on clinicians’ subjective analyses, resulting in interrater differences. Intraoperative microelectrode recording for deep brain stimulation targeting similarly relies on clinicians’ subjective evaluations of movement-related neural activity. Digital motion tracking can improve the diagnosis, assessment, and treatment of movement disorders by generating objective, standardized measures of patients’ kinematics. Motion tracking with concurrent neural recording also enables motor neuroscience studies to elucidate the neurophysiology underlying movements. Despite these promises, motion tracking has seen limited adoption in clinical settings due to the drawbacks of conventional motion tracking systems and practical limitations associated with clinical settings. However, recent advances in deep learning based computer vision algorithms have made accurate, robust markerless motion. tracking viable in any setting where digital video can be captured. Here, we review and discuss the potential clinical applications and technical limitations of deep learning based markerless motion tracking methods with a focus on DeepLabCut (DLC), an open-source software package that has been extensively applied in animal neuroscience research. We first provide a general overview of DLC, discuss its present usage, and describe the advantages that DLC confers over other motion tracking methods for clinical use. We then present our preliminary results from three ongoing studies that demonstrate the use of DLC for 1) movement disorder patient assessment and diagnosis, 2) intraoperative motor mapping for deep brain stimulation targeting and 3) intraoperative neural and kinematic recording for basic human motor neuroscience.


Introduction
Movement disorders such as Parkinson's disease (PD) and essential tremor (ET) are prevalent, debilitating diseases.PD affects 1.6% of the population (Pringsheim et al., 2014) and ET affects 4.6% of people older than 65 (Louis and Ferreira, 2010).These disorders are initially treated with neuromodulatory medications, and in cases where symptoms are not adequately controlled, neurosurgical interventions including deep brain stimulation (DBS) are considered.
The current method of movement disorder severity assessment is the administration of a rating scale by a trained clinician.The most common scale for PD (Mitchell et al., 2000) is the Movement Disorder Society sponsored update of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) (Fahn, 1987;Goetz et al., 2007).Despite the MDS-UPDRS being a validated metric for assessment of parkinsonism, interrater reliability is moderate (Heldman et al., 2011;Luiz et al., 2021) or low in the case of tremor ratings (Martinez-Martin et al., 1994;Richards et al., 1994;Goetz et al., 1995).This is remarkable given the extensive training necessary to become a movement disorders expert capable of administering these scales.Similar subjectively rated scales have been developed for ET, such as The Essential Tremor Rating Assessment Scale (TETRAS) (Elble et al., 2008).
DBS surgery also relies on subjective clinician judgments when aided by awake intraoperative microelectrode recording (MER).MER requires clinicians to judge the presence or absence of correspondence between neural activity and patient movement (Hutchison, 2009;Abosch et al., 2013).Tuning of DBS parameters requires additional subjective assessments of motor improvements.
While DBS is effective, its mechanism of action is not fully understood (Lozano et al., 2019).Though much has been learned about PD neuropathophysiology (Gonzalez-Escamilla et al., 2020), many questions remain.In the case of ET, even less is known (Haubenberger and Hallett, 2018).
Digital motion tracking could help to address these issues (Chen et al., 2016).Motion tracking can objectively quantify subtle movement variations during clinical exams, leading to more consistent assessments (Belić et al., 2019).Furthermore, motion tracking can aid clinicians by generating objective measures during intraoperative MER.Motion tracking combined with neural recording enables investigation of the neurophysiology of brain areas involved with movement disorders, informing future treatments.
In recent years, deep learning based markerless motion tracking software, such as DeepLabCut (DLC) (Mathis et al., 2018), has emerged as a flexible and reliable method to measure kinematics.We propose that markerless methods such as DLC have the potential to aid in the diagnosis, assessment, treatment, and neuroscience of movement disorders.In this review, we discuss the methodology of DLC, its current uses, and its applicability in clinical settings.We then describe three of our ongoing studies that utilize DLC: 1) a clinical trial for assessment and diagnosis of movement disorders, 2) the development of a tool for objective MER-assisted motor mapping during DBS surgery for PD and 3) a basic motor neuroscience study characterizing the relationship between thalamic neuron spiking and reach kinematics in ET.The basic setups and preliminary results of these studies are presented here as proofs-of-concept for the use of DLC in clinical settings; rigorous analysis and quantification of results will be presented in future publications.
2 Review of markerless motion tracking methods

Technical background
Conventional motion tracking methods such as inertial, magnetic, acoustic, and electromechanical sensors, and videobased tracking of reflective markers or LEDs (Zhou and Hu, 2008), all require that a device or marker be affixed to the body part of interest, and often require expensive equipment.
In the last decade, deep learning (the application of manylayered artificial neural networks) has seen considerable progress, especially in image recognition tasks (Voulodimos et al., 2018) such as pose estimation (Dang et al., 2019).Accordingly, several open-source deep learning based tools specifically designed for motion tracking, such as DLC (Mathis et al., 2018), have emerged.These tools allow for motion to be digitized directly from video, eliminating the need for sensors or markers.
DLC is an open-source, Python-based software package adapted from the feature extraction layers of DeeperCut (Insafutdinov et al., 2016), a pose estimation network trained on ImageNet (Deng et al., 2009).DLC networks can be adapted to track subjects in novel environments using transfer learning (Mathis et al., 2018).The typical workflow for motion tracking with DLC is the following: 1) collect video using any digital camera, 2) manually label the body parts or objects to be tracked in a subset of frames, 3) train the network with this input data, and 4) apply the network to the entire video.For each frame, DLC outputs the estimated pixel locations of the tracked points and numerical measures of confidence (Nath et al., 2019).(Mathis et al., 2018;Nath et al., 2019) using Google Scholar.The Kinect (Microsoft) and Leap Motion controller (Ultraleap) are consumer markerless motion tracking devices that do not utilize deep learning.For the "Human Tracking, Non-Clinical" and "Non-Human Tracking" categories, only studies which utilized DLC were enumerated; studies using other markerless motion tracking methods were excluded for brevity.
DLC is particularly adept at tracking a wide variety of subjects in unconventional settings (Mathis and Mathis, 2020).This flexibility is evidenced by the widespread adoption of DLC in animal neuroscience across many different species and scenarios.DLC has also been used to track humans in complex environments.The variety of studies utilizing DLC is enumerated in Table 1.
Markerless methods in general are attractive for clinical use because they do not require anything to be attached to the body.This reduces setup time, potential for error, and patient encumbrance, which is especially important for movement disorder patients with limited or excessive movements, and who may need to physically interact with clinicians for evaluation.Additionally, the capability of DLC to track motion in a wide range of settings is desirable as clinical environments, especially operating rooms, are visually complex due to reflective surfaces and variable lighting conditions.As such, many clinical studies have been conducted using markerless methods such as DLC, OpenPose, custom neural networks, Kinect (Microsoft), Leap Motion (Ultraleap), and other non-neural network solutions.These preliminary studies have focused on a range of applications, including quantifying gross kinematics for movement disorders or rehabilitation, capturing facial expression changes, and tracking movements of internal anatomy using radiography and endoscopic cameras.These studies are enumerated in Table 1, and studies particularly relevant to the use of markerless tracking for movement disorders are discussed in the following sections.
Based on these advantages, we have chosen to use DLC to pursue three novel clinical research questions, as described below.
3 Study 1: Clinical trial using deeplabcut for movement disorder diagnosis and assessment

Study 1: Description of current work
We designed a clinical trial (https://clinicaltrials.gov/ct2/show/ record/NCT04074772) to evaluate the use of DLC for automated movement disorder disease state evaluation and classification.We built a custom mobile frame to position three synchronized cameras (Blackfly USB3, Teledyne FLIR) in front of subjects (Figure 1A).Healthy control subjects and movement disorder patients with various diagnoses, including PD and ET, were filmed while performing hand movements used in movement disorder rating scales.
Subject-specific DLC networks were trained to track 11 points on the hand (Figure 1B), which were then reconstructed in 3D (Figure 1C).Subjects were also rated by a movement disorders neurologist.We expect that DLC output can be used to predict rating scale scores, and that dimensionality reduction and clustering of kinematic data will distinguish movement disorder patients with different diagnoses from each other and from controls.
By using an unobtrusive video recording system and DLC, we can obtain 3D positional data during clinical exams without restricting patient movements.This method can measure kinematic features of hard-to-detect movement variations in clinically relevant tasks like finger tapping and tremor, which are used to characterize disease state and evaluate subtle changes in treatment responses during DBS programming.
4 Study 2: Use of deeplabcut for functional targeting of deep brain stimulation electrodes 4.1 Study 2: Background and related work DBS applied to the subthalamic nucleus (STN) is an effective treatment for advanced PD (Bronstein et al., 2011).After initial imaging-based targeting, MER is performed to accurately identify the motor territory of the dorsolateral STN for implantation of the therapeutic electrode (Benazzouz et al., 2002;Sterio et al., 2002;Hutchison, 2009;Gonzalez-Escamilla et al., 2020).Studies in which STN MER was performed simultaneously with electromyography have identified regions of STN with neural activity correlated with tremor and active and/or passive movements (Lenz et al., 1988;Magariños-Ascone et al., 2000;Rodriguez-Oroz et al., 2001;Amtage et al., 2008).
To identify the motor territory of STN with MER, a neurologist administers functional tests while visually and auditorily assessing the MER voltage trace.Clinicians seek to identify functional territories of STN defined by correlations between neural activity and tremor, voluntary and/or passive contralateral movements.This procedure relies on clinicians' subjective judgments of the presence or absence of movementrelated neural activity.An automated system to assess MER and kinematics could augment clinicians' judgments and could be a useful tool for centers where MER expertise is lacking.

Study 2: Description of current work
Utilizing DLC intraoperatively, we can objectively identify functional relationships between STN neural activity and kinematics through statistical analysis of MER signals and tracked motion.To this end, we designed and executed a study using DLC to track PD patient movement during MER (Colorado Multiple Institution Review Board, COMIRB #17-1291).
Video was obtained using two cameras (Blackfly USB3) mounted on tripods aimed at the operating table (Figure 1D) and synchronized with the MER system (Neuro Omega, AlphaOmega).DLC was used to track 21 points on the hand, and was able to distinguish the patient's hand from the neurologist's (Figure 1E).Kinematics (Figure 1F) were compared to the MER signals offline using dynamic time warping and Monte Carlo methods to identify instances of correspondence between movements and neural activity.These objective assessments of motion-related neural activity largely agreed with clinician judgments.
To our knowledge, this study is the first attempt to develop an automated tool to assist clinicians in functional MER for DBS electrode targeting.While this study is only a first step and operates offline, it explores the feasibility of such a technique with potential for real-time intraoperative use.A key advantage of this technique is that it is unobtrusive and does not interfere with standard clinical procedures.
5 Study 3: Use of deeplabcut for human motor neuroscience 5.1 Study 3: Background and related work MER during DBS surgery presents a rare opportunity to record from single neurons in wakeful humans (Engel et al., 2005;Cash and Hochberg, 2015;Lee et al., 2019;Tekriwal et al., 2019).This allows for basic neuroscience experiments to elucidate the neurophysiology of the targets of DBS, guiding the development of future treatments (Stein and Bar-Gad, 2013;Gonzalez-Escamilla et al., 2020).Several studies have been conducted with conventional motion tracking methods, but because of the constraints of the operating room, kinematic tracking has been limited to the use of joysticks (Amirnovin et al., 2004;Zavala et al., 2017;Tekriwal et al., 2018Tekriwal et al., , 2022)), grip force dynamometers (Patil et al., 2004;Fischer et al., 2020), bendsensitive resistors (Hanson et al., 2012) and inertial sensors (Levy et al., 2002a,b;MacMillan et al., 2004;Tankus et al., 2017Tankus et al., , 2018)).
Only a single study has used markerless tracking during MER.London et al. (2021) used the Leap Motion controller (Ultraleap) to allow subjects to control a cursor in real-time with hand movements, allowing the identification of one neural population in the STN that encoded kinematics, and another that responded to unexpected action plan changes.

Study 3: Description of current work
The ventral intermediate nucleus of the thalamus (VIM) is the primary target of DBS for the treatment of ET (Chopra et al., 2013).While it is known that neural activity in the VIM covaries with tremor and active and passive movements (Lenz et al., 1990(Lenz et al., , 1994;;Zirh et al., 1998;Lenz et al., 2002;Hua and Lenz, 2005;Cajigas et al., 2020), the precise relationship between VIM neurons and kinematics remains unknown.Recent work in mice (Becker and Person, 2019) revealed that the interposed nucleus of the cerebellum is implicated in the precise control of braking at the reach endpoint.The deep motor nuclei of the cerebellum project to the VIM in humans (Roostaei et al., 2014), suggesting that the VIM may also contribute to braking.
To investigate whether VIM neurons encode a brake signal, we designed a study involving a center-out reaching task during awake VIM MER in ET patients (COMIRB #20-2979).We utilized a cart-mounted computer monitor with three cameras (Blackfly USB3) mounted above it that can be positioned over the operating table (Figure 1G).This monitor displayed a central target and eight outer targets (Figure 1H).During the task, the central and outer targets were shown sequentially, cycling through all visible outer targets in a pseudorandom order.Patients were instructed to point at the currently visible target.DLC was used to extract fingertip position (Figure 1I).Regression and Monte Carlo methods were used to determine how VIM unit firing rates related to reach kinematics.These analyses reveal a temporal congruence between VIM neural activity and reach braking, and encoding of fingertip position in VIM units.
To our knowledge this is the first study to examine free reaching movements during human VIM MER, and the first use of DLC in such a setting.The flexibility of DLC enables examination of naturalistic movements that have previously been difficult to study during MER, such as handwriting, tool use and facial expressions.

Discussion
Deep learning based markerless motion tracking techniques can improve movement disorder diagnosis, assessment, treatment and neuroscience.DLC is especially appropriate for these tasks because of its ease-of-use, flexibility, low cost, open-source availability and development community, as evidenced by its widespread and rapid adoption in animal and human motion tracking applications.We have demonstrated the utility of DLC in three ongoing studies: a clinical trial using DLC for movement disorder assessment and diagnosis, the development of an intraoperative tool for functional DBS implant targeting in the STN, and a motor neuroscience study relating VIM neural activity to reach kinematics.
Though powerful, DLC does have limitations.The DLC analysis pipeline requires initial manual labeling.Because the output of DLC is only as good as its training data, this presents a potential for variation in tracking performance between different users.As DLC networks are trained anew for each recording scenario, standardization of output may be difficult.DLC network training can take tens of hours even with GPU acceleration.The ability of DLC to generalize across subjects of different size, skin color, gender or age has not been well studied.DLC tracks each feature independently and thus does not incorporate biomechanical or temporal constraints on motion.Finally, DLC can occasionally output spurious results, so human review of final output is advisable, especially if tracking is to be used for clinical decision making.
Specific limitations apply to the three ongoing studies discussed herein.In all three studies, a technician was required to position, calibrate and operate the cameras.Simpler systems will be required for integration into clinical workflows.Custom camera mount frames were built for studies 1 and 3, yielding consistency of camera positioning but potentially sub-optimal capture for each subject.In Study 2, tripods were used, resulting in variable views for each subject.Implementing an online version of Study 2 would require initial DLC network training in the operating room, increasing surgical time.The kinematic data presented in Figure 1 are preliminary, and while observed to be qualitatively accurate, further quantification of accuracy through validation with established datasets will be required.
The potential applications of DLC extend beyond the efforts described here.While all three described studies focused on upper limb movements and thus do not account for abnormal lower limb, head or jaw movements, these symptoms can be tracked and studied with DLC.Motion tracking is also critical for developmental research (van Schaik and Dominici, 2020), and DLC is capable of tracking infants (Pérez et al., 2021).DLC can also be used track gaze (Zdarsky et al., 2021) as well as facial expression (Argyle et al., 2021;Namba et al., 2021), making it a useful tool for the assessment of various neuropathies and for cognitive psychology research.Given the promise and utility of DLC, we predict that it and other markerless motion tracking technologies will see widespread adoption for clinical applications.

Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers.Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

FIGURE 1
FIGURE 1 Description of three ongoing studies utilizing DeepLabCut (DLC) in clinical settings.Top row: Clinical trial using DLC for movement disorder assessment and diagnosis.Middle row: Development of an intraoperative tool for automated motor mapping during DBS implantation of STN for treatment of PD.Bottom row: Motor neuroscience experiment linking VIM unit activity to reach kinematics in ET patients.(A,D and G) depict the physical setups for each study.(B) Hand tracking during a clinical movement disorder exam.(C) Output of DLC tracking in the vertical dimension of pointer finger (blue) and thumb (orange) tips during a finger tap test.(E) Hand tracking during active (left) and passive (right) movements for intraoperative MER based motor mapping.DLC is able to identify and label the patient's hand and not the clinician's during passive testing.(F) Average camera coordinate vertical position in pixels of all fingertips during a "chain pull" movement as tracked by DLC.(H) Monitor and cameras used to administer and track the center-out reaching task.While all targets are displayed simultaneously here, only one target was presented at a time during the task.(I) DLC-tracked position in pixels of pointer finger tip over an entire center-out session, with each color representing reaches to and from a different target (seven targets total for this subject).

TABLE 1
Enumeration of markerless motion tracking studies in the literature.DeepLabCut (DLC) studies were identified by searching for articles citing the initial DLC publications