Conventional motion tracking methods such as inertial, magnetic, acoustic, and electromechanical sensors, and video-based tracking of reflective markers or LEDs (Zhou and Hu, 2008), all require that a device or marker be affixed to the body part of interest, and often require expensive equipment.

In the last decade, deep learning (the application of many-layered artificial neural networks) has seen considerable progress, especially in image recognition tasks (Voulodimos et al., 2018) such as pose estimation (Dang et al., 2019). Accordingly, several open-source deep learning based tools specifically designed for motion tracking, such as DLC (Mathis et al., 2018), have emerged. These tools allow for motion to be digitized directly from video, eliminating the need for sensors or markers.

DLC is an open-source, Python-based software package adapted from the feature extraction layers of DeeperCut (Insafutdinov et al., 2016), a pose estimation network trained on ImageNet (Deng et al., 2009). DLC networks can be adapted to track subjects in novel environments using transfer learning (Mathis et al., 2018). The typical workflow for motion tracking with DLC is the following: 1) collect video using any digital camera, 2) manually label the body parts or objects to be tracked in a subset of frames, 3) train the network with this input data, and 4) apply the network to the entire video. For each frame, DLC outputs the estimated pixel locations of the tracked points and numerical measures of confidence (Nath et al., 2019).

The accuracy of DLC has been validated in human studies; when camera views of subjects are unobstructed, DLC performs comparably to or as well as inertial sensors (Pérez et al., 2021), electromagnetic sensors (Pouw et al., 2020), infrared markers (Moro et al., 2020; Vonstad et al., 2020; Drazan et al., 2021; Moro et al., 2021; Needham et al., 2021) and manual video labeling (Papic et al., 2021). DLC performs as well as or better than other markerless methods (Liu et al., 2020; Vonstad et al., 2020; Cronin, 2021; Needham et al., 2021) such as OpenPose (Cao et al., 2019), which does not allow for network retraining, and LEAP (Pereira et al., 2019), which uses a shallower network.

DLC is particularly adept at tracking a wide variety of subjects in unconventional settings (Mathis and Mathis, 2020). This flexibility is evidenced by the widespread adoption of DLC in animal neuroscience across many different species and scenarios. DLC has also been used to track humans in complex environments. The variety of studies utilizing DLC is enumerated in Table 1.

An additional advantage of DLC is its open-source availability and associated development community (Mathis et al., 2020; Anderson et al., 2021). DLC development is ongoing (Nath et al., 2019), and third-party contributions have enabled real-time tracking (Forys et al., 2020; Kane et al., 2020; Nourizonoz et al., 2020; Schweihoff et al., 2021; Sehara et al., 2021) and 3D reconstruction (Gosztolai et al., 2020; Sheshadri et al., 2020; Dunn et al., 2021; Karashchuk et al., 2021; Zhang et al., 2021).

Markerless methods in general are attractive for clinical use because they do not require anything to be attached to the body. This reduces setup time, potential for error, and patient encumbrance, which is especially important for movement disorder patients with limited or excessive movements, and who may need to physically interact with clinicians for evaluation. Additionally, the capability of DLC to track motion in a wide range of settings is desirable as clinical environments, especially operating rooms, are visually complex due to reflective surfaces and variable lighting conditions. As such, many clinical studies have been conducted using markerless methods such as DLC, OpenPose, custom neural networks, Kinect (Microsoft), Leap Motion (Ultraleap), and other non-neural network solutions. These preliminary studies have focused on a range of applications, including quantifying gross kinematics for movement disorders or rehabilitation, capturing facial expression changes, and tracking movements of internal anatomy using radiography and endoscopic cameras. These studies are enumerated in Table 1, and studies particularly relevant to the use of markerless tracking for movement disorders are discussed in the following sections.

Based on these advantages, we have chosen to use DLC to pursue three novel clinical research questions, as described below.

Objective quantification of movement can augment movement disorder diagnosis and assessment. Kinematic metrics obtained with conventional tracking methods have shown high correlations with clinical ratings and have predicted disease states. Such studies have been performed with inertial sensors (Burkhard et al., 1999; Hoff et al., 2001; Salarian et al., 2007; Giuffrida et al., 2009; Patel et al., 2009; Kim et al., 2011; Pulliam et al., 2014; Ramsperger et al., 2016; Delrobaei et al., 2017; Jeon et al., 2017), electromagnetic sensors (Espay et al., 2009; Gao et al., 2018), infrared markers (Das et al., 2011) and smartwatches (Malekmohammadi et al., 2016; López-Blanco et al., 2019). Similar studies have also been performed using markerless methods (Li M. H. et al., 2018; Liu et al., 2019; Sato et al., 2019; Wong et al., 2019; Williams et al., 2020a; Pang et al., 2020; Jaber et al., 2021; Shin et al., 2021). We identified four studies that used DLC for movement disorder diagnosis and/or assessment. Miao et al. (2020) tracked gait using iPad videos to distinguish dystonia patients from controls. Stolk et al. (2020) tracked children with dyskinetic cerebral palsy to predict Dyskinesia Impairment Scale scores. Shin et al. (2020) and Williams et al. (2020b) predicted MDS-UPDRS scores by tracking movements in archival footage and cellphone videos, respectively.

We designed a clinical trial (https://clinicaltrials.gov/ct2/show/record/NCT04074772) to evaluate the use of DLC for automated movement disorder disease state evaluation and classification. We built a custom mobile frame to position three synchronized cameras (Blackfly USB3, Teledyne FLIR) in front of subjects (Figure 1A). Healthy control subjects and movement disorder patients with various diagnoses, including PD and ET, were filmed while performing hand movements used in movement disorder rating scales.

Subject-specific DLC networks were trained to track 11 points on the hand (Figure 1B), which were then reconstructed in 3D (Figure 1C). Subjects were also rated by a movement disorders neurologist. We expect that DLC output can be used to predict rating scale scores, and that dimensionality reduction and clustering of kinematic data will distinguish movement disorder patients with different diagnoses from each other and from controls.

By using an unobtrusive video recording system and DLC, we can obtain 3D positional data during clinical exams without restricting patient movements. This method can measure kinematic features of hard-to-detect movement variations in clinically relevant tasks like finger tapping and tremor, which are used to characterize disease state and evaluate subtle changes in treatment responses during DBS programming.

DBS applied to the subthalamic nucleus (STN) is an effective treatment for advanced PD (Bronstein et al., 2011). After initial imaging-based targeting, MER is performed to accurately identify the motor territory of the dorsolateral STN for implantation of the therapeutic electrode (Benazzouz et al., 2002; Sterio et al., 2002; Hutchison, 2009; Gonzalez-Escamilla et al., 2020). Studies in which STN MER was performed simultaneously with electromyography have identified regions of STN with neural activity correlated with tremor and active and/or passive movements (Lenz et al., 1988; Magariños-Ascone et al., 2000; Rodriguez-Oroz et al., 2001; Amtage et al., 2008).

To identify the motor territory of STN with MER, a neurologist administers functional tests while visually and auditorily assessing the MER voltage trace. Clinicians seek to identify functional territories of STN defined by correlations between neural activity and tremor, voluntary and/or passive contralateral movements. This procedure relies on clinicians’ subjective judgments of the presence or absence of movement-related neural activity. An automated system to assess MER and kinematics could augment clinicians’ judgments and could be a useful tool for centers where MER expertise is lacking.

Utilizing DLC intraoperatively, we can objectively identify functional relationships between STN neural activity and kinematics through statistical analysis of MER signals and tracked motion. To this end, we designed and executed a study using DLC to track PD patient movement during MER (Colorado Multiple Institution Review Board, COMIRB #17–1291).

Video was obtained using two cameras (Blackfly USB3) mounted on tripods aimed at the operating table (Figure 1D) and synchronized with the MER system (Neuro Omega, AlphaOmega). DLC was used to track 21 points on the hand, and was able to distinguish the patient’s hand from the neurologist’s (Figure 1E). Kinematics (Figure 1F) were compared to the MER signals offline using dynamic time warping and Monte Carlo methods to identify instances of correspondence between movements and neural activity. These objective assessments of motion-related neural activity largely agreed with clinician judgments.

To our knowledge, this study is the first attempt to develop an automated tool to assist clinicians in functional MER for DBS electrode targeting. While this study is only a first step and operates offline, it explores the feasibility of such a technique with potential for real-time intraoperative use. A key advantage of this technique is that it is unobtrusive and does not interfere with standard clinical procedures.

MER during DBS surgery presents a rare opportunity to record from single neurons in wakeful humans (Engel et al., 2005; Cash and Hochberg, 2015; Lee et al., 2019; Tekriwal et al., 2019). This allows for basic neuroscience experiments to elucidate the neurophysiology of the targets of DBS, guiding the development of future treatments (Stein and Bar-Gad, 2013; Gonzalez-Escamilla et al., 2020). Several studies have been conducted with conventional motion tracking methods, but because of the constraints of the operating room, kinematic tracking has been limited to the use of joysticks (Amirnovin et al., 2004; Zavala et al., 2017; Tekriwal et al., 2018, 2022), grip force dynamometers (Patil et al., 2004; Fischer et al., 2020), bend-sensitive resistors (Hanson et al., 2012) and inertial sensors (Levy et al., 2002a,b; MacMillan et al., 2004; Tankus et al., 2017, 2018).

Only a single study has used markerless tracking during MER. London et al. (2021) used the Leap Motion controller (Ultraleap) to allow subjects to control a cursor in real-time with hand movements, allowing the identification of one neural population in the STN that encoded kinematics, and another that responded to unexpected action plan changes.

The ventral intermediate nucleus of the thalamus (VIM) is the primary target of DBS for the treatment of ET (Chopra et al., 2013). While it is known that neural activity in the VIM covaries with tremor and active and passive movements (Lenz et al., 1990, 1994; Zirh et al., 1998; Lenz et al., 2002; Hua and Lenz, 2005; Cajigas et al., 2020), the precise relationship between VIM neurons and kinematics remains unknown. Recent work in mice (Becker and Person, 2019) revealed that the interposed nucleus of the cerebellum is implicated in the precise control of braking at the reach endpoint. The deep motor nuclei of the cerebellum project to the VIM in humans (Roostaei et al., 2014), suggesting that the VIM may also contribute to braking.

To investigate whether VIM neurons encode a brake signal, we designed a study involving a center-out reaching task during awake VIM MER in ET patients (COMIRB #20–2979). We utilized a cart-mounted computer monitor with three cameras (Blackfly USB3) mounted above it that can be positioned over the operating table (Figure 1G). This monitor displayed a central target and eight outer targets (Figure 1H). During the task, the central and outer targets were shown sequentially, cycling through all visible outer targets in a pseudorandom order. Patients were instructed to point at the currently visible target. DLC was used to extract fingertip position (Figure 1I). Regression and Monte Carlo methods were used to determine how VIM unit firing rates related to reach kinematics. These analyses reveal a temporal congruence between VIM neural activity and reach braking, and encoding of fingertip position in VIM units.

To our knowledge this is the first study to examine free reaching movements during human VIM MER, and the first use of DLC in such a setting. The flexibility of DLC enables examination of naturalistic movements that have previously been difficult to study during MER, such as handwriting, tool use and facial expressions.