Franz Hell
Carla Palleis
Jan H. Mehrkens
Thomas Koeglsperger
Kai Bötzel1*- 1Department of Neurology, Ludwig Maximilians University, Munich, Germany
- 2Graduate School of Systemic Neurosciences, Ludwig Maximilians University, Munich, Germany
- 3Department of Translational Neurodegeneration, German Center for Neurodegenerative Diseases (DZNE), Munich, Germany
- 4Department of Neurosurgery, Ludwig Maximilians University, Munich, Germany
Deep brain stimulation has developed into an established treatment for movement disorders and is being actively investigated for numerous other neurological as well as psychiatric disorders. An accurate electrode placement in the target area and the effective programming of DBS devices are considered the most important factors for the individual outcome. Recent research in humans highlights the relevance of widespread networks connected to specific DBS targets. Improving the targeting of anatomical and functional networks involved in the generation of pathological neural activity will improve the clinical DBS effect and limit side-effects. Here, we offer a comprehensive overview over the latest research on target structures and targeting strategies in DBS. In addition, we provide a detailed synopsis of novel technologies that will support DBS programming and parameter selection in the future, with a particular focus on closed-loop stimulation and associated biofeedback signals.
Introduction
Deep brain stimulation (DBS) has become the treatment of choice for movement disorder, such as Parkinson's disease (PD), medically intractable essential tremor (ET) and complicated segmental and generalized dystonia (1). In addition, DBS is increasingly used in other neurological disorders like neuropathic pain and epilepsy, and is being investigated for psychiatric disorders (2), such as obsessive-compulsive disorder, depression and Tourette syndrome and neurodegenerative diseases like Alzheimer's disease (3). DBS is thought to modulate the function of the target region by applying electrical current to the area (4). Recent reviews propose that DBS likely acts through multimodal, non-exclusive mechanisms including immediate neuromodulatory effects on local and network-wide electrical and neurochemical properties, synaptic plasticity and long-term neuronal reorganization, potentially also providing neuroprotective effects and leading to neurogenesis (4–7).
DBS surgery involves implantation of electrodes into one of several target regions and administering electrical current pulses that are generated by an implanted impulse generator. Although the effects of DBS on for example Parkinsonian symptoms and quality of life are generally satisfying (8), the clinical outcome may vary between patients (9) and side effects can be induced (10) due to the stimulation of different functional pathways or structures nearby the original target. New approaches, such as current steering (11) are able to restrict the volume of tissue activated (VTA) (12) and therefore promise a more precise stimulation of neural structures. Improving the initial targeting and later stimulation of specific neural structures and pathways involved in the generation of pathological neural activity as well as avoiding others will be a crucial point for improving the clinical DBS effect and, at the same time, limiting side-effects.
The setting of DBS parameters to optimize therapy is time-consuming and will likely get more complicated with new technological developments, introducing an ever increasing combination of parameters like pulse duration, stimulation frequency, stimulation contacts and so forth. In open loop DBS, which is the current standard protocol, these stimulation parameters are set by a clinician in a trial and error procedure and remain constant until manually updated, irrespective of disease fluctuations. In a closed loop DBS system, a sensor continuously records a feedback signal, a so-called biomarker, which is ideally correlated or causally linked to a clinical symptom. A second major point of interest in DBS research therefore is to develop more sophisticated strategies and automated algorithms on how to program and adjust stimulation parameters in a precise and effective manner.
Target Structures
Contemporary research in humans features investigations into different network structures connected to individual DBS targets and explores structural networks (13, 14) involved in the generation of disease symptoms. There are currently a handful of FDA approved DBS targets, including the subthalamic nucleus (STN), the internal segment of the globus pallidus (GPi), the nucleus ventralis intermedius (ViM), as well as several other investigational targets used for, often more than one for a given disorder or symptom (15). A popular target for DBS in medically intractable tremor, like Parkinsonian or essential tremor is the ViM. Studies using tractography show structural connectivity between ViM and motor cortical, subcortical, brainstem and cerebellar sites (16). Various other research groups show that the dentato-rubro-thalamic tract in the subthalamic region is implicated in tremor control (17) and report successful guidance of DBS surgery based on fiber tracking (18). Comparing STN DBS near tremor frequency in PD and DBS of the ventrolateral thalamus in ET, Cagnan and colleagues describe differences in the response of the behavioral tremor characteristics. They reason that different networks could be involved in essential and Parkinsonian rest tremor and conclude that these differences will be important in developing future strategies for closed loop DBS for tremor control (19).
Studies in dystonia patients have shown that ventral GPi stimulation is more efficient in alleviating dystonic symptoms (20). Using diffusion tensor tractography for investigating the connectivity patterns of different target structures and DBS electrode locations, Rozanski et al. report substantial differences in connectivity of dorsal and ventral GPi. The authors interpret their results in favor of functional differences in the ventral and dorsal GPi and recommend that specific targeting could play an important role in promoting distinct effects of DBS (21).
While PD patients show similar improvement in motor function after GPi- and STN-DBS (22), STN DBS is superior in improving off-drug phase motor symptoms (23). Therefore, the STN is often the preferred target to treat Parkinsonian symptoms, such as bradykinesia, tremor and rigidity. Accola et al. used STN LFP recordings from PD patients to investigate the relation between subthalamic fiber connectivity and oscillatory activity. The dorso-lateral portion of the STN, which shows the highest beta power in the STN, predominantly projected to premotor, motor, but also to associative and limbic areas. Ventral areas are connected to medial temporal regions, like hippocampus and amygdala (13). Recently, Tinkhauser et al. reported that beta oscillations recorded from directional contacts can be used as a predictor of the clinically most efficient contacts for stimulation in patients with PD (24). Various research groups (25–30) suggest that the posterior dorsolateral subthalamic region next to the red nucleus could be a “sweet spot” to help guiding DBS electrode placement in PD. However, the small size of the STN and its proximity to different axonal projections (31) can result in multiple side effects during high-frequency stimulation.
In summary, these results highlight the relevance of targeting specific (sub)-structures and networks in improving the clinical outcome after DBS surgery.
Improving Surgical Planning, Evaluation, and Stimulation
Functional neurosurgery has been driven by technological innovations and DBS has evolved over the years, including new approaches to surgical targeting, evaluation and in the delivery of therapy at the target. For a detailed overview see Gross and McDougal (32). Improving and personalizing the targeting of specific (sub)-structures and avoiding others will be crucial for improving the clinical effect and limiting stimulation induced side-effects. New evolving technologies are turning away from classical cylindrical electrodes toward directional stimulating leads. The VANTAGE study, a multi-center study investigating the benefits of using segmented electrodes and multiple-source axially asymmetric directional DBS could show that such an approach leads to similar therapeutic effects as the standard approach without steering. A follow up study reports that axially asymmetric current can reduce adverse effects as well as efficacy thresholds in a highly individual manner, while also expanding the therapeutic window as compared to ring-mode DBS (33).
New software now allows for a patient-specific reconstruction of DBS leads based on MRI and post-operative CT imaging, the reconstruction of nuclei and fiber tracts adjacent to stimulation sites and the mapping of intra- and perioperative electrophysiological recordings (34, 35). For instance, Lead-DBS, now available in version 2.0, is a semi-automated toolbox to model deep brain stimulation electrode locations based on structural and neurophysiological imaging (34, 36). This toolbox now contains PaCER, a fully automated tool for electrode trajectory and contact reconstruction (37). Lauro et al. provide the open source software systems DBSproc and DBStar for clinical research which co-register CT and MR data for individual target localization and diffusion tractographic analysis from automatically detected DBS contacts (38, 39). On the industry side, Boston Scientific bought Cicerone DBS (40), a platform for stereotactic neurosurgical planning, recording, and visualization for DBS initially developed by the McIntyre lab and turned it into the commercial available software GUIDE. Medtronic initially offered comparable software called Optivize and recently replaced it with its sequel, SureTune 3. The company Brainlab, which has recently partnered with Boston Scientific to develop GUIDE XT, also offers a DBS surgery planning software called ELEMENTS which enables displaying target structures, fiber tracts as well as electrode trajectories.
The VTA is a concept to model the spatial dimension of stimulation for a given set of stimulation parameters (12, 41–43). It can be calculated from individual therapeutic impedance and stimulation energy (total electrical energy delivered, TEED) (44). With a 3D brain atlas and MRI data, the VTA can provide an approximate reconstruction of brain structures surrounding the DBS electrode as a 3D activate/non-activate image. A clinical application of the prediction of the spatial extent of VTA was reported to be helpful in optimizing DBS parameter settings in PD patients (45). However, the application of VTA remains limited due to a lack of impedance calculation in the model and differential strength-duration curves of the response of axons with different diameters because VTAs are derived from volume conductor models with a homogenous and isotropic tissue medium and the axonal trajectories are assumed to be perfectly straight and perpendicular to the electrode shaft, as for example in DBSproc and DBStar (38, 39).
As the electric fields generated during multi-contact stimulation become more complex, new approaches are needed minimize the prediction error for VTA and to quantify axonal and pathway responses in patients-specific models (46, 47). The clinical software StimVision provides another algorithm to calculate the VTA using the artificial neural network technique to facilitate tractography-based DBS targeting (48). Tractography is a modeling technique used to visually represent nerve tracts in 3D space using data collected by diffusion MRI (49). Results from tractography can be combined with post-operative computational modeling to determine the VTA based on electrode contacts, as the implanted electrodes can influence activity not only in gray matter structures but might also influence activity in surrounding white matter structures, thereby potentially influencing networks (50–52). The influence of fiber pathways in DBS has been shown with blood flow, glucose metabolism and blood oxygenation level dependence (BOLD) imaging techniques in multiple studies (53–55), supporting the hypothesis that DBS affects larger neuronal networks with subsequent downstream axonal activation. Sweet and colleagues combined results from tractography with post-operative computational modeling in patients with tremor-dominant PD identifying that the most efficient VTA stimulates the dentatothalamic fiber tract. As mentioned above, this tract probably plays an important role in the occurrence of tremor in PD and targeting it may alleviate tremor symptoms (43).
Advancements in imaging methods, such as ultra-high field MRI and new learning algorithms (34, 56–59) promise to refine our conception and understanding of different neural structures and their wiring in health and disease and will support the investigation of personalized target structures, thus possibly individualizing DBS surgery.
New Sensing Devices and Feedback Signals
Today, DBS systems stimulate in an open-loop manner, meaning that stimulation parameters are pre-programmed and are not responsive to changes in the patient's clinical symptoms or in the underlying physiological activity. Although open-loop stimulation is state of the art, limitations like overall efficiency, reduction of efficiency over time or side-effects have become more obvious with growing clinical experience. DBS therapy adjustment also remains time-consuming, requiring clinicians to evaluate numerous combinations of stimulation parameters in order to achieve the optimal outcome. Selecting the right combination among many possibilities can have a major impact on the therapeutic effect (60). DBS practice currently requires patients to follow-up for months post-operatively to optimize the clinical effect of DBS. Disease and patient specific biomarkers could ideally help optimize therapy and help finding the right DBS parameter.
Medtronic now offers the implantable and rechargeable neurostimulator Medtronic® ACTIVA RC + S, a research system following the Activa PC + S system, which records electrophysiological signals from the implanted DBS electrodes and also offers inertial measurements. New miniature implants (61) with names like Neural dust (62) or Neurograins (63) will push the boundary of signal collection even further and ultimately promise to provide read and stimulation capabilities with a far greater spatial and temporal detail than available at present. There now are several companies actively pursuing brain computer interface technology by developing new neural implants, ranging from traditional medical device manufacturers like Medtronic, St. Jude Medical or Boston Scientific to tech start-ups like Neuralink, Kernel or Cortera, which in part work in close cooperation with several research institutes and are driven by funding from the DARPA program.
Looking forward, adaptive closed-loop stimulation systems that integrate feedback signals will ideally be able to rapidly respond to real-time patient needs and make human programming unnecessary (64). NeuroPace (California, USA) for example already provides a responsive neurostimulation system (RNS) for closed-loop cortical stimulation with FDA-approval in patients with drug-resistant epilepsy. It is capable of continuously sensing electrocorticography (ECoG) potentials (65). When recognizing a seizure-related pattern, the stimulator is activated to stop the seizure and store the ECoG potentials, date and time of seizure occurrence.
Optimally, biomarkers for adaptive closed loop DBS should be usable continuously after DBS implantation to make them applicable for clinical practice. Local field potentials and network connectivity measures based on electrophysiological signals with their high temporal resolution can already be measured with sensing DBS electrodes or other implanted neural sensors and hold great promise as biomarkers.
Biomarkers and Control Mechanisms
Regarding closed loop adaptive DBS, a distinction has to be made between feedback signals (biomarkers) and mechanisms of control. A biomarker describes a correlative or causal relation to a clinical symptom. Adaptive control mechanisms then define how to adjust stimulation based on the evolution of biomarkers.
Biomarkers
Electrophysiological measurements
Recordings of LFPs in the basal ganglia of PD patients show oscillations at several frequencies, including oscillations at low frequencies in the delta and theta band (1~7 Hz), alpha and beta band (8–35 Hz), gamma band (35–200 Hz) and high frequency oscillations (>200 Hz). It has been demonstrated that the beta activity amplitude is correlated with motor symptom severity without medication (66–68). Moreover, it has been reported that the reduction of rigidity and bradykinesia is correlated with a decrease in beta activity (69, 70). In line with this, STN DBS and dopaminergic medication has been shown to attenuate beta activity locally (71–75), while the degree of beta activity suppression has been shown to correlate with improvement in Parkinsonian motor symptoms (71, 76). Whereas, exaggerated beta activity is associated with bradykinesia and rigidity, dyskinesia symptoms are reported to be linked to increases in low (4–8 Hz) and gamma frequencies (60–90 Hz) (77, 78), akin to oscillatory activity observed during normal movement (79–82). High frequency oscillations (HFO), which are reported to be coupled to the phase of beta oscillations, are another promising biomarker associated with Parkinsonian symptoms, such as bradykinesia, rigidity as well as tremor, even in the ON medication state (83–85). They are typically found at ~250 Hz, while not being attenuated by dopaminergic medication, but rather shifted toward higher frequencies at 350 Hz (84–86).
Early approaches using local field potentials (LFP) as feedback signals for adaptive DBS incorporated the beta frequency amplitude as a mechanism to trigger the stimulation (87) demonstrated clinical improvement of symptoms compared to standard DBS. An approach by Meidahl et al. targets potentially pathological long beta bursts sparing supposedly functionally important short-term beta bursts (88, 89). Several other oscillatory biomarkers, such as pathological cross-frequency coupling (85, 90) or pathological coherence of neural activity between cortical and subcortical structures (91) have been reported to correlate with clinical symptoms and are discussed as potential feedback signals. Despite early success, challenges have yet to be overcome. Beta power in the STN for example correlates with rigidity and bradykinesia, but not with tremor (92, 93), which is linked to field potentials at tremor frequency. PD patients for example often show heterogenous clinical symptoms, a single, one-dimensional feedback signal might be only useful to a certain degree. Body measurements using electromyography or kinematic sensors allowing for the assessment of symptom severity and behavior could be a promising additional feedback source for adaptive DBS. For instance, Cagnan and colleagues stimulated patients with essential tremor and thalamic electrodes, while recording tremor amplitude and phase with inertial sensor units. They report that the amplitude of the tremor was modulated depending on the phase relative to the tremor cycle, at which stimulation pulses were delivered (94). Most neural biomarkers like beta frequency oscillations are multifaceted and not only linked to clinical symptoms, but also modulated during normal behavior like movement or cognition (95, 96) and are associated with medication (71, 97). Although biomarkers like beta activity seem to be stable months after DBS surgery (98, 99), it is also conceivable that they evolve with disease progression, as they are correlated with symptom severity (67), which naturally increases over time in neurodegenerative diseases.
The use of electrophysiological biomarkers in aDBS is also restricted due to an often unfavorably low signal-to-noise ratio and interference with external artifacts like movement, speaking and cognition (100). Also, stimulation can lead to artifacts when sensing is done near the site of stimulation, e.g., the sensing of β-bands in the STN with e.g., Activa PC + S can be contaminated by stimulation. This may be avoided by using ECoG sensing (101). ECoG is another invasive electrophysiological biomarker which directly records electrical potentials associated with brain activity from the cortex. When using ECoG as a biomarker in aDBS the sensing strip is implanted subdurally over the primary motor cortex during the same procedure as the electrode implantation subcortically. Gamma band activity (60–90 Hz) for example is associated with dyskinesia in PD patients and can therefore be used as a feedback signal to trigger stimulation (101).
For a detailed overview of oscillatory features related to pathological and physiological states in DBS patients, see Neumann et al. (102).
Neurochemical sensing
Neuronal sensor devices that detect local alterations in neurotransmitter release in response to DBS have been developed. The stimulation-evoked changes that resemble physiological neurotransmitter release are associated with the therapeutic effect of DBS (103). Grahn et al. developed a device that detects changes in dopamine concentration in rodents to adapt stimulation parameters (104). Lee et al. have developed a wirelessly controlled device, WINCS Harmoni®, which can measure in vivo neurotransmitter concentration across multiple anatomical targets using implanted neurochemical sensors. These devices provide real-time neurochemical feedback for closed loop control (105). Until now, the method has been used in preclinical DBS studies, but it is a promising tool for a better understanding and future improvement of a clinical application of closed loop DBS.
External mechanistic sensors
External wearable devices, such as accelerometers or EMG sensors can be used to infer symptoms and symptom severity like rigidity, bradykinesia and gait disorders (106, 107). Studies show that the measurement of tremor with accelerometers that adjust the stimulation frequency to tremor frequency lead to a better clinical result than conventional stimulation in patients with essential tremor (52, 94). In PD, the severity of motor dysfunction can be measured with a wireless external sensor device which is integrated into a smart glove containing two touch sensors, two 3D-accelerometers and a force sensor to assess tremor, rigidity and bradykinesia of hand and arm (108). Heldman et al. devised software to automatically optimize stimulation settings based upon objective motion sensor-based motor assessments. To assess symptom severity, a motion sensor was placed on the index finger of the more affected hand. The software then guided a procedure during which stimulation on each contact was iteratively increased. This was followed by an automated assessment of tremor and bradykinesia severity. After completing assessments at each setting, a software algorithm determined stimulation settings, leading to improved tremor and bradykinesia scores by an average of 35.7% (107, 109).
Control Mechanisms
Beta threshold targeting
One of the earliest approaches to adaptive closed loop DBS was beta threshold targeting. When the amplitude of oscillatory activity in the β-band exceeds a defined threshold, stimulation is turned on. It has already been shown that this approach can improve the therapeutic effect compared to standard DBS (87). Alternatively to threshold targeting, excessive β-synchronization in PD patients may selectively be regulated via aDBS by targeting pathological long β bursts while leaving possibly functionally relevant short bursts of β activity unaffected (88). However, as described above, one problem of this approach is that not only beta oscillations but also beta oscillatory characteristics, such as burst length are not only related to symptom severity, but also to medication and behavior (75, 110).
Noise cancellation
Cagnan et al. suggest a tool to detect the patient's tremor with an accelerometer attached to the affected hand, as described above. Using the effect of noise cancellation, a control mechanism based on this external mechanistic sensor switches on the thalamus stimulation in specific phases of the essential tremor (52). In this work the modulation of tremor turned out to depend on the phase of stimulation relative to the tremor cycle. However, only stimulation during the first half of the tremor cycle resulted in a reduction of tremor whereas during the second half of the tremor cycle harmonics in tremor were inducted (52). Also in PD patients, the effect of noise cancellation was used to cancel cortical oscillations within the tremor network with non-invasive transcranial alternating current stimulation (tACS) which can reduce the amplitude of resting tremor by 50% (111).
Stimulation on demand
Measuring biomarkers in real-time can be used for stimulation on demand in aDBS. Herron et al. used cortical electrodes sensing β-band desynchronization in ET patients when a movement was started. This desynchronization then triggered the stimulation to reduce the tremor while stimulation was switched off in resting state (112). Due to a delay in stimulation initiation, tremor at the beginning of a movement could not be prevented. One way to improve this would be if one is able to predict movement before it occurs.
Coordinated reset stimulation
An alternative stimulation protocol is the temporal stimulation pattern coordinated reset stimulation for research application (113). Abnormal neuronal synchrony in neurological diseases can be addressed by coordinated reset stimulation that delivers brief high-frequency pulse trains through different stimulation contacts of the DBS lead to reset abnormal synchronization. In PD the basal ganglia structures STN and GPe are known to generate rhythmic synchronized oscillations which are associated with PD symptoms(114). Coordinated reset stimulation can decrease these abnormal synchronous beta oscillations and hence improve bradykinesia and rigidity (115).
Limitations and Future Perspectives
Moving From Invasive to Non-invasive DBS
Although the implantation of DBS electrodes is a well-established procedure in movement disorders, it comes along with surgical risks and complications. Thus, a non-invasive approach could be a future direction. Non-invasive aDBS is proposed by Grossman et al. who have developed an experimental strategy in mice to target deeply situated neurons without manipulating the overlying cortex by applying high-frequency oscillating fields in different locations outside the brain (116). The interference between two applied fields cancels out the high-frequency activity, while an oscillation of low frequency corresponding to the difference between the two frequencies can emerge. With this low frequency neurons situated deeply in the hippocampus can be activated. The suggested approach is limited by the size of human brain that is much bigger than mouse brain and hence, more difficult to target deeply located structures, and by whether neural networks in the stimulation paths remain unaffected also in a larger brain (117). Another non-invasive approach is optogenetic stimulation, which was developed over the last decade. Optogenetics can selectively activate neurons deep in the rodent brain by using light to control neuronal ion channels in vivo. Thus, neural circuits can be manipulated by precise excitation and inhibition of specific circuit elements, moving from invasive toward non-invasive DBS (118, 119). Currently, optogenetics still require a chronically implanted optical fiber, hence, it is not yet a completely non-invasive technique.
However, the non-invasive approaches still need to be investigated much further. So far they have only been studied in animal models.
Future Perspectives
Most existing approaches to adaptive DBS so far have in common that they are carefully engineered based on a core principle and allow for a specific action given a certain signal. However, these approaches do not allow for learning optimal individual signal properties and control algorithms. In addition, each biomarker and control mechanism has its specific drawback as discussed above.
As a future direction, latent features derived from different signal sources could be used in parallel to establish a feedback driven stimulation algorithm based on the analysis of behavioral and physiological data and a suitable control mechanism. By integrating parameters derived from different sources, such as kinematic and electrophysiological measurements and other sensor like electromyography, patient state and disease symptoms severity and underlying neural activity could be ultimately learned and classified end to end (102, 120–122), using machine learning algorithms (Figure 1).
Figure 1. Schematic of general adaptive closed loop DBS for adaptive adjustment of deep brain stimulation (DBS) parameters based upon real time patient measurements, such as electrophysiological signals (e.g., LFP, ECoG, EMG), neurochemical parameters and behavioral measurements and machine learning. First, latent features from different possible signal sources are learned using machine learning approaches to extract behavioral (clinical) states (e.g., bradykinesia, rigidity, tremor) and corresponding and predictive latent neural states (e.g., beta and high frequency oscillations). Then, actual states are compared with ideal states to compute a reward and stimulation parameters (e.g., VTA, stimulation frequency, etc.) adjusted and finally learned via reinforcement learning (Q-Learning is shown as an example). In this closed-loop paradigm, the stimulation parameters (actions) are adjusted within clinical limits based on the reward and the extracted latent states.
In case that physiological and behavioral features, describing the neural and clinical state of the patient, can be reliably decoded and ideally predicted from measurements, reinforcement learning could be another option to learn and optimally control stimulation paradigms and optimize the clinical state (Figure 1). Reinforcement learning can provide optimal control in an environment with unknown transition probabilities (123). In reinforcement learning, an agent, in this case the DBS stimulation controller interacts with an uncertain environment, i.e., stimulating a mixture of neural structures with certain stimulation parameters with the goal to maximize a numerical long term reward, in this case the (long term) clinical improvement of the patient. Through the learned policy after training the controller ideally has identified the right stimulation action in every state (124).
A simple version of this idea could be realized in patients with tremor dominant PD. The amplitude of the tremor can be measured with kinematic sensors and then be used to describe the clinical state of the patient. Such a signal could then serve as a reward signal for reinforcement learning, with the reward simply being the difference between optimal clinical state (no tremor amplitude) and actual clinical state (actual tremor amplitude). With such an approach, the optimal stimulus could be learned and adjusted based on feedback signals, closing the loop. Alternative stimulation protocols and parameters (such as electrode contact, VTA, pulse-frequency, -width, -amplitude, -shape, timing relative to neural activity, etc.) could then be evaluated within a clinically acceptable range of stimulation energy. However, the vast amount of free parameters in DBS programming introduces a potentially very large search space to evaluate during reinforcement learning, even when constraining the search space to clinically acceptable parameters. Algorithms for reinforcement learning are commonly either model-free or model-based. While in model-free learning, the agent simply relies on trial-and-error experience to learn a policy that optimizes immediate and future reward, in model-based learning, the agent exploits previously learned lessons (125). Although model-free deep reinforcement learning algorithms are suited for learning a wide range of applications, they often require millions of training iterations to achieve good performance (126, 127), rendering this approach inappropriate for adaptive DBS trials in humans. In model-based reinforcement learning, experience is used to construct a model of the world, describing the transitions between states and associated outcomes, while suitable actions are chosen by searching or planning in this world model (128). To learn such models in the first place, however, a large number of training trials would also likely be required. Possibly animal models could help pioneering such an approach (129). Ultimately, only interventional studies can prove causal relationships and in this case the effects of adaptive deep brain stimulation on the clinical and overall state of the patient. However, applying countless experimental perturbations, which are necessary to gather enough observational data to learn from, can be costly and time consuming, even when done in animal models. Inferring the causal structure of brain networks from neuroimaging data is an important goal in neuroscience (130, 131) and various methods, such as Granger causality (132, 133), dynamic causal modeling (134, 135), structural equation modeling (136, 137) and causal Bayesian networks (138, 139) have been developed to infer causal relations from brain imaging data. Recently, van Wijk et al. applied dynamic causal modeling to explore the cortical-basal ganglia-thalamus loop in patients with PD and to study pathways that contribute to the suppression of beta oscillations induced by dopaminergic medication (140). Also recently, Bogacz et al. described a coupled oscillator model to predict the effects of deep brain stimulation (141). Ideally, causal inference methods based on i.e., causal Bayesian networks could also help give testable predictions on the effects of external manipulations (142), such as the effects of deep brain stimulation. In this way, different adaptive approaches could be explored or learned in silico and the number of interventional studies, that are required to establish an approach, could be reduced substantially (143).
Searching Strategy
This review is based on expert opinions and does not follow a systematic searching strategy.
Author Contributions
FH, JM, TK, and KB conceived the project. FH and CP did the literature search and wrote the manuscript. TK, JM, and KB wrote the manuscript.
Funding
This work has been supported by the Lüneburg Heritage for Parkinson's disease research.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
FH was supported by the Lüneburg heritage.
References
1. Fasano A, Lozano AM. Deep brain stimulation for movement disorders: 2015 and beyond. Curr Opin Neurol. (2015) 28:423–36. doi: 10.1097/WCO.0000000000000226
2. Holtzheimer PE, Mayberg HS. Deep brain stimulation for psychiatric disorders. Annu Rev Neurosci. (2011) 34:289–307. doi: 10.1146/annurev-neuro-061010-113638
3. Aldehri M, Temel Y, Alnaami I, Jahanshahi A, Hescham S. Deep brain stimulation for Alzheimer's disease: an update. Surg Neurol Int. (2018) 9:58. doi: 10.4103/sni.sni_342_17
4. Ashkan K, Rogers P, Bergman H, Ughratdar I. Insights into the mechanisms of deep brain stimulation. Nat Rev Neurol. (2017) 13:548–54. doi: 10.1038/nrneurol.2017.105
5. Chiken S, Nambu A. Disrupting neuronal transmission: mechanism of DBS? Front Syst Neurosci. (2014) 8:33. doi: 10.3389/fnsys.2014.00033
6. Chiken S, Nambu A. Mechanism of deep brain stimulation: inhibition, excitation, or disruption? Neuroscientist. (2015) 22:313–22. doi: 10.1177/1073858415581986
7. Herrington TM, Cheng JJ, Eskandar EN. Mechanisms of deep brain stimulation. J Neurophysiol. (2016) 115:19–38. doi: 10.1152/jn.00281.2015
8. Deuschl G, Paschen S, Witt K. Chapter 10 – clinical outcome of deep brain stimulation for Parkinson's disease. Handb Clin Neurol. (2013) 116:107–28. doi: 10.1016/B978-0-444-53497-2.00010-3
9. Merola A, Zibetti M, Angrisano S, Rizzi L, Ricchi V, Artusi CA, et al. Parkinson's disease progression at 30 years: a study of subthalamic deep brain-stimulated patients. Brain A J Neurol. (2011) 134:2074–84. doi: 10.1093/brain/awr121
10. Højlund A, Petersen MV, Sridharan KS, Østergaard K. Worsening of verbal fluency after deep brain stimulation in Parkinson's disease: a focused review. Comput Struct Biotechnol J. (2017) 15:68–74. doi: 10.1016/j.csbj.2016.11.003
11. Timmermann L, Jain R, Chen L, Maarouf M, Barbe MT, Allert N, et al. Multiple-source current steering in subthalamic nucleus deep brain stimulation for Parkinson's disease (the VANTAGE study): a non-randomised, prospective, multicentre, open-label study. Lancet Neurol. (2015) 14:693–701. doi: 10.1016/S1474-4422(15)00087-3
12. Butson CR, Cooper SE, Henderson JM, McIntyre CC. Patient-specific analysis of the volume of tissue activated during deep brain stimulation. NeuroImage. (2007) 34:661–70. doi: 10.1016/j.neuroimage.2006.09.034
13. Accolla EA, Herrojo Ruiz M, Horn A, Schneider GH, Schmitz-Hübsch T, Draganski B, et al. Brain networks modulated by subthalamic nucleus deep brain stimulation. Brain. (2016) 139:2503–15. doi: 10.1093/brain/aww182
14. Anderson D, Beecher G, Ba F. Deep brain stimulation in Parkinson's disease: new and emerging targets for refractory motor and nonmotor symptoms. Parkinsons Dis. (2017) 2017:5124328. doi: 10.1155/2017/5124328
15. Johnson MD, Miocinovic S, McIntyre CC, Vitek JL. Mechanisms and targets of deep brain stimulation in movement disorders. Neurotherapeutics. (2008) 5:294–308. doi: 10.1016/j.nurt.2008.01.010
16. Klein JC, Barbe MT, Seifried C, Baudrexel S, Runge M, Maarouf M, et al. The tremor network targeted by successful VIM deep brain stimulation in humans. Neurology. (2012) 78:787–95. doi: 10.1212/WNL.0b013e318249f702
17. Groppa S, Herzog J, Falk D, Riedel C, Deuschl G, Volkmann J. Physiological and anatomical decomposition of subthalamic neurostimulation effects in essential tremor. Brain. (2014) 137:109–21. doi: 10.1093/brain/awt304
18. Coenen VA, Varkuti B, Parpaley Y, Skodda S, Prokop T, Urbach H, et al. Postoperative neuroimaging analysis of DRT deep brain stimulation revision surgery for complicated essential tremor. Acta Neurochir (Wien). (2017) 159:779–87. doi: 10.1007/s00701-017-3134-z
19. Cagnan H, Little S, Foltynie T, Limousin P, Zrinzo L, Hariz M, et al. The nature of tremor circuits in parkinsonian and essential tremor. Brain. (2014) 137:3223–34. doi: 10.1093/brain/awu250
20. Tisch S, Zrinzo L, Limousin P, Bhatia KP, Quinn N, Ashkan K, et al. Effect of electrode contact location on clinical efficacy of pallidal deep brain stimulation in primary generalised dystonia. J Neurol Neurosurg Psychiatr. (2007) 78:1314–9. doi: 10.1136/jnnp.2006.109694
21. Rozanski VE, Vollmar C, Cunha JP, Tafula SMN, Ahmadi SA, Patzig M, et al. Connectivity patterns of pallidal DBS electrodes in focal dystonia: a diffusion tensor tractography study. NeuroImage. (2014) 84:435–42. doi: 10.1016/j.neuroimage.2013.09.009
22. Follett KA, Weaver FM, Stern M, Hur K, Harris CL, Luo P, et al. Pallidal versus subthalamic deep-brain stimulation for Parkinson's disease. N Engl J Med. (2010) 362:2077–91. doi: 10.1056/NEJMoa0907083
23. Odekerken VJJ, Boel JA, Schmand BA, de Haan RJ, Figee M, van den Munckhof P, et al. GPi vs STN deep brain stimulation for Parkinson disease: three-year follow-up. Neurology. (2016) 86, 755–61. doi: 10.1212/WNL.0000000000002401
24. Tinkhauser G, Pogosyan A, Debove I, Nowacki A, Shah SA, Seidel K, et al. Directional local field potentials: a tool to optimize deep brain stimulation. Mov Disord. (2018) 33:159–64. doi: 10.1002/mds.27215
25. Sauleau P, Raoul S, Lallement F, Rivier I, Drapier S, Lajat Y, et al. Motor and non motor effects during intraoperative subthalamic stimulation for Parkinson's disease. J Neurol. (2005) 252:457–64. doi: 10.1007/s00415-005-0675-5
26. Coenen VA, Prescher A, Schmidt T, Picozzi P, Gielen FLH. What is dorso-lateral in the subthalamic Nucleus (STN)?–a topographic and anatomical consideration on the ambiguous description of today's primary target for deep brain stimulation (DBS) surgery Acta Neurochir (Wien). (2008) 150:1163–5. doi: 10.1007/s00701-008-0136-x
27. Vertinsky AT, Coenen VA, Lang DJ, Kolind S, Honey CR, Li D, et al. Localization of the subthalamic nucleus: optimization with susceptibility-weighted phase MR imaging. Am J Neuroradiol. (2009) 30:1717–24. doi: 10.3174/ajnr.A1669
28. Caire F, Maubon A, Moreau JJ, Cuny E. The mamillothalamic tract is a good landmark for the anterior border of the subthalamic nucleus on axial MR images. Stereotact Funct Neurosurg. (2011) 89:286–90. doi: 10.1159/000329356
29. Welter ML, Schüpbach M, Czernecki V, Karachi C, Fernandez-Vidal S, Golmard JL, et al. Optimal target localization for subthalamic stimulation in patients with Parkinson disease. Neurology. (2014) 82:1352–61. doi: 10.1212/WNL.0000000000000315
30. Horn A, Neumann WJ, Degen K, Schneider GH, Kühn AA. Toward an electrophysiological “sweet spot” for deep brain stimulation in the subthalamic nucleus. Hum Brain Mapp. (2017) 38:3377–90. doi: 10.1002/hbm.23594
31. Güngör A, Baydin SS, Holanda VM, Middlebrooks EH, Isler C, Tugcu B, et al. Microsurgical anatomy of the subthalamic nucleus: correlating fiber dissection results with 3-T magnetic resonance imaging using neuronavigation. J Neurosurg. (2018) 1:1–17. doi: 10.3171/2017.10.JNS171513
32. Gross RE, McDougal ME. Technological advances in the surgical treatment of movement disorders. Curr Neurol Neurosci Rep. (2013) 13:371. doi: 10.1007/s11910-013-0371-2
33. Steigerwald F, Müller L, Johannes S, Matthies C, Volkmann J. Directional deep brain stimulation of the subthalamic nucleus: a pilot study using a novel neurostimulation device. Mov Disord. (2016) 31:1240–3. doi: 10.1002/mds.26669
34. Horn A, Kühn AA. Lead-DBS: a toolbox for deep brain stimulation electrode localizations and visualizations. NeuroImage. (2015) 107:127–35. doi: 10.1016/j.neuroimage.2014.12.002
35. Duchin Y, Shamir RR, Patriat R, Kim J, Vitek JL, Sapiro G, et al. Patient-specific anatomical model for deep brain stimulation based on 7 Tesla, MRI. PLoS ONE. (2018) 13:e0201469. doi: 10.1371/journal.pone.0201469
36. Horn A, Li N, Dembek TA, Kappel A, Boulay C, Ewert S, et al. Lead-DBS v2: towards a comprehensive pipeline for deep brain stimulation imaging. NeuroImage. (2019) 184:293–316. doi: 10.1016/j.neuroimage.2018.08.068
37. Husch A, Petersen VM, Gemmar P, Goncalves J, Hertel F. PaCER–a fully automated method for electrode trajectory and contact reconstruction in deep brain stimulation. NeuroImage Clin. (2018) 17:80–9. doi: 10.1016/j.nicl.2017.10.004
38. Lauro PM, Vanegas-Arroyave N, Huang L, Taylor PA, Zaghloul KA, Lungu C, et al. DBSproc: an open source process for DBS electrode localization and tractographic analysis. Hum Brain Mapp. (2016) 37:422–33. doi: 10.1002/hbm.23039
39. Lauro PM, Lee S, Ahn M, Barborica A, Asaad WF. DBStar: an open-source tool kit for imaging analysis with patient-customized deep brain stimulation platforms. Stereotact Funct Neurosurg. (2018) 96:13–21. doi: 10.1159/000486645
40. Miocinovic S, Noecker AM, Maks CB, Butson CR, McIntyre CC. Cicerone: stereotactic neurophysiological recording and deep brain stimulation electrode placement software system. Acta Neurochir Suppl. (2007) 97:561–7. doi: 10.1007/978-3-211-33081-4_65
41. Butson CR, McIntyre CC. Tissue and electrode capacitance reduce neural activation volumes during deep brain stimulation. Clin Neurophysiol. (2005) 116:2490–500. doi: 10.1016/j.clinph.2005.06.023
42. Schmidt C, Grant P, Lowery M, van Rienen U. Influence of uncertainties in the material properties of brain tissue on the probabilistic volume of tissue activated. IEEE Trans Biomed Eng. (2013) 60:1378–87. doi: 10.1109/TBME.2012.2235835
43. Sweet JA, Walter BL, Gunalan K, Chaturvedi A, McIntyre CC, Miller JP. Fiber tractography of the axonal pathways linking the basal ganglia and cerebellum in Parkinson disease: implications for targeting in deep brain stimulation. J Neurosurg. (2014) 120:988–96. doi: 10.3171/2013.12.JNS131537
44. Mädler B, Coenen VA. Explaining clinical effects of deep brain stimulation through simplified target-specific modeling of the volume of activated tissue. Am J Neuroradiol. (2012) 33:1072–80. doi: 10.3174/ajnr.A2906
45. Frankemolle AMM, Wu J, Noecker AM, Voelcker-Rehage C, Ho JC, Vitek JL, et al. Reversing cognitive-motor impairments in Parkinson's disease patients using a computational modelling approach to deep brain stimulation programming. Brain. (2010) 133:746–61. doi: 10.1093/brain/awp315
46. Chaturvedi A, Luján JL, McIntyre CC. Artificial neural network based characterization of the volume of tissue activated during deep brain stimulation. J Neural Eng. (2013) 10:056023. doi: 10.1088/1741-2560/10/5/056023
47. Gunalan K, Howell B, McIntyre CC. Quantifying axonal responses in patient-specific models of subthalamic deep brain stimulation. NeuroImage. (2018) 172:263–77. doi: 10.1016/j.neuroimage.2018.01.015
48. Noecker AM, Choi KS, Riva-Posse P, Gross RE, Mayberg HS, McIntyre CC. StimVision software: examples and applications in subcallosal cingulate deep brain stimulation for depression. Neuromodulation. (2018) 21:191–6. doi: 10.1111/ner.12625
49. Hagmann P, Jonasson L, Maeder P, Thiran JP, Wedeen VJ, Meuli R. Understanding diffusion MR imaging techniques: from scalar diffusion-weighted imaging to diffusion tensor imaging and beyond. Radiographics. (2006) 26:S205–23. doi: 10.1148/rg.26si065510
50. McConnell GC, So RQ, Hilliard JD, Lopomo P, Grill WM. Effective deep brain stimulation suppresses low-frequency network oscillations in the basal ganglia by regularizing neural firing patterns. J Neurosci. (2012) 32:15657–68. doi: 10.1523/JNEUROSCI.2824-12.2012
51. Davidson CM, de Paor AM, Cagnan H, Lowery MM. Analysis of oscillatory neural activity in series network models of Parkinson's disease during deep brain stimulation. IEEE Trans Biomed Eng. (2016) 63:86–96. doi: 10.1109/TBME.2015.2475166
52. Cagnan H, Pedrosa D, Little S, Pogosyan A, Cheeran B, Aziz T, et al. Stimulating at the right time: phase-specific deep brain stimulation. Brain. (2017) 140:132–45. doi: 10.1093/brain/aww286
53. Grafton ST, Turner RS, Desmurget M, Bakay R, Delong M, Vitek J, et al. Normalizing motor-related brain activity: subthalamic nucleus stimulation in Parkinson disease. Neurology. (2006) 66:1192–9. doi: 10.1212/01.wnl.0000214237.58321.c3
54. Kahan J, Mancini L, Urner M, Friston K, Hariz M, Holl E, et al. Therapeutic subthalamic nucleus deep brain stimulation reverses cortico-thalamic coupling during voluntary movements in Parkinson's disease. PLoS ONE. (2012) 7:e50270. doi: 10.1371/journal.pone.0050270
55. Figee M, Wielaard I, Mazaheri A, Denys D. Neurosurgical targets for compulsivity: what can we learn from acquired brain lesions? Neurosci Biobehav Rev. (2013) 37:328–39. doi: 10.1016/j.neubiorev.2013.01.005
56. Milletari F, Navab N, Ahmadi SA. V-net: fully convolutional neural networks for volumetric medical image segmentation. In: Proceedings–2016 4th International Conference on 3D Vision, 3DV 2016. Stanford, CA (2016). p. 565–71.
57. Amoroso N, La Rocca M, Bruno S, Maggipinto T, Monaco A, Bellotti R, et al. Brain structural connectivity atrophy in Alzheimer's disease. arXiv:170902369v2. (2017) [physics.med-ph] 1–16. Available online at: http://arxiv.org/abs/1709.02369 (accessed March 21, 2019).
58. Shen D, Wu G, Suk HI. Deep learning in medical image analysis. Annu Rev Biomed Eng. (2017) 19:221–48. doi: 10.1146/annurev-bioeng-071516-044442
59. Stephan KE, Schlagenhauf F, Huys QJM, Raman S, Aponte EA, Brodersen KH, et al. Computational neuroimaging strategies for single patient predictions. NeuroImage. (2017) 145:180–99. doi: 10.1016/j.neuroimage.2016.06.038
60. Amon A, Alesch F. Systems for deep brain stimulation: review of technical features. J Neural Transm. (2017) 124:1083–91. doi: 10.1007/s00702-017-1751-6
61. Seo D, Neely RM, Shen K, Singhal U, Alon E, Rabaey JM, et al. Wireless recording in the peripheral nervous system with ultrasonic neural dust. Neuron. (2016) 91:529–39. doi: 10.1016/j.neuron.2016.06.034
62. Neely RM, Piech DK, Santacruz SR, Maharbiz MM, Carmena JM. Recent advances in neural dust: towards a neural interface platform. Curr Opin Neurobiol. (2018) 50:64–71 doi: 10.1016/j.conb.2017.12.010
63. Cai H, Lokhandwala M, Zhu J, Kilfoyle C, Lee J, Larson L, et al. A software-defined radio for wireless brain implants network. In: Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. New York, NY: ACM (2018). p. 852–4. doi: 10.1145/3241539.3267727
64. Habets JGV, Heijmans M, Kuijf ML, Janssen MLF, Temel Y, Kubben PL. An update on adaptive deep brain stimulation in Parkinson's disease. Mov Disord. (2018) 33:1834–43. doi: 10.1002/mds.115
65. Thomas GP, Jobst BC. Critical review of the responsive neurostimulator system for epilepsy. Med Devices. (2015) 8:405–11. doi: 10.2147/MDER.S62853
66. Levy R, Hutchison WD, Lozano A M, Dostrovsky JO. Synchronized neuronal discharge in the basal ganglia of parkinsonian patients is limited to oscillatory activity. J Neurosci. (2002) 22:2855–61. doi: 10.1523/JNEUROSCI.22-07-02855.2002
67. Neumann WJ, Degen K, Schneider GH, Brücke C, Huebl J, Brown P, et al. Subthalamic synchronized oscillatory activity correlates with motor impairment in patients with Parkinson's disease. Mov Disord. (2016) 31:1748–51. doi: 10.1002/mds.26759
68. Beudel M, Oswal A, Jha A, Foltynie T, Zrinzo L, Hariz M, et al. Oscillatory beta power correlates with akinesia-rigidity in the parkinsonian subthalamic nucleus. Mov Disord. (2017) 32:174–5. doi: 10.1002/mds.26860
69. Kühn AA, Kupsch A, Schneider GH, Brown P. Reduction in subthalamic 8-35 Hz oscillatory activity correlates with clinical improvement in Parkinson's disease. Eur J Neurosci. (2006) 23:1956–60. doi: 10.1111/j.1460-9568.2006.04717.x
70. Kühn AA, Tsui A, Aziz T, Ray N, Brücke C, Kupsch A, et al. Pathological synchronisation in the subthalamic nucleus of patients with Parkinson's disease relates to both bradykinesia and rigidity. Exp Neurol. (2009) 215:380–7. doi: 10.1016/j.expneurol.2008.11.008
71. Kühn AA, Kempf F, Brücke C, Gaynor Doyle L, Martinez-Torres I, Pogosyan A, et al. High-frequency stimulation of the subthalamic nucleus suppresses oscillatory beta activity in patients with Parkinson's disease in parallel with improvement in motor performance. J Neurosci. (2008) 28:6165–73. doi: 10.1523/JNEUROSCI.0282-08.2008
72. Jenkinson N, Brown P. New insights into the relationship between dopamine, beta oscillations and motor function. Trends Neurosci. (2011) 34:611–8. doi: 10.1016/j.tins.2011.09.003
73. Quinn EJ, Blumenfeld Z, Velisar A, Koop MM, Shreve LA, Trager MH, et al. Beta oscillations in freely moving Parkinson's subjects are attenuated during deep brain stimulation. Mov Disord. (2015) 30:1750–8. doi: 10.1002/mds.26376
74. Neumann WJ, Staub F, Horn A, Schanda J, Mueller J, Schneider GH, et al. Deep brain recordings using an implanted pulse generator in Parkinson's disease. Neuromodulation. (2016) 19:20–3. doi: 10.1111/ner.12348
75. Tinkhauser G, Pogosyan A, Tan H, Herz DM, Kühn AA, Brown P. Beta burst dynamics in Parkinson's disease OFF and ON dopaminergic medication. Brain. (2017) 140:2968–81. doi: 10.1093/brain/awx252
76. Eusebio A, Thevathasan W, Doyle Gaynor L, Pogosyan A, Bye E, Foltynie T, et al. Deep brain stimulation can suppress pathological synchronisation in parkinsonian patients. J Neurol Neurosurg Psychiatr. (2011) 82:569–73. doi: 10.1136/jnnp.2010.217489
77. Alonso-Frech F, Zamarbide I, Alegre M, Rodríguez-Oroz MC, Guridi J, Manrique M, et al. Slow oscillatory activity and levodopa-induced dyskinesias in Parkinson's disease. Brain. (2006) 129:1748–57. doi: 10.1093/brain/awl103
78. Alegre M, López-Azcárate J, Alonso-Frech F, Rodríguez-Oroz MC, Valencia M, Guridi J, et al. Subthalamic activity during diphasic dyskinesias in Parkinson's disease. Mov Disord. (2012) 27:1178–81. doi: 10.1002/mds.25090
79. Kühn AA, Williams D, Kupsch A, Limousin P, Hariz M, Schneider GH, et al. Event-related beta desynchronization in human subthalamic nucleus correlates with motor performance. Brain. (2004) 127:735–46. doi: 10.1093/brain/awh106
80. Brücke C, Kempf F, Kupsch A, Schneider GH, Krauss JK, Aziz T, et al. Movement-related synchronization of gamma activity is lateralized in patients with dystonia. Eur. J. Neurosci. (2008) 27:2322–9. doi: 10.1111/j.1460-9568.2008.06203.x
81. Brücke C, Huebl J, Schonecker T, Neumann WJ, Yarrow K, Kupsch A, et al. Scaling of movement is related to pallidal oscillations in patients with dystonia. J Neurosci. (2012) 32:1008–19. doi: 10.1523/JNEUROSCI.3860-11.2012
82. Lofredi R, Neumann WJ, Bock A, Horn A, Huebl J, Siegert S, et al. Dopamine-dependent scaling of subthalamic gamma bursts with movement velocity in patients with Parkinson's disease. Elife. (2018) 7:e31895. doi: 10.7554/eLife.31895.
83. Özkurt TE, Schnitzler A. A critical note on the definition of phase-amplitude cross-frequency coupling. J Neurosci Methods. (2011) 201:438–43. doi: 10.1016/j.jneumeth.2011.08.014
84. Özkurt TE, Butz M, Homburger M, Elben S, Vesper J, Wojtecki L, et al. High frequency oscillations in the subthalamic nucleus: a neurophysiological marker of the motor state in Parkinson's disease. Exp Neurol. (2011) 229:324–31. doi: 10.1016/j.expneurol.2011.02.015
85. van Wijk BCM, Beudel M, Jha A, Oswal A, Foltynie T, Hariz MI, et al. Subthalamic nucleus phase–amplitude coupling correlates with motor impairment in Parkinson's disease. Clin Neurophysiol. (2016) 127:2010–9. doi: 10.1016/j.clinph.2016.01.015
86. Wang J, Hirschmann J, Elben S, Hartmann CJ, Vesper J, Wojtecki L, et al. High-frequency oscillations in Parkinson's disease: spatial distribution and clinical relevance. Mov Disord. (2014) 29:1265–72. doi: 10.1002/mds.25962
87. Little S, Pogosyan A, Neal S, Zavala B, Zrinzo L, Hariz M, et al. Adaptive deep brain stimulation in advanced Parkinson disease. Ann Neurol. (2013) 74:449–57. doi: 10.1002/ana.23951
88. Meidahl AC, Tinkhauser G, Herz DM, Cagnan H, Debarros J, Brown P. Adaptive deep brain stimulation for movement disorders: the long road to clinical therapy. Mov Disord. (2017) 32:810–9. doi: 10.1002/mds.27022
89. Tinkhauser G, Pogosyan A, Little S, Beudel M, Herz DM, Tan H, et al. The modulatory effect of adaptive deep brain stimulation on beta bursts in Parkinson's disease. Brain. (2017) 140:1053–67. doi: 10.1093/brain/awx010
90. de Hemptinne C, Swann NC, Ostrem JL, Ryapolova-Webb ES, San Luciano M, Galifianakis NB, et al. Therapeutic deep brain stimulation reduces cortical phase-amplitude coupling in Parkinson's disease. Nat Neurosci. (2015) 18:779–86. doi: 10.1038/nn.3997
91. Cole SR, Peterson EJ, van der Meij R, de Hemptinne C, Starr PA, Voytek B. Nonsinusoidal oscillations underlie pathological phase-amplitude coupling in the motor cortex in Parkinson's disease. bioRxiv. 049304. doi: 10.1101/049304
92. Little S, Brown P. What brain signals are suitable for feedback controzl of deep brain stimulation in Parkinson's disease? Ann N Y Acad Sci. (2012) 1265:9–24. doi: 10.1111/j.1749-6632.2012.06650.x
93. Lenka A, Hegde S, Jhunjhunwala KR, Pal PK. Interactions of visual hallucinations, rapid eye movement sleep behavior disorder and cognitive impairment in Parkinson's disease: a review. Parkinsonism Relat Disord. (2016) 22:1–8. doi: 10.1016/j.parkreldis.2015.11.018
94. Cagnan H, Brittain J-S, Little S, Foltynie T, Limousin P, Zrinzo L, et al. Phase dependent modulation of tremor amplitude in essential tremor through thalamic stimulation. Brain. (2013) 136:3062–75. doi: 10.1093/brain/awt239
95. Foffani G, Bianchi AM, Baselli G, Priori A. Movement-related frequency modulation of beta oscillatory activity in the human subthalamic nucleus. J Physiol. (2005) 568(Pt 2):699–711. doi: 10.1113/jphysiol.2005.089722
96. Herz DM, Zavala BA, Bogacz R, Brown P. Neural correlates of decision thresholds in the human subthalamic nucleus. Curr Biol. (2016) 26:916–20. doi: 10.1016/j.cub.2016.01.051
97. Williams D, Kühn A, Kupsch A, Tijssen M, Van Bruggen G, Speelman H, et al. The relationship between oscillatory activity and motor reaction time in the parkinsonian subthalamic nucleus. Eur J Neurosci. (2005) 21:249–58. doi: 10.1111/j.1460-9568.2004.03817.x
98. Staub F, Neumann WJ, Horn A, Schanda J, Schneider GH, Brown P, et al. EP 4. Long term recordings of deep brain activity from the subthalamic nucleus in PD patients using PC+S. Clin Neurophysiol. (2016) 127:e176. doi: 10.1016/j.clinph.2016.05.199
99. Steiner LA, Neumann WJ, Staub-Bartelt F, Herz DM, Tan H, Pogosyan A, et al. Subthalamic beta dynamics mirror Parkinsonian bradykinesia months after neurostimulator implantation. Mov Disord. (2017) 32:1183–90. doi: 10.1002/mds.27068
100. Hosain MK, Kouzani A, Tye S. Closed loop deep brain stimulation: an evolving technology. Australas Phys Eng Sci Med. (2014) 37:619–34. doi: 10.1007/s13246-014-0297-2
101. Swann NC, De Hemptinne C, Thompson MC, Miocinovic S, Miller AM, Gilron R, et al. Adaptive deep brain stimulation for Parkinson's disease using motor cortex sensing. J Neural Eng. (2018) 15:046006. doi: 10.1088/1741-2552/aabc9b
102. Neumann W-J, Turner RS, Blankertz B, Mitchell T, Kühn AA, Richardson RM. Toward electrophysiology-based intelligent adaptive deep brain stimulation for movement disorders. Neurotherapeutics. (2019) 16:105–18. doi: 10.1007/s13311-018-00705-0
103. Van Gompel JJ, Chang S-Y, Goerss SJ, Kim IY, Kimble C, Bennet KE, et al. Development of intraoperative electrochemical detection: wireless instantaneous neurochemical concentration sensor for deep brain stimulation feedback. Neurosurg Focus. (2010) 29:E6. doi: 10.3171/2010.5.FOCUS10110
104. Grahn PJ, Mallory GW, Khurram OU, Berry BM, Hachmann JT, Bieber AJ, et al. A neurochemical closed-loop controller for deep brain stimulation: toward individualized smart neuromodulation therapies. Front Neurosci. (2014) 8:169. doi: 10.3389/fnins.2014.00169
105. Lee KH, Lujan JL, Trevathan JK, Ross EK, Bartoletta JJ, Park HO, et al. WINCS Harmoni: closed-loop dynamic neurochemical control of therapeutic interventions. Sci Rep. (2017) 7:46675. doi: 10.1038/srep46675
106. Bötzel K, Marti FM, Rodríguez MÁC, Plate A, Vicente AO. Gait recording with inertial sensors – how to determine initial and terminal contact. J Biomech. (2016) 49:332–7. doi: 10.1016/j.jbiomech.2015.12.035
107. Heldman DA, Pulliam CL, Urrea Mendoza E, Gartner M, Giuffrida JP, Montgomery Jr EB, et al. Computer-guided deep brain stimulation programming for Parkinson's disease. Neuromodulation. (2016) 19:127–32. doi: 10.1111/ner.12372
108. Niazmand K, Tonn K, Kalaras A, Fietzek UM, Mehrkens JH, Lueth TC. Quantitative evaluation of Parkinson's disease using sensor based smart glove. In: Proceedings–IEEE Symposium on Computer-Based Medical Systems. Bristol, UK (2011).
109. Pulliam CL, Heldman DA, Orcutt TH, Mera TO, Giuffrida JP, Vitek JL. Motion sensor strategies for automated optimization of deep brain stimulation in Parkinson's disease. Parkinsonism Relat Disord. (2015) 21:378–82. doi: 10.1016/j.parkreldis.2015.01.018
110. Hell F, Plate A, Mehrkens JH, Bötzel K. Subthalamic oscillatory activity and connectivity during gait in Parkinson's disease. NeuroImage Clin. (2018) 19:396–405. doi: 10.1016/j.nicl.2018.05.001
111. Brittain JS, Probert-Smith P, Aziz TZ, Brown P. Tremor suppression by rhythmic transcranial current stimulation. Curr Biol. (2013) 23:436–40. doi: 10.1016/j.cub.2013.01.068
112. Herron JA, Thompson MC, Brown T, Chizeck HJ, Ojemann JG, Ko AL. Cortical brain-computer interface for closed-loop deep brain stimulation. IEEE Trans Neural Syst Rehabil Eng. (2017) 25:2180–7. doi: 10.1109/TNSRE.2017.2705661
113. Zeitler M, Tass PA. Augmented brain function by coordinated reset stimulation with slowly varying sequences. Front Syst Neurosci. (2015) 9:49. doi: 10.3389/fnsys.2015.00049
114. Ebert M, Hauptmann C, Tass PA. Coordinated reset stimulation in a large-scale model of the STN-GPe circuit. Front Comput Neurosci. (2014) 8:154. doi: 10.3389/fncom.2014.00154
115. Adamchic I, Hauptmann C, Barnikol UB, Pawelczyk N, Popovych O, Barnikol TT, et al. Coordinated reset neuromodulation for Parkinson's disease: proof-of-concept study. Mov Disord. (2014) 29:1679–84. doi: 10.1002/mds.25923
116. Grossman N, Bono D, Dedic N, Kodandaramaiah SB, Rudenko A, Suk HJ, et al. Noninvasive deep brain stimulation via temporally interfering electric fields. Cell. (2017) 169:1029–41.e16. doi: 10.1016/j.cell.2017.05.024
117. Lozano AM. Waving hello to noninvasive deep-brain stimulation. N Engl J Med. (2017) 377:1096–98. doi: 10.1056/NEJMcibr1707165
118. Gradinaru V, Mogri M, Thompson KR, Henderson JM, Deisseroth K. Optical deconstruction of parkinsonian neural circuitry. Science. (2009) 324:354–9. doi: 10.1126/science.1167093
119. Bernstein JG, Garrity PA, Boyden ES. Optogenetics and thermogenetics: technologies for controlling the activity of targeted cells within intact neural circuits. Curr Opin Neurobiol. (2012) 22:61–71. doi: 10.1016/j.conb.2011.10.023
120. Schirrmeister RT, Springenberg JT, Fiederer LDJ, Glasstetter M, Eggensperger K, Tangermann M, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp. (2017) 38:5391–420. doi: 10.1002/hbm.23730
121. Hell F, Mehrkens JH, Bötzel K. Feedback signals for adaptive DBS. Current perspectives and future directions | Auf dem Weg zur adaptiven Hirnstimulation. Neurophysiologie Labor. (2018) 40:179–88. doi: 10.1016/j.neulab.2018.02.001
122. Hell F, Köglsperger T, Boetzel K. Improving the standard for deep brain stimulation therapy: target structures and feedback signals for adaptive stimulation. Current Perspectives and future directions. Cureus. (2018) 10:e2468. doi: 10.7759/cureus.2468
123. Sutton RS, Barto AG, Williams RJ. Reinforcement learning is direct adaptive optimal control. IEEE Control Syst. (1992) 9:1054.
124. Sutton RS, Barto AG. Reinforcement learning: an introduction. IEEE Trans Neural Netw. (1998) 9:1054. doi: 10.1109/TNN.1998.712192
125. Huys QJM, Cruickshank A, Seriès P. Reward-based learning, model-based and model-free. In: Jaeger D, Jung R, editors. Encyclopedia of Computational Neuroscience. New York, NY: Springer New York (2014). p. 1–10. doi: 10.1007/978-1-4614-7320-6_674-1
126. Schulman J, Levine S, Moritz P, Jordan MI, Abbeel P. Trust region policy optimization. CoRR abs/1502.0. (2015) Available online at: http://arxiv.org/abs/1502.05477 (accessed March 21, 2019).
127. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. CoRR abs/1707.0. (2017) Available online at: http://arxiv.org/abs/1707.06347
128. Dayan P, Niv Y. Reinforcement learning: the good, the bad and the ugly. Curr Opin Neurobiol. (2008) 18:185–96. doi: 10.1016/j.conb.2008.08.003
129. Temel Y. Deep brain stimulation in animal models. Handb Clin Neurol. (2013) 116:19–25. doi: 10.1016/B978-0-444-53497-2.00002-4
130. Smith SM, Miller KL, Salimi-Khorshidi G, Webster M, Beckmann CF, Nichols TE, et al. Network modelling methods for FMRI. NeuroImage. (2011) 54:875–91. doi: 10.1016/j.neuroimage.2010.08.063
131. Grosse-Wentrup M, Janzing D, Siegel M, Schölkopf B. Identification of causal relations in neuroimaging data with latent confounders: an instrumental variable approach. NeuroImage. (2016) 125:825–33. doi: 10.1016/j.neuroimage.2015.10.062
132. Granger CWJ. Investigating causal relations by econometric models and cross-spectral methods. Econometrica. (1969) 37:424. doi: 10.2307/1912791
133. Gregoriou GG, Gotts SJ, Zhou H, Desimone R. High-frequency, long-range coupling between prefrontal and visual cortex during attention. Science. (2009) 324:1207–10. doi: 10.1126/science.1171402
135. Daunizeau J, David O, Stephan KE. Dynamic causal modelling: a critical review of the biophysical and statistical foundations. NeuroImage. (2011) 58:312–22. doi: 10.1016/j.neuroimage.2009.11.062
136. McLntosh AR, Gonzalez-Lima F. Structural equation modeling and its application to network analysis in functional brain imaging. Hum Brain Mapp. (1994) 2:2–22. doi: 10.1002/hbm.460020104
137. Atlas LY, Bolger N, Lindquist MA, Wager TD. Brain mediators of predictive cue effects on perceived pain. J Neurosci. (2010) 30:12964–77. doi: 10.1523/JNEUROSCI.0057-10.2010
138. Grosse-Wentrup M, Schölkopf B, Hill J. Causal influence of gamma oscillations on the sensorimotor rhythm. NeuroImage. (2011) 56:837–42. doi: 10.1016/j.neuroimage.2010.04.265
139. Weichwald S, Meyer T, Özdenizci O, Schölkopf B, Ball T, Grosse-Wentrup M. Causal interpretation rules for encoding and decoding models in neuroimaging. NeuroImage. (2015) 110:48–59. doi: 10.1016/j.neuroimage.2015.01.036
140. van Wijk BCM, Cagnan H, Litvak V, Kühn AA, Friston KJ. Generic dynamic causal modelling: an illustrative application to Parkinson's disease. NeuroImage. (2018) 181:818–30. doi: 10.1016/j.neuroimage.2018.08.039
141. Weerasinghe G, Duchet B, Cagnan H, Brown P, Bick C, Bogacz R. Predicting the effects of deep brain stimulation using a reduced coupled oscillator model. bioRxiv. [preprint]. (2018). doi: 10.1101/448290
142. Pearl J. Causality: Models, Reasoning, and Inference. 2nd ed. Boston, MA: Cambridge University Press (2011).
Keywords: deep brain stimulation, machine learning, adaptive, feedback, DBS target, reinforcement learning
Citation: Hell F, Palleis C, Mehrkens JH, Koeglsperger T and Bötzel K (2019) Deep Brain Stimulation Programming 2.0: Future Perspectives for Target Identification and Adaptive Closed Loop Stimulation. Front. Neurol. 10:314. doi: 10.3389/fneur.2019.00314
Received: 13 December 2018; Accepted: 12 March 2019;
Published: 03 April 2019.
Edited by:
Aristide Merola, University of Cincinnati, United StatesReviewed by:
Andrew P. Duker, University of Cincinnati, United StatesMariachiara Sensi, Azienda Ospedaliero Universitaria di Ferrara, Italy
Copyright © 2019 Hell, Palleis, Mehrkens, Koeglsperger and Bötzel. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Franz Hell, franz.hell@med.uni-muenchen.de
Kai Bötzel, kai.boetzel@med.uni-muenchen.de
†These authors have contributed equally to this work