Advancing brain-machine interfaces: moving beyond linear state space models
Adam G. Rouse1,2,3 
Marc H. Schieber1,2,3*- 1Department of Neurology, University of Rochester, Rochester, NY, USA
- 2Department of Neurobiology and Anatomy, University of Rochester, Rochester, NY, USA
- 3Department of Biomedical Engineering, University of Rochester, Rochester, NY, USA
Advances in recent years have dramatically improved output control by Brain-Machine Interfaces (BMIs). Such devices nevertheless remain robotic and limited in their movements compared to normal human motor performance. Most current BMIs rely on transforming recorded neural activity to a linear state space composed of a set number of fixed degrees of freedom. Here we consider a variety of ways in which BMI design might be advanced further by applying non-linear dynamics observed in normal motor behavior. We consider (i) the dynamic range and precision of natural movements, (ii) differences between cortical activity and actual body movement, (iii) kinematic and muscular synergies, and (iv) the implications of large neuronal populations. We advance the hypothesis that a given population of recorded neurons may transmit more useful information than can be captured by a single, linear model across all movement phases and contexts. We argue that incorporating these various non-linear characteristics will be an important next step in advancing BMIs to more closely match natural motor performance.
Introduction
Research and development of brain-machine interfaces (BMIs) to restore lost motor function has expanded dramatically in recent years. Whereas, not long ago the state of the art in both non-human primates (Serruya et al., 2002; Taylor et al., 2002) and humans (Hochberg et al., 2006) consisted of controlling a cursor on a computer screen, recent advances in restoring upper limb function have incorporated robotic arms with the ability to grip and manipulate objects using either virtual (Carmena et al., 2003; Chadwick et al., 2011) or robotic (Velliste et al., 2008; Hochberg et al., 2012; Collinger et al., 2013; Wodlinger et al., 2015) hands and digits. In addition to restoring upper limb function, innovative exoskeletons are being used to restore trunk and leg function (Fitzsimmons et al., 2009; Lebedev and Nicolelis, 2011). Functional electrical stimulation also has made possible restoration of movement of the subject's own limbs (Moritz et al., 2008; Pancrazio and Peckham, 2009; Ethier et al., 2012). Many of these advances have transitioned from the research laboratory with non-human primates to the clinical world with human subjects.
Different but inter-related improvements have contributed to these advances. Better recording systems that allow chronically implanted electrodes to record an increasing number of channels of neural activity simultaneously have had a major impact (Stieglitz et al., 2009; Homer et al., 2013). Better understanding of how to extract various neural signals, paradoxically including less focus on precise spike sorting and more use of simple detection methods like threshold crossings (Fraser et al., 2009; Chestek et al., 2011; Hochberg et al., 2012; Wodlinger et al., 2015), the application of statistical models to better estimate firing rates (Cunningham et al., 2010), and the implementation of Kalman filtering (Wu et al., 2003; Li et al., 2009), all have improved BMI performance as well. Better appreciation of the neural adaptation that occurs under BMI conditions has led to dynamic updating of decoding algorithms that enable BMI learning to be both faster and more robust to external changes (Gilja et al., 2012; Orsborn et al., 2012; Zhang and Chase, 2013; Shanechi et al., 2014). A focus on overcoming the technical challenges of chronically monitoring larger and larger amounts of neural activity and on controlling increasingly complex devices has advanced the field substantially and quickly.
Over the same period during which these advances have been made, relatively little has changed in how our understanding of natural motor physiology is applied to BMI control. Current BMI designs almost always assume that neural encoding is a linear, time-invariant system with independent degrees of freedom (DOFs), and therefore implement control algorithms that map neural inputs to a constant set of output variables with a fixed gain. Yet neural control of mammalian motor systems and the behaviors they produce cannot be explained fully with such an idealized model.
Here we examine selected aspects of motor behavior and physiology to explore ways in which current scientific understanding might be exploited to advance the design of BMIs toward achieving performance closer to that of natural human movement. We discuss four different topics. First, we consider differences in motor performance for very fast or very slow movements and examine how BMI decoding might better emulate similar principles. Second, we address evidence that cortical activity—even in areas with significant spinal projections—differs considerably from a veridical representation of actual movement of the body and we consider the implications of these differences for continuous BMI operation. Third, we explore the evidence that natural movement is not statistically independent across different joints or muscles but rather is coordinated and we ask how this might be incorporated more extensively in BMI design. And fourth, we explore the implications of having a large neuronal population to generate movement, examining whether such a system can be modeled effectively with a single, linear state space, and asking whether adhering to such a model has become a limiting oversimplification in current BMI designs. Although the four topics we explore may appear diverse and unrelated, they share the common theme that incorporating a deeper understanding of the non-linear dynamics of normal motor behavior and physiology—during different phases of movement and in different contexts—can advance BMI design. Whereas, scientists generally seek to identify the simplest explanation for the largest set of observations and engineers seek to provide the simplest design to achieve a specific function (Müller et al., 2003), here we argue that the next wave of advances in BMI technology will require incorporating additional levels of complexity.
Dealing with a Wide Dynamic Range of Movement
Humans are capable of performing skilled movements on a wide range of spatial and temporal scales, from the athletic prowess of throwing or kicking a ball at speeds approaching 100 miles per hour (44.7 m/s) to the fine motor skills required for watchmaking and surgery. But throwing a ball is not necessarily controlled in the same fashion as knotting a suture. We therefore consider how control might vary depending on the extent to which a subject intends to make a gross movement quickly vs. a fine movement accurately.
Dynamic Range of Movements
In natural movements, Fitts's law (Fitts, 1954) describes a tradeoff between speed and accuracy: The faster the movement, the less accurate it will be; greater accuracy is achieved with slower movement. This principle has been documented in numerous natural movement tasks and under many different conditions (Card et al., 1978; Jagacinski and Monk, 1985; Epps, 1986; MacKenzie, 1992), in neural correlates (Ifft et al., 2011), and in BMI tasks with direct neural control (Felton et al., 2009).
Current BMIs, however, do little to emulate the robust range of behavior observed to follow Fitts's law for either very fast or very accurate movements, relying instead on the designer to pick a fixed gain between input neural signals and output magnitude. This practice effectively limits good BMI performance to a narrow range of the speed/accuracy trade-off. Enabling the user's neural activity to select the movement speed and the associated accuracy dynamically depending on the current phase or context of the task could improve BMI control, providing a range of performance closer to that of natural movements.
Improving the Precision of BMIs
Fine, skilled movements presumably require more precise neural control signals than gross movements, whether the control signal is encoding muscle activation, force, joint position, or velocity. A study by Slifkin and Newell (1999) found, for example, that whereas the average maximum voluntary contraction (MVC) of the index finger against a load cell was 31.07 N with a standard deviation of ~2.1 N, when producing forces of 5% MVC (~1.55 N) the standard deviation was only ~0.09 N. Variability in normal human force production thus scales with the magnitude of force being produced. Such signal-dependent noise appears in many natural motor behaviors (Harris and Wolpert, 1998). Smaller movements are made with smaller errors.
A system that linearly transforms an input to output with constant noise cannot be optimized for fine, accurate movement and for gross, fast movement simultaneously. Indeed, cortical microcircuits in the motor system recently have been proposed to adjust neuron tuning according to the level of precision required. By functionally varying the overall strength of excitatory/inhibitory drive and changing the tuning widths of individual M1 neurons, the accuracy and precision of movement encoding by a population of neurons might be adjusted dynamically (Mahan and Georgopoulos, 2013; Georgopoulos, 2014). Although natural adjustment of neuronal tuning may be present in the input neurons, current BMIs that rely on the linear sum of neural activity nevertheless show poor adjustment of precision compared to that found in natural movements. Such suboptimal adjustment of precision might result from the small number of input neurons compared to the natural motor system and/or the lack of adjustment for precision in the tuning and decoding models used currently.
BMIs therefore might be improved by creating decoding schemes that more explicitly allow for encoding of the speed-accuracy tradeoff from a recorded subpopulation. If we construct a BMI model that uses instantaneous neuronal firing rates to encode velocity in a single dimension with a simple linear encoder, the magnitude of the error is uniform for all encoded velocities (Figure 1A). If instead the neurons in the model are assigned to encode a rescaling of velocity, the square root of velocity for example, the errors near zero velocity are smaller than those for high velocity (Figure 1B), more closely emulating the signal-dependent noise of natural behavior (Figure 1C). Applying such approaches to create noise profiles that more closely match natural behavior is likely to create BMI controlled movements that appear more natural by allowing greater precision for fine movements.
Figure 1. Error in simulated velocity decoding. A simulated population of neurons was scaled to encode velocities (with a maximum system velocity equal to 1) using two different methods. (A) The simulated population encodes linear velocity, producing uniform error in the decoded velocity that is independent of the desired velocity. (B) The simulated population (with the same dynamic range and noise properties of the neurons) is now scaled to encode the square root of velocity, producing error in the decoded velocity that becomes smaller for velocities closer to zero. (C) The standard deviation of decoded velocity is plotted as a function of desired velocity for the linear model (A, dotted) and for the square root model (B, dashed), emphasizing that when decoding linear velocity the error is constant across desired velocities, whereas for the square root model the error is smaller close to zero velocity and larger for higher velocities.
The Relationship of Cortical Activity to Physical Movement
The majority of current BMIs that control motor output use neural activity recorded from motor and premotor areas of the cerebral cortex as input for two reasons. First, many aspects of natural movements are represented in the activity of neurons in these cortical areas. Second, the cerebral cortex can be accessed for neural recording relatively easily compared to deeper parts of the brain and spinal cord. But using only cortical activity has other advantages and disadvantages for BMI control: (i) certain features of cortical activity are not output to physical movement of the body, (ii) the tuning of cortical neurons changes when controlling a BMI, and (iii) certain aspects of physical movement may not be controlled directly from the cortex.
Motor Imagery, Mirror Neurons, and BMI Control
During motor imagery, when humans imagine themselves performing movements without actually making any movement, activation appears in many of the same cortical motor areas that are activated during physical movement performance, including the primary motor cortex (Ersland et al., 1996; Grafton et al., 1996; Porro et al., 1996; Pfurtscheller and Neuper, 1997; Anderson et al., 2011; Ajiboye et al., 2012). Likewise during action observation, when a monkey observes another individual performing a particular movement, a large subpopulation of neurons discharges in a fashion similar to their discharge when the monkey executes the same action itself. These “mirror neurons,” as well as other neurons that are activated when the monkey observes a cursor moved by the investigator, have been observed not only in premotor cortex but in primary motor cortex as well (Cisek and Kalaska, 2004; Tkach et al., 2007; Dushanova and Donoghue, 2010; Casile, 2013; Vigneswaran et al., 2013). Many neurons in motor cortex thus discharge during motor imagery and/or action observation, when subjects are not making any physical movement.
BMI experiments with normal subjects are perhaps the most clear-cut demonstration that neurons active during natural movements can also be activated voluntarily in the absence of physical movement. In such BMI experiments, normal monkeys typically first perform a task involving physical movement of their native limb while neural activity is recorded from the cortex. This activity then is used to calibrate a decoding algorithm that relates the recorded neuronal activity to the observed limb movements. But after switching to “brain-control”—now controlling a cursor or prosthesis directly through the decoding algorithm—the monkey often stops making any physical movement or producing any EMG activity, even though allowed to move freely (Taylor et al., 2002; Carmena et al., 2003). Meanwhile, the recorded neurons continue to modulate, voluntarily controlling the external BMI device. Like mirror neurons, these neurons are activated during physical movement of the native limb, but are not obligatorily coupled to physical movement. Though in most BMI experiments neurons have not been tested for such, many of them might have activity during action observation.
Indeed, we know that neurons activated during action observation can be used for voluntary BMI control, because in both normal monkeys and paralyzed humans neuronal activity recorded during action observation can be used effectively to develop the initial decoding algorithm (Velliste et al., 2008; Chadwick et al., 2011; Collinger et al., 2013). Although using neurons activated by action observation currently provides an advantage for calibrating BMIs in paralyzed or amputated subjects, using mirror neurons might have disadvantages as well. In one recent study, for example, having a particular object present during the initial calibration was needed to enable the subject subsequently to close the prosthetic hand optimally around that object (Wodlinger et al., 2015), possibly indicating that the recorded population included a substantial number of mirror neurons encoding that particular grasp shape. More detailed scientific understanding of the differences in neuronal activity during motor imagery vs. action observation vs. action execution may enable BMI decoding based on these differences to improve performance.
Changes in Cortical Activity during BMI Control
The change from “hand-control” to “brain-control” changes cortical activity. Many neurons, for example, show changes in their directional tuning (Taylor et al., 2002; Ganguly and Carmena, 2009). Such changes may result from alterations in proprioceptive feedback because a normal subject's native limb that moved during hand-control either moves differently or does not move at all during brain-control. Differences may exist as well in the descending regulation of sensory input to the spinal cord, which changes during voluntary movement as compared to rest (Seki et al., 2003). Visual inputs change too. If the subject is controlling a prosthetic arm, its visual representation obviously will differ from that of a native arm, as will incorporation of the prosthesis into the internal body schema (i.e., embodiment). Even if the subject is controlling a cursor on a computer screen, the visual motion of the cursor viewed by the subject will differ, being smoother and more accurate during hand-control but showing more jitter during brain-control. In addition, many changes in cortical activity may represent adaptation on the part of the subject to fit the linear model of the BMI decoder (Wolpaw, 2010; Chase and Schwartz, 2011). More detailed scientific understanding of such changes in cortical activity that occur upon switching to brain-control may improve BMI performance.
Transformations in the Spinal Cord and Phases of Motor Control
Though classically viewed as a simple communication channel between the brain and the motor periphery, the spinal cord now is known to contain complex circuits that make important contributions to natural movements. Beyond the basic reflex pathways that can elicit movement from sensory input through only one or two synapses without cortical interaction, central pattern generators in the spinal cord can produce complex rhythmic behaviors without patterned input from the brain or feedback from the periphery (Shik and Orlovsky, 1976; Stein, 1978; Grillner, 1985). Descending signals from the brain are likely to engage parts of this spinal circuitry for production of other, non-rhythmic movements as well (Georgopoulos and Grillner, 1989). Indeed, in most mammalian species, the descending fibers of the corticospinal tract end on spinal interneurons, not motoneurons. And in macaques, which do have direct cortico-motoneuronal (CM) projections, spinal interneurons output different information (Maier et al., 1998; Fetz et al., 2002). For example, whereas most CM-cells were active selectively when subjects exerted either flexion or extension wrist forces, a high percentage of spinal interneurons were active for both flexion and extension force production, as well as at rest. With such complexity in the spinal cord, it becomes apparent that spinal cord circuitry may be doing much more than simply relaying the current input of descending cortical signals to generate muscle activity.
An emerging distinction between motor signals in the cortex vs. those in the spinal cord has to do with relative degrees of dynamic vs. static function in controlling non-cyclical limb movements. Neurons in the motor cortex typically are relatively quiescent during maintenance of a steady posture, become intensely active leading limb movement, and then show declining activity as a new steady posture is established. In large part such observations are attributable to a stronger relationship to movement velocity than to position (Moran and Schwartz, 1999). Moreover, strong rotational dynamics of joint M1 neuron firing rate trajectories may reflect a complex, dynamical system responsible for the encoding of movement (Churchland et al., 2012; Hall et al., 2014). In comparison to cortical neurons, spinal interneurons show more static activity. In monkeys generating wrist forces isometrically and auxotonically, for example, cortical neurons produced relatively transient signals for ensuing motor actions while spinal neurons generated more sustained activity, suggesting that to some degree cortical signals to change state are integrated by spinal circuitry (Shalit et al., 2012). Indeed, recent models of spinal-like regulators have demonstrated that oversimplified step inputs from the brain could be transformed by spinal circuitry to replicate much of observed center-out reaching behavior (Tsianos et al., 2014). Implementing such spinal-like circuitry in BMIs may substantially improve the quality of the transitions between movement and posture.
The transition from movement to posture is but one example of what more generally might be considered different sequential phases of motor control. A single decoder cannot be expected to deal with all phases efficiently. Cortical neurons can be identified, for example, that are active specifically in relation to rest/posture in contrast to movement (Humphrey and Reed, 1983; Williams et al., 2013; Velliste et al., 2014). Including such neural activity in the same linear decoding algorithm that drives movement velocity would be counterproductive. But if one decoder used movement-related activity to drive motion during movement phases, and another decoder used posture-related activity to maintain position during postural phases, smoother and more efficient performance might be obtained. Such control would require yet another decoding algorithm to identify transitions between phases, sometimes referred to as changes of state (Kemere et al., 2008; Aggarwal et al., 2013; Kang et al., 2015). Sequential phases or states may include not just alternation between movement and posture, but also inattention, watchful waiting for an instructional cue, preparation of a specific motor plan, reaction time following a go cue, then reaching, grasping, manipulation, and others as well. Detecting these various phases and decoding them differently may be an important step toward achieving natural human performance with BMIs (see also section More Neurons than Controlled DOFs creates a Null Space, below).
Controlling Multiple Degrees of Freedom
Independent Degrees of Freedom
Analysis of voluntary movement consistently has shown that natural movement almost never occurs in isolation at a single joint or by activation of a single muscle, even the movement of a single finger (Hager-Ross and Schieber, 2000). For example, typists and pianists produce simultaneous movements of multiple digits even when striking a single key (Flanders and Soechting, 1992; Engel et al., 1997). Nevertheless, most of the variance of complex multi-joint movements can be reduced mathematically to a relatively small number of principal components, each of which captures a pattern of simultaneous motion at multiple joints (Santello et al., 1998; Mason et al., 2004). Such findings suggest that the number of DOFs being controlled actively during many natural movements might be less than the number of DOFs actually moving.
Bernstein (1967) first defined this problem of redundant DOFs in the musculoskeletal system: Many movements made in three-dimensional space engage more than three joint angles and more than three muscles. Mathematically, therefore, a given movement can be made in many different ways, i.e., infinite possible solutions can successfully accomplish a given movement task. Observing blacksmiths, for example, Bernstein noticed that while the joints of the arm might take quite different trajectories during a series of hammer strikes, the endpoint of contact was very consistent. This and other observations of considerable variation in certain DOFs while others are controlled precisely was formulated subsequently by Scholz and Schöner (1999) as the uncontrolled manifold hypothesis: Within the high dimensional configuration space that completely defines movements for a given task, there exists two orthogonal subspaces. Motion in a controlled subspace contains a set of actively controlled variables that are being monitored and controlled by the subject and are most important to completing the task. The other, uncontrolled subspace contains all motion orthogonal to the controlled variables and thus has no effect on successful task completion. Increased variability has been observed in the uncontrolled subspace compared to the controlled subspace in a wide variety of natural movement tasks (Scholz et al., 2001; Latash et al., 2002; Tseng et al., 2002; Kang et al., 2004), and recently in neuronal activity as well (Kaufman et al., 2014; Law et al., 2014).
Current BMI design, however, remains limited in strategies that take into account the relative importance of the various DOFs in different tasks, instead controlling the same fixed set of DOFs of the prosthetic device independently at all points in time. Likewise, regression and updating algorithms assume each DOF is encoded equally at all times. As the number of DOFs increase, BMI control becomes more difficult because more DOFs must be monitored and controlled by the subject. Models that more closely align BMI control at a given time with a subset of DOFs selected judiciously for the current task may enable more intuitive and precise control.
Kinematic and Muscular Synergies
One means of selecting subsets of multiple DOFs for BMI control is to look for naturally occurring patterns of simultaneous motion at multiple joints or patterns of simultaneous activation of multiple muscles. A small number of fixed patterns of multi-joint motion (Santello et al., 1998; Mason et al., 2004) or multi-muscle activity (d'Avella and Bizzi, 2005), each varying in amplitude and timing, in theory could produce a very large repertoire of smoothly coordinated motor output. Synergies that distribute forces across the fingers can also provide a balance between flexibility and stability (Latash et al., 2007).
Indeed, synergies identified with dimensionality reduction techniques—such as principal component analysis or non-negative matrix factorization—can provide a simplified view of complex movements. Two important scientific questions are (i) whether such synergies are, in fact, used by the nervous system in controlling natural movement, and if so, (ii) in what part(s) of the nervous system the synergies are instantiated. Recent studies indicate that many fundamental synergies may be organized in the brainstem and spinal cord, rather than the cortex (Buford and Davidson, 2004; Cheung et al., 2009; Baker, 2011; Roh et al., 2011; Giszter and Hart, 2013). Once these two questions have been answered, BMI performance might be enhanced by recording from these regions and using the decoded output to drive the relevant synergies rather than the individual degrees of freedom.
Whether or not synergies are used naturally by the nervous system, performance might be improved by incorporating synergies in BMI design. One synergy already in use involves control of arm endpoint in 3 dimensions (i.e., the location of the hand) with a robotic arm that has 4 rotational DOFs: 3 at the shoulder and 1 at the elbow (Lebedev et al., 2005; Velliste et al., 2008; Hochberg et al., 2012). Rather than providing the subject with independent control of all 4 DOFs, BMI output typically drives motion of the arm's endpoint in the 3 Cartesian coordinates (e.g., horizontal, vertical, depth), and this 3-dimensional output is partitioned across the 4 rotational DOFs by a fixed subroutine. Eliminating one DOF in this manner simplifies the control task for the subject at the cost of restricting the ways in which the robotic arm can move. Further incorporation of such simplifying synergies may enable the apparent complexity of movements achieved with BMIs to grow more rapidly than the complexity of control actually required of the subject.
Some synergies might be incorporated even in device hardware. For example, in the majority of hand motion for grasping, the four fingers extend roughly in parallel to open the hand. Rather than providing separate actuators (motors and cables) to extend each finger independently, a robotic hand could have one motor with a cable that divides to attach to each finger, reducing 4 DOFs to 1. Furthermore, extension of the fingers is rarely if ever used to apply substantial forces to objects, this being accomplished with finger flexion. If the extensor cable to each finger could be elastic, then independent flexion of each finger still could be achieved by independent flexion actuators for each finger.
Synergies may be especially useful in controlling a prosthetic hand. Whereas, motion of the shoulder, elbow, and wrist to transport and orient the hand involves 7 rotational DOFs, motion of the thumb and fingers involves 22. Yet even in sophisticated uses of the hand such as typing or piano playing, rarely if ever are individual DOFs moved independently (Soechting and Flanders, 1992, 1997; Engel et al., 1997). Current BMI decoding methods nevertheless assume separate channels for each individual digit, with no relationship between them. And state-of-the-art robotic hands now provide almost as many DOFs as are found in the natural hand (Dalley et al., 2010; Johannes et al., 2011; Resnik et al., 2013; Hutchinson, 2014). For most uses of the hand in activities of daily living, current BMI systems that attempt to control all the DOFs in the hand independently may, in fact, be overly complex.
Nevertheless, identifying an optimal set of synergies for controlling a prosthetic hand is far from simple. An orthogonal basis set of the multiple joints of the hand created with dimensionality reduction has obvious advantages both in its simplicity as well as straight-forward implementation in BMI applications (Ciocarlie et al., 2007; Vinjamuri et al., 2011; Velliste et al., 2012). Alternatively, observation can be used to select a limited set of basis functions. In one recent study, a human subject grasped a variety of objects by controlling a robotic hand through four independent basis functions identified by clinical observation of hand use: (i) pinch between the thumb and index, (ii) flexion and extension of the ring and little fingers in parallel, (iii) ab/adduction of all the fingers, and (iv) opposition of the thumb (Wodlinger et al., 2015). Yet such a simple, orthogonal basis set may fail to capture certain desirable features of hand motion. For example, while the flexion/extension of all five digits is a synergy commonly identified by analysis of hand movements, the thumb and index finger also move more independently than the other three digits. Thus, difficulty arises in trying to design a simple basis set that allows for a single degree of freedom that controls the opening and closing of all digits of the hand while also allowing independent control of the thumb and index finger.
While simplifying control, fixed synergies thus necessarily limit the ability to create all of the diverse movements of which humans are capable. An alternative model by Arbib and colleagues described “virtual fingers” and a schema that added or subtracted the number of digits to a central gripping controller depending on the size and shape of the object to be grasped (Arbib et al., 1985). This virtual fingers model essentially creates different synergies depending on the context of object size and shape. Extending the use of different sets of synergies depending on the particular task or context (e.g., throwing a ball vs. typing) may advance BMI control substantially toward normal human performance. Fully independent control of all the hand's DOFs may be valuable only for the most sophisticated uses of the hand.
More Inputs than Outputs
Redundancy in Neuron Populations
Of the roughly 100 billion neurons in the human cerebral cortex, approximately 1.4 million send axons through the corticospinal tract to synapse on motoneurons and interneurons in the spinal cord (Lassek, 1954). In achieving the remarkable performance of normal humans, these fibers convey much of natural cortical control to the “physical plant,” which consists of approximately 600 muscles and 200 mechanical DOFs at the joints. At present, BMIs can sample only a small subset of any neuron population. As long as the sampling is reasonable, linear decoders can extract a representation of native limb motion and drive BMI end-effectors in multiple DOFs. A current trend, however, is to record increasing numbers of neural signals (Fitzsimmons et al., 2009; Lebedev and Nicolelis, 2011) with the goals not only of controlling more DOFs, but also of making the BMI more robust both to natural variability in neuron firing and to dropout of previously recorded neurons. How does the number of neurons used to control a BMI affect performance?
In the simplest BMI, a single neuron might be used to control each DOF (Fetz and Baker, 1973; Law et al., 2014). Figure 2A schematically illustrates 2 neurons (gray arrows) controlling 2 dimensions, using the familiar population vector approach. Each gray arrow represents a different neuron, the direction of the arrow represents that neuron's preferred direction, and the length of the arrow represents its firing rate. The black arrow represents the resulting population vector, and the cardioid curve indicates the idealized linear model of cosine tuning relative to the current output, i.e., the population vector. This arrangement has a major limitation: the noisiness of the individual neurons will produce equivalent noise in the resultant output.
Figure 2. Population vectors for neural encoding of two variables, x and y. Each neuron is represented by a gray arrow pointing in each cell's preferred direction. The length of each arrow represents the neuron's firing rate. The dotted circle represents a normalized baseline rate for each neuron, with arrow lengths greater than the dotted line representing increased firing rates and shorter lengths representing decreased firing rates. The black arrow represents the population vector sum that predicts the variables, x and y. The solid cardioid curve represents the expected firing rates of perfectly tuned neurons with various preferred directions when x = 0.5 and y = 0.87. (A) Two perfectly tuned neurons can encode x and y exactly, but the noise of the output will be equivalent to the noise of the two inputs. (B) More independently noisy neurons (here 20 neurons) can be combined to generate an accurate estimate of x and y with less noise. (C) When noise is correlated across neurons, adding more neurons to the population may not improve prediction. In this example, noise produced a correlated reduction in the firing rates of most neurons, resulting in a low amplitude prediction (small population vector). (D) In this example, although the neural activity does not match the tuning model (solid cardioid) as accurately as in (B), the predicted x and y values nevertheless are similar. This departure from the tuning model for x and y might represent either increased noise, or the encoding of additional features.
Increasingly precise output can be obtained by linear summation of the observed firing rates of more neurons. Figure 2B illustrates such a situation, now using 20 neurons. In an idealized linear encoding model—where all neurons have the same noise properties and are independent—adding more neurons reduces error. Regardless of the specific partitioning of linear encoding across a population of n neurons, the predicted error decreases as , as follows from the central limit theorem. Adding more neurons thus progressively reduces error albeit requiring progressively increasing numbers of additional neurons for a given magnitude of reduction in error. In practice, as the number of neurons used for BMI control has increased, achieving BMI accuracy that matches idealized encoding behavior has become increasingly difficult. BMI performance tends to plateau once ~50 recorded neurons have been incorporated, after which much less improvement occurs as more neurons are added (Homer et al., 2013; Tehovnik et al., 2013).
This departure from ideal encoding results in part from the fact that neurons typically are not entirely independent of one another. Sampled neurons often have correlations related not only to the encoded signal, but also to other signals. The correlations produced by these other signals—signals that typically are unknown, unmonitored, and not experimentally controlled—constitute noise with respect to the signal being encoded (Lee et al., 1998). When the noise is correlated, even weakly as is typically observed in cortical recordings (Zohary et al., 1994), perfectly accurate linear encoding becomes impossible. Noise correlations, for example, could cause the estimate for a given trial to converge to an incorrect value. Figure 2C illustrates such an example in which a correlated decrease in firing amongst many members of a neuron population causes the magnitude of the resulting population vector to be too small. Even as more and more neurons are added to the decoded population, such noise correlations prevent converging toward perfect decoding.
As more and more neurons are added, incorporating some individual neurons to decode a given variable may actually be detrimental to BMI performance. The relative strength of various signals among different neurons may make adding some neurons corrupted by correlated noise worse than using only those with strong, independent signals. In a BMI application, the ability to generalize control from an initial regression is critical and runs the risk of overfitting if the amount of sampled data is too small for the number of neurons. Also, the ability of the user and/or the decoder to adapt and learn quickly may be diminished as the neural space that must be explored becomes larger. Recognizing such issues, algorithms have been developed to identify those neurons that provide the most independent information, permitting more parsimonious selection of neurons for input to the decoding algorithm (Singhal et al., 2010; Kahn et al., 2011; Xu et al., 2013).
Additionally, the noise levels across a population related to a given variable may not be stationary with time. Figure 2D illustrates a situation in which the individual neurons appear to be relatively noisy. Though the population vector is similar to that of Figure 2B, in Figure 2D the individual neurons appear to deviate randomly from the idealized cosine tuning model indicated by the cardioid curve. Yet this apparent noisiness might represent another signal encoded by the same neuron population (see section Selective Encoding of Variables at Different Times below). How can we make judicious choices regarding the number of neurons used for BMIs?
More Neurons than Controlled DOFs Creates a Null Space
Although the optimal trade-off between the number of neurons recorded and the number of DOFs being controlled by linear BMIs has yet to be well understood, some insight can be gained by considering the simultaneous firing rates of a population of n recorded neurons as an n-dimensional space. When used as a linear signal to control d DOFs, the n-dimensional neural signal is projected into a smaller d-dimensional active control sub-space, leaving a null space of dimensionality m = n–d. Neural activity that projects along the m null-space dimensions has no effect on the d output DOFs. If the neurons are independent predictors, then most noise tends to result in changes in this null space of the joint neural state, allowing an ensemble of “noisy” neural signals to encode the output more precisely.
Yet the null space may be more than a repository for noise. Much of what appears to be noise related to a given signal may represent neural activity related to other signals encoded by the same population of neurons. Below we explore two other potentially valuable aspects of the null space that permit: (i) motor learning with rapid flexibility, and (ii) non-linear encoding that repartitions the active control space vs. the null space depending on the phase of movement.
The Null Space and Motor Learning with Rapid Flexibility
While the null-space can be considered an “uncontrolled manifold” in a particular movement scenario (Scholz and Schöner, 1999; Latash et al., 2007), in the case of a neuronal state space, the uncontrolled manifold is not entirely uncontrolled. During both natural arm movements and BMI output, the joint neural state tends to follow a subset of preferred trajectories that use a subset of the null space, rather than using all possible trajectories distributed throughout the n-dimensional neural space that could provide an equivalent output (Kaufman et al., 2014; Law et al., 2014). Preferred trajectories through the neural state space may reflect the network architecture of pre-existing synaptic connections in which the neurons are embedded (Sadtler et al., 2014). Yet to control novel BMIs, monkeys can learn to use relatively novel neural trajectories (Jarosiewicz et al., 2008; Ganguly and Carmena, 2009; Ganguly et al., 2011; Law et al., 2014). A similar process of learning to use novel neural trajectories may underlie the natural process of learning new motor skills and then switching rapidly at will between one skill and another, according to the context. The relatively large number of neurons, n > d, is no longer entirely redundant when additional trajectories through the neural state space must be utilized in additional contexts. Finding and utilizing such additional trajectories might entail learning to associate a previously learned trajectory with a new context, modifying a previously learned trajectory for use in a new context, or learning an entirely new trajectory through the neural state space.
Some evidence that motor learning and rapid flexibility in various contexts involve changes in neural trajectories can be gleaned from studies that use spike-triggered averaging of EMG activity to assess functional connectivity among those neurons that provide last-order inputs to particular motoneuron pools. Broad synchrony facilitations in spike-triggered averages of EMG activity provide evidence that synchronous spikes are discharged by multiple neurons with inputs to the same motoneuron pool, indicating that groups of such neurons receive common or serial inputs (Baker and Lemon, 1998; Schieber and Rivlis, 2005). Among M1 cortico-motoneuronal (CM) cells, such spike synchronization is most common between CM-cells that have output effects in similar sets of muscles (muscle fields), suggesting that these groups of neurons may be recruited together to facilitate a particular set of muscles (Jackson et al., 2003). The prevalence of synchrony effects in M1 neurons increases with long-term training at an individuated finger movement task (Schieber, 2002), suggesting that such long-term training increases the common inputs to neurons that all input in turn to the same motoneuron pool. This change in common inputs will alter the neural trajectory during the practiced movements. Yet the size of synchrony effects also can change rapidly when novel motor behaviors are being performed (Davidson et al., 2007), suggesting that different common inputs to the population of neurons with last-order inputs to a given motoneuron pool become active in different contexts, again indicating different neural trajectories for different contexts. These rapid changes depending on context may involve processing in cortical minicolumns and corticostriatal circuits, where the level of functional connectivity (as measured by cross-correlations between simultaneously recorded spikes in layer 2/3 and layer 5 of a minicolumn, for example) has been found to vary with the type and difficulty of the task being performed (Opris et al., 2012, 2013; Santos et al., 2014). For optimal performance, BMIs will need to take into account such context-dependent changes in neuron activity and the changes in neural trajectories they represent.
Selective Encoding of Variables at Different Times
This idea that the firing of an individual neuron simultaneously carries representations of multiple motor parameters has been widely accepted in neurophysiological studies for some time (Humphrey et al., 1970; Thach, 1978). In general, however, neurophysiological studies make the implicit assumption that the relative weighting of the encoded motor variables remains stationary over the time course of single movements and entire sessions. The same assumption typically is made in current BMI controllers.
Yet some neurophysiological studies have indicated that the motor variables being encoded are not constant, even within a single movement. Time-resolved linear regression, for example, demonstrated that in single M1 and premotor cortex neurons, direction is represented most strongly early in the course of reaching movements, target position is represented most strongly later, and distance is represented most strongly still later in the same movements (Fu et al., 1995). Similarly, using time-resolved analysis of variance, we found recently that the firing rates of M1 neurons vary depending on location early in the course of reach-to-grasp movements, and then on the hand shape used to grasp the object later (Rouse and Schieber, 2014). Such changes in the strength of representation of different features can be viewed as rotations of the d controlled dimensions in the n-dimensional neural state space, which will repartition the m null-space dimensions across the time of a single movement. Hence different variables are represented more or less selectively at different phases of a single movement, and the apparent “null” space provides room for such rotations to occur.
Figure 3 and the Supplemental Video both illustrate this hypothetical repartitioning of the active control space vs. the null space, again using the familiar population vector approach. As in the 2-dimensional examples of Figure 2, each arrow represents a different neuron, the direction of the arrow represents that neuron's preferred direction, and the length of the arrow represents its firing rate. In Figure 2, the d = 2 active dimensions were represented as the ordinate and abscissa of the plot, and the output of the single, linear model depended only on the resulting population vector sum in those two dimensions. Similarly in Figure 3A, we plot a population of 20 neurons (colored arrows) projected into the plane of two output dimensions, x and y, and the resulting population vector (black). As in Figure 2B, at the point in time illustrated in Figure 3A each individual neuron's firing rate closely matches the idealized linear model represented by the cardioid curve drawn to indicate the cosine tuning of individual neurons to the variables x and y relative to the population vector.
Figure 3. Multi-dimensional population vector selectively encoding two pairs of variables at different times. Each of 20 neurons is represented by a colored arrow pointing in its preferred direction. The length of each arrow represents that neuron's firing rate. The two planes, x⊥y (tan) and u⊥v (blue), each show the individual neuron's firing rates and preferred directions projected into the two pairs of dimensions. The population vector sum (heavy black line) projected into each plane represents the population's estimate of the two pairs of variables: x and y, u and v. The black cardioid curve represents the predicted firing rates given the population vector if each neuron was ideally cosine-tuned to a given pair of variables (x and y, or u and v). The two panels (A,B) represent two different points in time. At the first time point (A), the individual neuron firing rates are related most closely to x and y, matching the idealized cosine tuning to those two variables, as represented by the cardioid in that plane. But at the second time point (B), the firing rates of the same 20 neurons are related most closely to u and v. The firing rates of these neurons may not be representing both pairs of variables equally at all points in time, but rather selectively encoding one pair of variables at each time. At either time point, if a single linear decoder were used to estimate all four dimensions—x, y, u, and v—simultaneously from the population, one pair would be estimated accurately and the other pair inaccurately. But using two different decoders—one to estimate x and y, the other to estimate u and v—and then selecting the currently decoded output by assessing which idealized model is better fit at the time, would enable more accurate decoding overall. The Supplemental Video provides an animated version of this Figure. (N.B. To illustrate a 4-dimensional space in 3-dimensions, we have made u linearly dependent on x and y in these images; but in the actual high-dimensional neural space, all four variables can be linearly independent).
The remaining m = 20–2 = 18-dimensional null space has no direct effect on the output (population vector) of this model in this x⊥y plane. But consider the neural activity in two of these null dimensions, u and v. The neuron firing rates and resulting population vector projected in the u⊥v plane also are illustrated in Figure 3A. A different cardioid curve is shown here to represent the cosine tuning that pertains when the population vector is calculated using a second idealized model based on the projections of the neuron firing rates in the u⊥v plane. In the u⊥v plane, the firing rates of the individual neurons match this second model poorly, not unlike the example of Figure 2D.
Now consider a different point in time when the individual neurons are firing at different rates, illustrated in Figure 3B. Now the firing rates match the cardioid in the x⊥y plane poorly, but match the cardioid in the u⊥v plane well. The same neuron population that encoded x and y previously, now is encoding u and v. We hypothesize that in this manner a given neuron population may encode different variables selectively during different phases of a single movement. In such a construct, treating the 20 neuron population as a 4-dimensional output space that continuously encodes x, y, u, and v plus a 16-dimensional null space would be suboptimal. For BMI purposes, decoding different variables at different points in time would provide more accurate output for the two variables as each becomes most relevant to the current phase (or context) of movement. At each point in time, the model most heavily weighted in the BMI output could be selected by having the computer assess which idealized model is better fit by the neuron firing rates currently being generated by the brain.
Such selective neural encoding of different sets of variables at different times might be dismissed as simply reflecting an inability to create a single linear model fitting all the observed firing rates well. An inaccurate single model might result from having insufficient data (numbers of trials), insufficient numbers of neurons (e.g., n >> 20 is needed), or insufficient sampling of the high-dimensional parameter space (…u, v, …x, y…). Yet if a given population of neurons does indeed represent different output features at different times, applying a single linear model cannot achieve high accuracy at all points in time regardless of how many neurons or how much data are incorporated. Non-linear models will be needed to repartition the active control space vs. the null space.
As reviewed above (section The Null Space and Motor Learning with Rapid Flexibility), we envision that during natural behavior the nervous system achieves the non-linear repartitioning that selects different controlled variables by coactivating, or even synchronizing, various subpopulations of neurons at different times. In view of the wide range of inputs that impinge on a single α-motoneuron (including Ia afferents, Ia-inhibitory interneurons, excitatory spinal interneurons, and CM-cells) or that affect a single M1 neuron (including inputs from the primary somatosensory cortex, ventral premotor cortex, dorsal premotor cortex, supplementary motor area, and thalamus), we hypothesize that non-linear repartitioning may involve various inputs predominating at different times, with synaptic summation that is not necessarily linear. The extent to which the nervous system naturally uses non-linear repartitioning to output different features from the same neuron population at different times remains an open scientific question.
In any case, BMI performance might be improved by implementing methods that allow for selective encoding of the variables most relevant to the current phase of movement. This could be achieved by applying different decoding algorithms to sequentially capture different features of complex movements (Ethier et al., 2011; Jiang et al., 2011; Srinivasan and da Silva, 2011; Shanechi et al., 2012; Aggarwal et al., 2013; Kang et al., 2015). Rather than continuously decoding all DOF simultaneously, the BMI controller might use population neural activity to encode only a subset of variables while other variables are either passively dampened or even held constant by the BMI. Controlling a robotic arm and hand to reach and grasp, for example, might be improved by using neural activity sequentially, first to encode the reach location to which the arm transports the hand, and then as the hand arrives near the object, switching to encode the grasp while damping further movement of the arm. This could allow the user to focus on grasping the object precisely, without the distraction of simultaneously continuing to control the entire arm.
Conclusions
A general theme that emerges from our considerations is that natural motor control is not a single process that applies universally in all situations. The control of small, fine movements differs from that of large, gross movements. The control of posture is not achieved by producing movement with zero velocity. Many movements may be controlled through small numbers of synergies, and only the most sophisticated performances may require individuated control of large numbers of DOFs. In generating complex movements, the same neuronal population may transmit information on different sets of output variables sequentially rather than simultaneously. To advance the performance of BMIs further toward that of normal humans will require similar strategies that go beyond one-size-fits-all, linear state space models.
Achieving such advances necessarily will require increasing the complexity of BMI controllers. Decoding will need to be more flexible and applied differently at different times, possibly driven by inputs recorded from different parts of the nervous system. Moreover, supervisory algorithms will be needed to identify the contexts and movement phases that define this dynamic relationship between neural signals and output DOFs. Implementing such designs will go further to translate our knowledge of natural motor control physiology, advancing BMIs toward normal human performance.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
This work was funded by National Institutes of Health R01s NS065902 and NS079664. The authors thank the anonymous reviewers for helpful suggestions and Marsha Hayles for editorial comments.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/article/10.3389/fnsys.2015.00108
References
Aggarwal, V., Mollazadeh, M., Davidson, A. G., Schieber, M. H., and Thakor, N. V, (2013). State-based decoding of hand and finger kinematics using neuronal ensemble and LFP activity during dexterous reach-to-grasp movements. J. Neurophysiol. 109, 3067–3081. doi: 10.1152/jn.01038.2011
Ajiboye, A. B., Simeral, J. D., Donoghue, J. P., Hochberg, L. R., and Kirsch, R. F. (2012). Prediction of imagined single-joint movements in a person with high-level tetraplegia. IEEE Trans. Biomed. Eng. 59, 2755–2765. doi: 10.1109/TBME.2012.2209882
Anderson, W. S., Weiss, N., Lawson, H. C., Ohara, S., Rowland, L., and Lenz, F. A. (2011). Demonstration of motor imagery- and phantom-movement related neuronal activity in human thalamus. Neuroreport 22, 88–92. doi: 10.1097/WNR.0b013e328342c98a
Arbib, M. A., Iberall, T., and Lyons, D. (1985). “Coordinated control programs for movements of the hand,” in Hand Function and the Neocotex (Experimental Brain Research Series), eds A. W. Goodwin and I. Darian-Smith (Heidelberg: Springer–Verlag), 111–129.
Baker, S. N. (2011). The primate reticulospinal tract, hand function and functional recovery. J. Physiol. 589, 5603–5612. doi: 10.1113/jphysiol.2011.215160
Baker, S. N., and Lemon, R. N. (1998). Computer simulation of post-spike facilitation in spike-triggered averages of rectified EMG. J. Neurophysiol. 80, 1391–1406.
Bernstein, N. A. (1967). The Co-ordination and Regulation of Movements, 1st Edn. Oxford: Pergamon Press.
Buford, J. A., and Davidson, A. G. (2004). Movement-related and preparatory activity in the reticulospinal system of the monkey. Exp. Brain Res. 159, 284–300. doi: 10.1007/s00221-004-1956-4
Card, S. K., English, W. K., and Burr, B. J. (1978). Evaluation of mouse, rate-controlled isometric joystick, step keys, and text keys for text selection on a CRT. Ergonomics 21, 601–613. doi: 10.1080/00140137808931762
Carmena, J. M., Lebedev, M. A., Crist, R. E., O'Doherty, J. E., Santucci, D. M., Dimitrov, D. F., et al. (2003). Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol 1:E42. doi: 10.1371/journal.pbio.0000042
Casile, A. (2013). Mirror neurons (and beyond) in the macaque brain: an overview of 20 years of research. Neurosci. Lett. 540, 3–14. doi: 10.1016/j.neulet.2012.11.003
Chadwick, E. K., Blana, D., Simeral, J. D., Lambrecht, J., Kim, S. P., Cornwell, A. S., et al. (2011). Continuous neuronal ensemble control of simulated arm reaching by a human with tetraplegia. J. Neural Eng. 8:034003. doi: 10.1088/1741-2560/8/3/034003
Chase, S. M., and Schwartz, A. B. (2011). “Inference from populations: going beyond models,” in Progress in Brain Research, eds A. M. Green, C. E. Chapman, J. F. Kalaska, and F. Lepore (Amsterdam: Elsevier), 103–112. doi: 10.1016/B978-0-444-53355-5.00007-5
Chestek, C. A., Gilja, V., Nuyujukian, P., Foster, J. D., Fan, J. M., Kaufman, M. T., et al. (2011). Long-term stability of neural prosthetic control signals from silicon cortical arrays in rhesus macaque motor cortex. J. Neural Eng. 8:045005. doi: 10.1088/1741-2560/8/4/045005
Cheung, V. C. K., Piron, L., Agostini, M., Silvoni, S., Turolla, A., and Bizzi, E. (2009). Stability of muscle synergies for voluntary actions after cortical stroke in humans. Proc. Natl. Acad. Sci. U. S. A. 106, 19563–19568. doi: 10.1073/pnas.0910114106
Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Foster, J. D., Nuyujukian, P., Ryu, S. I., et al. (2012). Neural population dynamics during reaching. Nature 487, 51–56. doi: 10.1038/nature11129
Ciocarlie, M., Goldfeder, C., and Allen, P. (2007). Dimensionality reduction for hand-independent dexterous robotic grasping. 2007 IEEE/RSJ Int. Conf. Intell. Robot. Syst. 20, 3270–3275. doi: 10.1109/IROS.2007.4399227
Cisek, P., and Kalaska, J. F. (2004). Neural correlates of mental rehearsal in dorsal premotor cortex. Nature 431, 993–996. doi: 10.1038/nature03005
Collinger, J. L., Wodlinger, B., Downey, J. E., Wang, W., Tyler-Kabara, E. C., Weber, D. J., et al. (2013). High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381, 557–564. doi: 10.1016/S0140-6736(12)61816-9
Cunningham, J. P., Gilja, V., Ryu, S. I., and Shenoy, K. V (2010). Methods for estimating neural firing rates, and their application to brain-machine interfaces. Neural Netw. 22, 249–266. doi: 10.1016/j.neunet.2009.02.004
Dalley, S. A., Wiste, T. E., Varol, H. A., and Goldfarb, M. (2010). A multigrasp hand prosthesis for transradial amputees. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2010, 5062–5065. doi: 10.1109/iembs.2010.5626225
Davidson, A. G., Chan, V., O'Dell, R., and Schieber, M. H. (2007). Rapid changes in throughput from single motor cortex neurons to muscle activity. Science 318, 1934–1937. doi: 10.1126/science.1149774
d'Avella, A., and Bizzi, E. (2005). Shared and specific muscle synergies in natural motor behaviors. Proc. Natl. Acad. Sci. U. S. A. 102, 3076–3081. doi: 10.1073/pnas.0500199102
Dushanova, J., and Donoghue, J. P. (2010). Neurons in primary motor cortex engaged during action observation. Eur. J. Neurosci. 31, 386–398. doi: 10.1111/j.1460-9568.2009.07067.x
Engel, K. C., Flanders, M., and Soechting, J. F. (1997). Anticipatory and sequential motor control in piano playing. Exp. brain Res. 113, 189–199. doi: 10.1007/BF02450317
Epps, B. W. (1986). Comparison of six cursor control devices based on Fitts' law models. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 30, 327–331. doi: 10.1177/154193128603000403
Ersland, L., Rosén, G., Lundervold, A., Smievoll, A. I., Tillung, T., Sundberg, H., et al. (1996). Phantom limb imaginary fingertapping causes primary motor cortex activation: an fMRI study. Neuroreport 8, 207–210. doi: 10.1097/00001756-199612200-00042
Ethier, C., Oby, E. R., Bauman, M. J., and Miller, L. E. (2012). Restoration of grasp following paralysis through brain-controlled stimulation of muscles. Nature 485, 368–371. doi: 10.1038/nature10987
Ethier, C., Sachs, N. A., Miller, L. E., and Collection, A. D. (2011). Continuous state-dependent decoders for brain machine interfaces. Neural Eng. (NER), 2011 5th International IEEE/EMBS Conference on IEEE (Cancun), 473–477. doi: 10.1109/NER.2011.5910589
Felton, E. A., Radwin, R. G., Wilson, J. A., and Williams, J. C. (2009). Evaluation of a modified Fitts law brain-computer interface target acquisition task in able and motor disabled individuals. J. Neural Eng. 6:056002. doi: 10.1088/1741-2560/6/5/056002
Fetz, E. E., and Baker, M. A. (1973). Operantly conditioned patterns of precentral unit activity and correlated responses in adjacent cells and contralateral muscles. J. Neurophysiol. 36, 179–204.
Fetz, E. E., Perlmutter, S. I., Prut, Y., Seki, K., and Votaw, S. (2002). Roles of primate spinal interneurons in preparation and execution of voluntary hand movement. Brain Res. Rev. 40, 53–65. doi: 10.1016/S0165-0173(02)00188-1
Fitts, P. M. (1954). The information capacity of human motor system in controlling the ampltude of movement. J. Exp. Psychol. 47, 381–391. doi: 10.1037/h0055392
Fitzsimmons, N. A., Lebedev, M. A., Peikon, I. D., and Nicolelis, M. A. L. (2009). Extracting kinematic parameters for monkey bipedal walking from cortical neuronal ensemble activity. Front. Integr. Neurosci. 3:3. doi: 10.3389/neuro.07.003.2009
Flanders, M., and Soechting, J. F. (1992). Kinematics of typing: parallel control of the two hands. J. Neurophysiol. 67, 1264–1274.
Fraser, G. W., Chase, S. M., Whitford, A., and Schwartz, A. B. (2009). Control of a brain-computer interface without spike sorting. J. Neural Eng. 6:055004. doi: 10.1088/1741-2560/6/5/055004
Fu, Q. G., Flament, D., Coltz, J. D., and Ebner, T. J. (1995). Temporal encoding of movement kinematics in the discharge of primate primary motor and premotor neurons. J. Neurophysiol. 73, 836–854.
Ganguly, K., and Carmena, J. M. (2009). Emergence of a stable cortical map for neuroprosthetic control. PLoS Biol. 7:e1000153. doi: 10.1371/journal.pbio.1000153
Ganguly, K., Dimitrov, D. F., Wallis, J. D., and Carmena, J. M. (2011). Reversible large-scale modification of cortical networks during neuroprosthetic control. Nat. Neurosci. 14, 662–667. doi: 10.1038/nn.2797
Georgopoulos, A. P. (2014). Cell directional spread determines accuracy, precision, and length of the neuronal population vector. Exp. Brain Res. 232, 2391–2405. doi: 10.1007/s00221-014-3936-7
Georgopoulos, A. P., and Grillner, S. (1989). Visuomotor coordination in reaching and locomotion. Science 245, 1209–1210. doi: 10.1126/science.2675307
Gilja, V., Nuyujukian, P., Chestek, C. A., Cunningham, J. P., Yu, B. M., Fan, J. M., et al. (2012). A high-performance neural prosthesis enabled by control algorithm design. Nat. Neurosci. 15, 1752–1757. doi: 10.1038/nn.3265
Giszter, S. F., and Hart, C. B. (2013). Motor primitives and synergies in the spinal cord and after injury–the current state of play. Ann. N. Y. Acad. Sci. 1279, 114–126. doi: 10.1111/nyas.12065
Grafton, S. T., Arbib, M. A., Fadiga, L., and Rizzolatti, G. (1996). Localization of grasp representations in humans by positron emission tomography. 2. Observation compared with imaginations. Exp. Brain Res. 112, 103–111. doi: 10.1007/BF00227183
Grillner, S. (1985). Neurobiological bases of rhythmic motor acts in vertebrates. Science 228, 143–149. doi: 10.1126/science.3975635
Hager-Ross, C., and Schieber, M. H. (2000). Quantifying the independence of human finger movements: comparisons of digits, hands, and movement frequencies. J. Neurosci. 20, 8542–8550.
Hall, T. M., de Carvalho, F., and Jackson, A. (2014). A common structure underlies low-frequency cortical dynamics in movement, sleep, and sedation. Neuron 83, 1185–1199. doi: 10.1016/j.neuron.2014.07.022
Harris, C. M., and Wolpert, D. M. (1998). Signal-dependent noise determines motor planning. Nature 394, 780–784. doi: 10.1038/29528
Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., et al. (2012). Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485, 372–375. doi: 10.1038/nature11076
Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., et al. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171. doi: 10.1038/nature04970
Homer, M. L., Nurmikko, A. V., Donoghue, J. P., and Hochberg, L. R. (2013). Sensors and decoding for intracortical brain computer interfaces. Annu. Rev. Biomed. Eng. 15, 383–405. doi: 10.1146/annurev-bioeng-071910-124640
Humphrey, D. R., and Reed, D. J. (1983). Separate cortical systems for control of joint movement and joint stiffness: reciprocal activation and coactivation of antagonist muscles. Adv. Neurol. 39, 347–372.
Humphrey, D. R., Schmidt, E. M., and Thompson, W. D. (1970). Predicting measures of motor performance from multiple cortical spike trains. Science 170, 758–762. doi: 10.1126/science.170.3959.758
Hutchinson, D. T. (2014). The quest for the bionic arm. J. Am. Acad. Orthop. Surg. 22, 346–351. doi: 10.5435/JAAOS-22-06-346
Ifft, P. J., Lebedev, M. A., and Nicolelis, M. A. L. (2011). Cortical correlates of fitts' law. Front. Integr. Neurosci. 5:85. doi: 10.3389/fnint.2011.00085
Jackson, A., Gee, V. J., Baker, S. N., and Lemon, R. N. (2003). Synchrony between neurons with similar muscle fields in monkey motor cortex. Neuron 38, 115–125. doi: 10.1016/S0896-6273(03)00162-4
Jagacinski, R. J., and Monk, D. M. (1985). Fitts' law in two dimensions with hand and head movements. J. Mot. Behav. 17, 77–95. doi: 10.1080/00222895.1985.10735338
Jarosiewicz, B., Chase, S. M., Fraser, G. W., Velliste, M., Kass, R. E., and Schwartz, A. B. (2008). Functional network reorganization during learning in a brain-computer interface paradigm. Proc. Natl. Acad. Sci. U.S.A. 105, 19486–19491. doi: 10.1073/pnas.0808113105
Jiang, B., Wang, R., Zhang, Q., Zhang, J., Zheng, X., and Zhao, T. (2011). A pilot study on two stage decoding strategies. Proc. IEEE EMBS Conf. 2011, 3700–3703. doi: 10.1109/iembs.2011.6090627
Johannes, M. S., Bigelow, J. D., Burck, J. M., Harshbarger, S. D., and Kozlowski, M. V (2011). An overview of the developmental process for the modular prosthetic limb. Johns Hopkins APL Tech. Dig. 30, 207–216.
Kahn, K., Schieber, M. H., Thakor, N. V., and Sarma, S., V (2011). Neuron selection for decoding dexterous finger movements. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2011, 4605–4608. doi: 10.1109/IEMBS.2011.6091140
Kang, N., Shinohara, M., Zatsiorsky, V. M., and Latash, M. L. (2004). Learning multi-finger synergies: an uncontrolled manifold analysis. Exp. Brain Res. 157, 336–350. doi: 10.1007/s00221-004-1850-0
Kang, X., Sarma, S., Santaniello, S., Schieber, M. H., and Thakor, N. (2015). Task-independent cognitive state transition detection from cortical neurons during 3D reach-to-grasp movements. IEEE Trans. Neural Syst. Rehabil. Eng. 23, 676–682. doi: 10.1109/TNSRE.2015.2396495
Kaufman, M. T., Churchland, M. M., Ryu, S. I., and Shenoy, K. V (2014). Cortical activity in the null space: permitting preparation without movement. Nat. Neurosci. 17, 440–448. doi: 10.1038/nn.3643
Kemere, C., Santhanam, G., Yu, B. M., Afshar, A., Ryu, S. I., Meng, T. H., et al. (2008). Detecting neural-state transitions using hidden Markov models for motor cortical prostheses. J. Neurophysiol. 100, 2441–2452. doi: 10.1152/jn.00924.2007
Lassek, A. M. (1954). The Pyramidal Tract: Its Status in Medicine. Springfield, IL: Charles C. Thomas.
Latash, M. L., Scholz, J. P., Danion, F., and Schöner, G. (2002). Finger coordination during discrete and oscillatory force production tasks. Exp. Brain Res. 146, 419–432. doi: 10.1007/s00221-002-1196-4
Latash, M. L., Scholz, J. P., and Schöner, G. (2007). Toward a new theory of motor synergies. Motor Control 11, 276–308.
Law, A. J., Rivlis, G., and Schieber, M. H. (2014). Rapid acquisition of novel interface control by small ensembles of arbitrarily selected primary motor cortex neurons. J. Neurophysiol. 112, 1528–1548. doi: 10.1152/jn.00373.2013
Lebedev, M. A., Carmena, J. M., O'Doherty, J. E., Zacksenhouse, M., Henriquez, C. S., Principe, J. C., et al. (2005). Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain-machine interface. J. Neurosci. 25, 4681–4693. doi: 10.1523/JNEUROSCI.4088-04.2005
Lebedev, M. A., and Nicolelis, M. A. L. (2011). Toward a whole-body neuroprosthetic. Prog. Brain Res. 194, 47–60. doi: 10.1016/B978-0-444-53815-4.00018-2
Lee, D., Port, N. L., Kruse, W., and Georgopoulos, A. P. (1998). Variability and correlated noise in the discharge of neurons in motor and parietal areas of the primate cortex. J. Neurosci. 18, 1161–1170.
Li, Z., O'Doherty, J. E., Hanson, T. L., Lebedev, M. A., Henriquez, C. S., and Nicolelis, M. A. L. (2009). Unscented Kalman filter for brain-machine interfaces. PLoS ONE 4:e6243. doi: 10.1371/journal.pone.0006243
MacKenzie, I. S. (1992). Fitts' law as a research and design tool in human-computer interaction. Hum. Comput. Interact. 7, 91–139. doi: 10.1207/s15327051hci0701_3
Mahan, M. Y., and Georgopoulos, A. P. (2013). Motor directional tuning across brain areas: directional resonance and the role of inhibition for directional accuracy. Front. Neural Circuits 7:92. doi: 10.3389/fncir.2013.00092
Maier, M. A., Perlmutter, S. I., and Fetz, E. E. (1998). Response patterns and force relations of monkey spinal interneurons during active wrist movement. J. Neurophysiol. 80, 2495–2513.
Mason, C. R., Theverapperuma, L. S., Hendrix, C. M., and Ebner, T. J. (2004). Monkey hand postural synergies during reach-to-grasp in the absence of vision of the hand and object. J. Neurophysiol. 91, 2826–2837. doi: 10.1152/jn.00653.2003
Moran, D. W., and Schwartz, A. B. (1999). Motor cortical representation of speed and direction during reaching. J. Neurophysiol. 82, 2676–2692.
Moritz, C. T., Perlmutter, S. I., and Fetz, E. E. (2008). Direct control of paralysed muscles by cortical neurons. Nature 456, 639–642. doi: 10.1038/nature07418
Müller, K., Anderson, C. W., and Birch, G. E. (2003). Linear and nonlinear methods for brain-computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 165–169. doi: 10.1109/TNSRE.2003.814484
Opris, I., Fuqua, J. L., Huettl, P. F., Gerhardt, G. A., Berger, T. W., Hampson, R. E., et al. (2012). Closing the loop in primate prefrontal cortex: inter-laminar processing. Front. Neural Circuits 6:88. doi: 10.3389/fncir.2012.00088
Opris, I., Santos, L., Gerhardt, G. A., Song, D., Berger, T. W., Hampson, R. E., et al. (2013). Prefrontal cortical microcircuits bind perception to executive control. Sci. Rep. 3:2285. doi: 10.1038/srep02285
Orsborn, A. L., Dangi, S., Moorman, H. G., and Carmena, J. M. (2012). Closed-loop decoder adaptation on intermediate time-scales facilitates rapid BMI performance improvements independent of decoder initialization conditions. IEEE Trans. Neural Syst. Rehabil. Eng. 20, 468–477. doi: 10.1109/TNSRE.2012.2185066
Pancrazio, J. J., and Peckham, P. H. (2009). Neuroprosthetic devices: how far are we from recovering movement in paralyzed patients? Expert Rev. Neurother. 9, 427–430. doi: 10.1586/ern.09.12
Pfurtscheller, G., and Neuper, C. (1997). Motor imagery activates primary sensorimotor area in humans. Neurosci. Lett. 239, 65–68. doi: 10.1016/S0304-3940(97)00889-6
Porro, C. A., Francescato, M. P., Cettolo, V., Diamond, M. E., Baraldi, P., Zuiani, C., et al. (1996). Primary motor and sensory cortex activation during motor performance and motor imagery: a functional magnetic resonance imaging study. J. Neurosci. 16, 7688–7698.
Resnik, L., Klinger, S. L., and Etter, K. (2013). The DEKA Arm: Its features, functionality, and evolution during the Veterans Affairs Study to optimize the DEKA Arm. Prosthet. Orthot. Int. 38, 492–504. doi: 10.1177/0309364613506913
Roh, J., Cheung, V. C. K., and Bizzi, E. (2011). Modules in the brain stem and spinal cord underlying motor behaviors. J. Neurophysiol. 106, 1363–1378. doi: 10.1152/jn.00842.2010
Rouse, A. G., and Schieber, M. H. (2014). “Location then object representations sequentially predominate in the widely distributed activation of the primary motor cortex during reach to grasp,” in Society for Neuroscience Annual Meeting (Washington, DC).
Sadtler, P. T., Quick, K. M., Golub, M. D., Chase, S. M., Ryu, S. I., Tyler-Kabara, E. C., et al. (2014). Neural constraints on learning. Nature 512, 423–426. doi: 10.1038/nature13665
Santello, M., Flanders, M., and Soechting, J. F. (1998). Postural hand synergies for tool use. J. Neurosci. 18, 10105–10115.
Santos, L., Opris, I., Hampson, R., Godwin, D. W., Gerhardt, G. A., and Deadwyler, S. (2014). Functional dynamics of primate cortico-striatal networks during volitional movements. Front. Syst. Neurosci. 8:27. doi: 10.3389/fnsys.2014.00027
Schieber, M. H., and Rivlis, G. (2005). A spectrum from pure post-spike effects to synchrony effects in spike-triggered averages of electromyographic activity during skilled finger movements. J. Neurophysiol. 94, 3325–3341. doi: 10.1152/jn.00007.2005
Scholz, J. P., Reisman, D., and Schöner, G. (2001). Effects of varying task constraints on solutions to joint coordination in a sit-to-stand task. Exp. Brain Res. 141, 485–500. doi: 10.1007/s002210100878
Scholz, J. P., and Schöner, G. (1999). The uncontrolled manifold concept: identifying control variables for a functional task. Exp. Brain Res. 126, 289–306. doi: 10.1007/s002210050738
Seki, K., Perlmutter, S. I., and Fetz, E. E. (2003). Sensory input to primate spinal cord is presynaptically inhibited during voluntary movement. Nat. Neurosci. 6, 1309–1316. doi: 10.1038/nn1154
Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R., and Donoghue, J. P. (2002). Instant neural control of a movement signal. Nature 416, 141–142. doi: 10.1038/416141a
Shalit, U., Zinger, N., Joshua, M., and Prut, Y. (2012). Descending systems translate transient cortical commands into a sustained muscle activation signal. Cereb. Cortex 22, 1904–1914. doi: 10.1093/cercor/bhr267
Shanechi, M. M., Hu, R. C., Powers, M., Wornell, G. W., Brown, E. N., and Williams, Z. M. (2012). Neural population partitioning and a concurrent brain-machine interface for sequential motor function. Nat. Neurosci. 15, 1715–1722. doi: 10.1038/nn.3250
Shanechi, M. M., Orsborn, A., Moorman, H., Gowda, S., and Carmena, J. M. (2014). High-performance brain-machine interface enabled by an adaptive optimal feedback-controlled point process decoder. Proc. IEEE EMBS Conf. 36, 6493–6496. doi: 10.1109/embc.2014.6945115
Shik, M. L., and Orlovsky, G. N. (1976). Neurophysiology of locomotor automatism. Physiol. Rev. 56, 465–501.
Singhal, G., Aggarwal, V., Acharya, S., Aguayo, J., He, J., and Thakor, N. (2010). Ensemble fractional sensitivity: a quantitative approach to neuron selection for decoding motor tasks. Comput. Intell. Neurosci. 2010, 648202. doi: 10.1155/2010/648202
Slifkin, A. B., and Newell, K. M. (1999). Noise, information transmission, and force variability. J. Exp. Psychol. Hum. Percept. Perform. 25, 837–851. doi: 10.1037/0096-1523.25.3.837
Soechting, J. F., and Flanders, M. (1992). Organization of sequential typing movements. J. Neurophysiol. 67, 1275–1290.
Soechting, J. F., and Flanders, M. (1997). Flexibility and repeatability of finger movements during typing: analysis of multiple degrees of freedom. J. Comput. Neurosci. 4, 29–46. doi: 10.1023/A:1008812426305
Srinivasan, L., and da Silva, M. (2011). Breaking the fixed-arrival-time restriction in reaching movements of neural prosthetic devices. IEEE Trans. Biomed. Eng. 58, 1555–1564. doi: 10.1109/TBME.2010.2101599
Stein, P. S. G. (1978). Motor systems, with specific reference to the control of locomotion. Annu. Rev. Neurosci. 1, 61–81. doi: 10.1146/annurev.ne.01.030178.000425
Stieglitz, T., Rubehn, B., Henle, C., Kisban, S., Herwik, S., Ruther, P., et al. (2009). “Brain-computer interfaces: an overview of the hardware to record neural signals from the cortex,” in Progress in Brain Research, eds J. Verhaagen, E. M. Hol, I. Huitenga, J. Wijholds, A. B. Bergen, G. J. Boer, and D. J. Swaab (Amsterdam: Elsevier), 297–315. doi: 10.1016/S0079-6123(09)17521-0
Taylor, D. M., Tillery, S. I. H., and Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science 296, 1829–1832. doi: 10.1126/science.1070291
Tehovnik, E. J., Woods, L. C., and Slocum, W. M. (2013). Transfer of information by BMI. Neuroscience 255, 134–146. doi: 10.1016/j.neuroscience.2013.10.003
Thach, W. T. (1978). Correlation of neural discharge with pattern and force of muscular activity, joint position, and direction of intended next movement in motor cortex and cerebellum. J. Neurophysiol. 41, 654–676.
Tkach, D., Reimer, J., and Hatsopoulos, N. G. (2007). Congruent activity during action and action observation in motor cortex. J. Neurosci. 27, 13241–13250. doi: 10.1523/JNEUROSCI.2895-07.2007
Tseng, Y., Scholz, J. P., and Schöner, G. (2002). Goal-equivalent joint coordination in pointing: affect of vision and arm dominance. Motor Control 6, 183–204.
Tsianos, G. A., Goodner, J., and Loeb, G. E. (2014). Useful properties of spinal circuits for learning and performing planar reaches. J. Neural Eng. 11:056006. doi: 10.1088/1741-2560/11/5/056006
Velliste, M., McMorland, A. J. C., Diril, E., Clanton, S. T., and Schwartz, A. B. (2012). State-space control of prosthetic hand shape. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2012, 964–967. doi: 10.1109/EMBC.2012.6346093
Velliste, M., Kennedy, S. D., Schwartz, A. B., Whitford, A. S., Sohn, J.-W., and McMorland, A. J. C. (2014). Motor cortical correlates of arm resting in the context of a reaching task and implications for prosthetic control. J. Neurosci. 34, 6011–6022. doi: 10.1523/JNEUROSCI.3520-13.2014
Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., and Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature 453, 1098–1101. doi: 10.1038/nature06996
Vigneswaran, G., Philipp, R., Lemon, R. N., and Kraskov, A. (2013). M1 corticospinal mirror neurons and their role in movement suppression during action observation. Curr. Biol. 23, 236–243. doi: 10.1016/j.cub.2012.12.006
Vinjamuri, R., Weber, D. J., Mao, Z.-H., Collinger, J. L., Degenhart, A. D., Kelly, J. W., et al. (2011). Toward synergy-based brain-machine interfaces. IEEE Trans. Inf. Technol. Biomed. 15, 726–736. doi: 10.1109/TITB.2011.2160272
Williams, J. J., Rouse, A. G., Thongpang, S., Williams, J. C., and Moran, D. W. (2013). Differentiating closed-loop cortical intention from rest: building an asynchronous electrocorticographic BCI. J. Neural Eng. 10:046001. doi: 10.1088/1741-2560/10/4/046001
Wodlinger, B., Downey, J. E., Tyler-Kabara, E. C., Schwartz, A. B., Boninger, M. L., and Collinger, J. L. (2015). Ten-dimensional anthropomorphic arm control in a human brain-machine interface: difficulties, solutions, and limitations. J. Neural Eng. 12:016011. doi: 10.1088/1741-2560/12/1/016011
Wolpaw, J. R. (2010). Brain-computer interface research comes of age: traditional assumptions meet emerging realities. J. Mot. Behav. 42, 351–353. doi: 10.1080/00222895.2010.526471
Wu, W., Black, M. J., Gao, Y., Bienenstock, E., Serruya, M. D., Shaikhouni, A., et al. (2003). Neural decoding of cursor motion using a Kalman Filter. Adv. Neural Inf. Process. Syst. 133–140.
Xu, K., Wang, Y., Wang, Y., Wang, F., Hao, Y., Zhang, S., et al. (2013). Local-learning-based neuron selection for grasping gesture prediction in motor brain machine interfaces. J. Neural Eng. 10:026008. doi: 10.1088/1741-2560/10/2/026008
Zhang, Y., and Chase, S. M. (2013). A stabilized dual Kalman filter for adaptive tracking of brain-computer interface decoding parameters. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2013, 7100–7103. doi: 10.1109/EMBC.2013.6611194
Keywords: brain-computer interface, hand, kinematic synergy, motor cortex, movement phase, muscle synergy, neuroprosthetics, null space
Citation: Rouse AG and Schieber MH (2015) Advancing brain-machine interfaces: moving beyond linear state space models. Front. Syst. Neurosci. 9:108. doi: 10.3389/fnsys.2015.00108
Received: 11 January 2015; Accepted: 13 July 2015;
Published: 28 July 2015.
Edited by:
Mikhail Lebedev, Duke University, USAReviewed by:
Ioan Opris, Wake Forest University School of Medicine, USAAaron Batista, University of Pittsburgh, USA
Copyright © 2015 Rouse and Schieber. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Marc H. Schieber, Department of Neurology, University of Rochester Medical Center, 601 Elmwood Avenue, Box 673, Rochester, NY 14642, USA, mhs@cvs.rochester.edu