Daniel C. Comstock
Michael J. Hove
Ramesh Balasubramaniam- 1Cognitive and Information Sciences, University of California, Merced, Merced, CA, United States
- 2Department of Psychological Science, Fitchburg State University, Fitchburg, MA, United States
It has long been known that the auditory system is better suited to guide temporally precise behaviors like sensorimotor synchronization (SMS) than the visual system. Although this phenomenon has been studied for many years, the underlying neural and computational mechanisms remain unclear. Growing consensus suggests the existence of multiple, interacting, context-dependent systems, and that reduced precision in visuo-motor timing might be due to the way experimental tasks have been conceived. Indeed, the appropriateness of the stimulus for a given task greatly influences timing performance. In this review, we examine timing differences for sensorimotor synchronization and error correction with auditory and visual sequences, to inspect the underlying neural mechanisms that contribute to modality differences in timing. The disparity between auditory and visual timing likely relates to differences in the processing specialization between auditory and visual modalities (temporal vs. spatial). We propose this difference could offer potential explanation for the differing temporal abilities between modalities. We also offer suggestions as to how these sensory systems interface with motor and timing systems.
Introduction
Many behavioral studies have examined human timing ability in tasks of sensorimotor synchronization (SMS) where subjects synchronize their movements to an external rhythm.
Comparisons between auditory metronomes and visual flashing metronomes reveal that movement synchronization is less variable and can occur at faster rates with auditory metronomes (Chen et al., 2002; Repp, 2003; Repp and Penel, 2004; Lorås et al., 2012). However, visuo-motor synchronization greatly improves when synchronizing with a moving periodic visual metronome (Hove et al., 2010). Adding a changing velocity profile to the moving visual metronome further reduces variability in SMS tapping (Hove et al., 2013a; Iversen et al., 2015), and Gan et al. (2015) suggests that a more realistic velocity profile can bring visual SMS to be as temporally precise as auditory SMS, at moderate but not fast tempi. While most studies of SMS look at finger tapping, others have included synchronized circle drawing, gait, dancing, and eye movements in the context of modality-specific timing effects (e.g., Repp and Su, 2013).
Studies on auditory and visual interference also suggest auditory timing is more prominent. When concurrent auditory metronomes and visual flashing metronomes are presented out-of-phase, the auditory sequences interfere with visuomotor timing, but not vice versa (Chen et al., 2002; Repp and Penel, 2002, 2004). The interference effect is considerably reduced with moving visual metronomes and is tied to training and experience as the auditory dominance is stronger in musicians and weaker in video gamers (Hove et al., 2013a). Similarly, auditory cues can improve visual temporal discrimination (Morein-Zamir et al., 2003; Parise and Spence, 2008). This effect only holds for the temporal domain however, as the visual system dominates when auditory and visual stimuli conflict in the spatial domain; spatial dominance in the visual modality is apparent in the well-known “ventriloquist effect” (Vroomen et al., 2001).
Role of Error Correction in Timing
Error correction is a crucial component of any SMS task. By inducing perturbations and errors in SMS, we can gain insight into the underlying timing mechanisms. A common method to induce errors in a SMS task is to occasionally perturb an otherwise isochronous metronome (Repp, 2000, 2001a,b; Praamstra et al., 2003; Repp and Keller, 2004; Jang et al., 2016; Jantzen et al., 2018). Error correction in SMS can be broken down into two distinct mechanisms: a phase-correction mechanism for correcting errors in relative phase, and a period-correction mechanism that corrects changes to the internal timekeeper period (Repp, 2001b; Repp and Keller, 2004). Period corrections require conscious awareness of the error as it involves a conscious updating of the internal rhythm; while a phase correction can happen even with errors too small for conscious awareness and does not involve updating the central timekeeper period and so is considered a more peripheral process than period correction (Repp, 2001b, 2005). An error corrected under the phase-correction mechanism is typically a gradual adjustment that occurs over several beats, while an error corrected under the period-correction mechanism will be evidenced by a pronounced correction, usually followed by a more gradual phase-correction-like pattern after the initial large correction (Repp, 2001b).
While error correction has been well documented in auditory SMS, relatively little work has investigated error correction in visual SMS. In a recent study comparing error correction for auditory and flashing visual sequences, we observed error corrections for perturbations in the auditory condition that were modulated by the direction of the perturbations, but no such modulation was found for perturbations in the visual condition (Comstock and Balasubramaniam, 2017a). This suggests the visual system may not engage in the same SMS timing mechanisms as the auditory system. Additional evidence for a discrepancy in error correction for auditory and visual sequences can be gleaned from the autocorrelation structure of adjacent taps: unlike auditory SMS, tapping with visual flashes does not produce a negative lag1 autocorrelation that can indicate of the presence of a robust central timekeeping and error-correction mechanism (Hove and Keller, 2010). However, visuomotor synchronization with moving and apparent-motion metronomes do produce a negative lag1 autocorrelation, suggesting that a moving visual metronome may engage error correction (Hove and Keller, 2010; Hove et al., 2010); note that negative lag1 autocorrelation does not necessarily stem from error correction and can arise from other timing factors (e.g., Wing and Kristofferson, 1973). It remains unclear if error correction will occur with perturbations in moving visual metronomes or with larger phase perturbations in a flashing visual metronome.
Underlying Physiology of the Auditory and Visual Timing System
Brain Networks Involved in Timing Activity
Investigating the neural underpinnings in auditory and visual timing is a massive undertaking due to the many different timing subprocesses and tasks, including: SMS, interval timing, rhythm perception, timing recall, time perception, etc. Excellent reviews of the brain mechanisms involved in various timing activities include: a review of neural activity in music production (Zatorre et al., 2007); a review of neural activity involved in time perception (Wiener et al., 2010); and an overview of neural activation in SMS as part of a larger review of SMS (Repp and Su, 2013). This body of work consistently demonstrates that temporal processing across tasks and sensory modalities relies heavily on the motor system. This motor network includes the supplemental motor area (SMA), primary motor cortex, lateral premotor cortex, anterior cingulate, basal ganglia, and cerebellum (Repp and Su, 2013). Auditory rhythm perception activates the motor system and is closely linked to movement (Janata et al., 2012; Iversen and Balasubramaniam, 2016; Ross et al., 2016a,b). The SMA is also strongly implicated in motor timing (Coull et al., 2016; Merchant and Yarrow, 2016), and along with the pre-SMA could be a hub of motor timing (Schwartze et al., 2012). Subcortical regions are especially active during sub-second time perception (Wiener et al., 2010), sub-second interval timing (Repp and Su, 2013), and rhythm timing (Grahn and Rowe, 2009; Wiener et al., 2010; Coull et al., 2011; Teki et al., 2011; Hove et al., 2013b). There is evidence of a dorsal auditory stream connecting the auditory cortex to the motor cortex through the posterior parietal cortex that plays a role in rhythm perception (Patel and Iversen, 2014; Ross et al., 2018). Interestingly this dorsal stream is also implicated in visual and tactile rhythm perception (Araneda et al., 2017; Rauschecker, 2017), adding to the idea of a common timing system tied to the motor system. Further evidence of the common timing system is found in a study of auditory and visual synchronization that dissociated modality and tapping stability –putamen activation was highest when synchronizing to auditory beeps, moderate with a frequency-modulated siren and with a moving visual metronome, and lowest with a flashing visual metronome, closely paralleling behavioral performance (Hove et al., 2013b).
While visual SMS activates many of the same motor regions as auditory SMS (Hove et al., 2013b; Araneda et al., 2017), some activations are specific to the visual system. The visual cortex shows activity related to interval timing that follows the expected scalar property, such that size of timing errors measured in the visual cortex scale in proportion to size of the interval being timed as predicted by Weber's law (Shuler, 2016). Additionally, Zhou et al. (2014) found evidence that visual feature processing in the early visual cortex can contribute to duration perception, furthering the notion that at least some timing information is processed independently within the visual cortex. Additionally, in visual rhythm perception, the visual cortex plays a role predicting rhythmic onsets (Comstock and Balasubramaniam, 2017b, 2018). The additional activations with visual timing tasks, taken together with behavioral results, suggest the timing accuracy in visual processing may be compared to the auditory system due to the additional computational demands of processing the higher complexity of visual spatial information along with temporal information.
Role of Cortical Oscillations in Timing Encoding and Spreading Information Across the Brain
In addition to looking at the networks and regions involved in temporal processing, a growing body of work shows the role of cortical oscillations in encoding timing across multiple frequency bands. Cortical oscillations play a role in connecting regions across the brain, with higher frequencies utilized for localized interaction and lower frequencies for longer range interaction (Sarnthein et al., 1998; Von Stein and Sarnthein, 2000). This pattern of oscillations is used to connect and calibrate disparate timing systems in the brain (Gupta and Chen, 2016). Oscillations relating to timing appear to arise from multiple context-specifc timing systems in the brain (Wiener and Kanai, 2016). The question is then how these functionally and anatomically disparate systems integrate and interact. It appears that oscillations from different timing systems are coordinated within the striatum (Matell and Meck, 2004; Gu et al., 2015).
Beta band activity (~20 Hz) is tied to the motor system and several studies indicate beta's role in predicting timing of auditory rhythms (Fujioka et al., 2009, 2012, 2015). Additionally, beta activity reflects top-down imposition of metrical structure on auditory rhythms (Iversen et al., 2009). Recently, beta activity has also been linked to timing predictions within the visual system in response to visual rhythms (Comstock and Balasubramaniam, 2017b).
With rhythm perception, evidence shows that internal oscillations arise to match the fundamental frequency of the rhythm, and frequency of the meter (Nozaradan et al., 2011), as well as to the frequency of imagined rhythms (Okawa et al., 2017). These findings align with the Neural Resonance Theory that posits neural rhythms synchronize to auditory rhythms, and these neural rhythms can influence attention, expectancy, and motor planning (Large and Snyder, 2009). As of yet, it is unclear if this same neural resonance to meter would arise with visual stimuli.
Neural Underpinnings of Error Correction
The neural correlates of error correction reveal more evidence for multiple interacting and overlapping timing mechanisms. Error detection of timing perturbations in auditory SMS tasks modulates the P1, N1, and N2 auditory ERP components depending on both the size and direction of the perturbation (Praamstra et al., 2003; Jang et al., 2016). Jantzen et al. (2018) also found a theta response stemming from the Pre-SMA and anterior cingulate for error detection, an increase in theta coupling between the SMA and the motor cortex for late perturbations. In visual error detection, the visual P1 component is reduced in latency only for large late perturbations (Comstock and Balasubramaniam, 2017a). Each of these instances show cortical activation specific to a type of perturbation, although these effects are generally limited to larger perturbations.
Smaller perturbations that elicit a phase-correction response are believed to be driven primarily by subcortical mechanisms. Applying repetitive TMS to downregulate motor and premotor cortices produced no effect on phase correction (Doumas et al., 2005), whereas phase-correction was impaired by repetitive TMS to the cerebellum (Bijsterbosch et al., 2011). This fits with the suggestion that phase-correction is primarily subcortical based on evidence from how rapidly the movement trajectory changes after a perturbation (Hove et al., 2014). A possible network that exhibits the rapid timing required for the phase-correction response is a cortico-striatal circuit connecting the cerebellum to the SMA-striatal network via the thalamus (Kotz et al., 2016).
The data on the neural underpinnings of error correction suggest multiple timing systems, each with specific roles, yet able to coordinate for rapid response. Commensurate with this idea is work suggesting the basal ganglia integrates various timing systems through oscillation comparators (Matell and Meck, 2004; Gu et al., 2015). The limited data on visual error correction, however, leave open how well this network can interface with the visual timing systems.
Evidence the Auditory System has Privileged Access to Timing Systems
Considering the auditory system's timing advantage along with the prominence of the motor system in timing processing, we suggest that the auditory system's advantage in timing stems from its stronger coupling to the motor system. Auditory timing compared to visual timing tasks often yield more activation in motor structures, such as the SMA and premotor cortex (Jäncke et al., 2000). Even when visual SMS tasks employed the modality-appropriate moving visual metronomes, audio-motor synchronization with auditory beeps yielded greater activation in the putamen (Hove et al., 2013b). Likewise, priming a visual rhythm with a similar auditory rhythm resulted in increased putamen activation compared to a visual rhythm alone, while a visual rhythm yielded no priming effect on an auditory rhythm (Grahn et al., 2011). The finding that the increased visual synchronization ability provided by a bouncing ball does not transfer to purely perceptual rhythm perception provides further evidence of the role of motor coupling in timing tasks (Silva and Castro, 2016). Additionally, the privileged link between auditory and motor systems can be seen in Parkinson's disease, a disorder that impairs movement due to cell loss within the basal ganglia (Davie, 2008). For example, Parkinsonian gait can improve when cued by an external rhythm, and these interventions are more effective when synchronizing with auditory metronomes than with flashing visual metronomes (Rochester et al., 2005; Arias and Cudeiro, 2008).
Visual timing activities recruit timing centers within the visual system that, based on behavioral results, are less precise compared to the auditory timing system. In Jäncke et al. (2000), visual timing tasks resulted in increased activity in the right superior cerebellum, vermis, and right inferior parietal lobe compared to auditory timing tasks. Visual timing tasks also recruit areas MT, V5, and the superior parietal lobe, tying into the dorsal visual stream (Jantzen et al., 2005), and visual rhythm perception induces increased beta activity at event onsets arising from the visual cortex (Comstock and Balasubramaniam, 2017b). It is unclear if these timing activations in the visual system are the result of compensating for a weaker connection to the motor timing system. It may be that the temporal processing in the visual system is additional processing of visual information required to interface with the motor system.
While differences in coupling strength to the motor system are crucial for modality timing differences, other factors are likely. To that end, it is clear that the visual system is able to pick out high speed temporal information, for example, V1 will phase lock its input/output to up to a 100 Hz visual flashing stimuli (Williams et al., 2004). This suggest that entrainment is not easily transferred to the systems involved in time/rhythm perception, especially at the time frame usually involved in rhythm perception, indicating that the issue may be one of translation. A likely place for that translation would be within the dorsal pathway, which has been found to have neurons with high temporal resolution in macaques, with higher temporal resolution in the auditory dorsal stream (Rauschecker, 2017). If there is a higher temporal resolution of the auditory dorsal stream than in the visual dorsal stream, then it may give explanation as to why the visual system cannot synchronize at the higher frequencies achieved by the auditory system. Of course, it cannot be ruled out that the difference in temporal resolution is due to different levels of timing precision available to the dorsal stream. Reduced timing precision in the visual stream may be caused by increased necessary processing due to richer sensory input of the visual system compared to the auditory system. Indeed, greater processing requirements and longer processing time may help to account for the inability of the visual system to allow for synchronization at the higher tempos allowed by the auditory system.
Role of the Vestibular-Tactile-Somatosensory System
Another link between auditory and motor systems is that auditory rhythm perception may be tied to the vestibular-tactile-somatosensory (VTS) system, which is important for movement and dance, and therefore closely tied to the motor system and attuned to timing (Todd and Lee, 2015). In addition to its ties for movement, the VTS system is clearly tied to the auditory system with regards to rhythm perception (Phillips-Silver and Trainor, 2005, 2007, 2008; Trainor et al., 2009), and through common neural activation (Araneda et al., 2017). These ties between the auditory and VTS system may be an additional factor in the dominance of the auditory system in the temporal domain.
Since VTS rhythms are ubiquitous in fetal life through the mother's gait, heart rate, breathing, etc., and since these networks are tied into auditory rhythm systems, it is likely that the VTS system is heavily tied into the timing systems used in auditory rhythm perception and in motor rhythm production (Provasi et al., 2014). This is further strengthened by the fact that movement and rhythms are linked and proprioception (part of the VTS system) plays a large role in perception of rhythms that is tied into auditory rhythm perception and production (Trainor et al., 2009). Interactions between the VTS system with visual rhythm perception remains mostly unexplored at this point however, so it is unclear how much this system plays a supramodal role in the timing involved in rhythm perception/production, or if it is only tied to the auditory and motor rhythm timing systems. Further research in this area is needed to answer these questions.
Evolutionary Origins of Sensorimotor Synchronization
In an evolutionary context, it makes sense that auditory and motor systems would be tightly interconnected. First, rhythms in language are critical for both perception and production and may be a driver of SMS ability (Patel, 2006). Beyond language, matching movement to sound is a necessary result of human evolution that allows for the social and cultural inclination of humanity via music (Hagen and Bryant, 2003; Brown and Jordania, 2013). Dance is also tightly connected with music and culture and can provide a further explanatory account of human SMS capability and the connection between the motor and auditory systems (Fitch, 2016; Iversen, 2016; Laland et al., 2016; Ravignani and Cook, 2016).
Beyond humans, common adaptations appear to increase SMS ability in several non-human species capable of some level of audio-motor entrainment such as parrots (Patel et al., 2009), bonobos (Large and Gray, 2015), and sea-lions (Cook et al., 2013). Although some animals can exhibit rhythmic capabilities, some remarkably well like Ronan the sea-lion (Rouse et al., 2016), they are in some ways limited compared to humans (Patel and Iversen, 2014; Merker et al., 2015). Even though there are animals that can entrain to auditory rhythms, only humans appear to be naturally inclined to do so (Wilson and Cook, 2016). Finally, there is some evidence that non-human primates are able to synchronize their movements to predictable visual stimuli (Takeya et al., 2017), yet there has been much less research on visual SMS compared to auditory SMS in non-humans.
General Synthesis and Future Directions
In looking at how the brain processes timing information, it is clear that many context sensitive mechanisms interact and coordinate to provide optimal timing output. Much of this interaction appears to happen within the motor system and likely involves the subcortical systems to coordinate the various mechanisms. Current research suggests that oscillations play a key role coordinating the interactions among various timing circuits. However, it is not clear if the various timing systems compute measures of time in the same way. When considering that auditory and visual systems take in very different kinds of information and use it in different ways, i.e., auditory has a stronger temporal precision, and visual has a strong spatial bias, it seems likely that the timing mechanisms themselves may greatly differ.
Consider the difference between extracting timing information between a moving visual rhythm and an auditory rhythm. Moving visual stimuli contain more information than auditory stimuli, such that while entraining to auditory stimuli, prediction of the onset of the next event involves encoding the interval between two events and utilizing that information to predict the onset of the next event. With a moving visual rhythmic stimulus, that interval information is present, but so is information on position/velocity/acceleration. This means predictions of the onset of the next event can be made as part of a continuous process. The fact that even with this information, visual SMS is at best equal to auditory SMS except at fast speeds, begs the question as to why visual SMS is less capable. One possible explanation for this is that the visual system has to encode much more information, and further, encoding that information into a form that is usable by the motor network may require extra processing. This may explain the timing activity found within the visual cortex during visual SMS. Even when there is a simple flashing metronome, there is a measure of timing activity originating from the visual cortex. Considering the reduced temporal ability with visual flashing metronomes, it suggests there may be a translation issue in harnessing a system not optimized to temporal processing the way the auditory system has been, resulting in a weaker connection to the motor timing network.
Different timing systems likely employ varying mechanisms and computational principles that are appropriate to the time scale, cellular properties, and general needs of the system. Existing computational models that capture a range of these phenomena across levels include: pacemaker accumulator models, multiple oscillator models, memory trace models, random process models, ramping activity models, delay line models, and state space trajectory-based models (Addyman et al., 2016; Hass and Durstewitz, 2016). Such models help illustrate the variety of ways to process timing information within a neural network. Evidence also suggests that cells with specific timing mechanisms exist in the basal ganglia and cerebellum (Lusk et al., 2016), yet other areas with multiple functional properties also process timing, such as in the prefrontal cortex (Hyman et al., 2012) and hippocampus (MacDonald et al., 2011). The areas that have multiple functions, as in the hippocampus and prefrontal cortex, will then likely have different computational approach than more specialized timing structures.
Given that there are multiple ways to process timing, and that many forms of cognition require some form of temporal processing, it would be surprising to find that timing mechanisms are not ubiquitous in the brain. This raises an important question. If many different timing mechanisms are available for a given task, and only one output (through action), how do neural systems arrive at the best timing information to use? A strong candidate explanation for this would implicate a mechanism that helps integration through an optimal Bayesian process (Hass and Durstewitz, 2016). Evidence from multimodal sensory integration suggests that when timing information is presented from multiple modalities, the modalities are combined and weighted based on reliability in Bayesian optimal solution (Ernst and Banks, 2002). Since most timing related activity requires motor output, we would expect that the source of timing to be utilized would be determined before, or as that timing information becomes available to the motor system. This seems to make the case that the striatal cells operating as a comparator may be the seat of the Bayesian process to determine the optimal timing source for motor timing.
Since there is some disparity in the amount of work on auditory and visual SMS error correction, there is a need to further study the error correction capabilities within visual SMS. It is currently unknown if visual error correction can be as fast as auditory error correction when dealing modality appropriate stimuli, such as a moving visual sequence or bouncing ball. Another major area of needed work is in understanding the mechanisms by which the Bayesian optimal timing source is chosen in cases where multiple sources are available. If timing mechanisms are as ubiquitous in the brain as evidence suggests, then there may be a variety of ways these mechanisms interface with the motor timing system to produce a single output. Further imaging and computational work is required to understanding this mechanism.
Author Contributions
All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.
Funding
This work was partially supported by a grant from the National Science Foundation BCS-1460633.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
Addyman, C., French, R. M., and Thomas, E. (2016). Computational models of interval timing. Curr. Opin. Behav. Sci. 8, 140–146. doi: 10.1016/j.cobeha.2016.01.004
Araneda, R., Renier, L., Ebner-Karestinos, D., Dricot, L., and De Volder, A. G. (2017). Hearing, feeling or seeing a beat recruits a supramodal network in the auditory dorsal stream. Eur. J. Neurosci. 45, 1439–1450. doi: 10.1111/ejn.13349
Arias, P., and Cudeiro, J. (2008). Effects of rhythmic sensory stimulation (auditory, visual) on gait in Parkinson's disease patients. Exp. Brain Res. 186, 589–601. doi: 10.1007/s00221-007-1263-y
Bijsterbosch, J. D., Lee, K. H., Hunter, M. D., Tsoi, D. T., Lankappa, S., Wilkinson, I. D., et al. (2011). The role of the cerebellum in sub-and supraliminal error correction during sensorimotor synchronization: evidence from fMRI and TMS. J. Cogn. Neurosci. 23, 1100–1112. doi: 10.1162/jocn.2010.21506
Brown, S., and Jordania, J. (2013). Universals in the world's musics. Psychol. Music 41, 229–248. doi: 10.1177/0305735611425896
Chen, Y., Repp, B. H., and Patel, A. D. (2002). Spectral decomposition of variability in synchronization and continuation tapping: comparisons between auditory and visual pacing and feedback conditions. Hum. Mov. Sci. 21, 515–532. doi: 10.1016/S0167-9457(02)00138-0
Comstock, D., and Balasubramaniam, R. (2017a). “Differences in neural correlates of error correction in auditory and visual sensorimotor synchronization,” in Poster presented at the 24th Annual Cognitive Neuroscience Society Meeting (San Francisco, CA).
Comstock, D., and Balasubramaniam, R. (2017b). “Beta-band response synchronizes and predicts rhythmic flashing visual stimuli,” in Poster presented at the 47th Annual Society for Neuroscience Meeting (Washington, DC).
Comstock, D. C., and Balasubramaniam, R. (2018). Neural responses to perturbations in visual and auditory metronomes during sensorimotor synchronization. Neuropsychologia 117, 55–66. doi: 10.1016/j.neuropsychologia.2018.05.013
Cook, P., Rouse, A., Wilson, M., and Reichmuth, C. (2013). A California sea lion (Zalophus californianus) can keep the beat: motor entrainment to rhythmic auditory stimuli in a non vocal mimic. J. Comp. Psychol. 127:412. doi: 10.1037/a0032345
Coull, J. T., Cheng, R. K., and Meck, W. H. (2011). Neuroanatomical and neurochemical substrates of timing. Neuropsychopharmacology 36, 3–25. doi: 10.1038/npp.2010.113
Coull, J. T., Vidal, F., and Burle, B. (2016). When to act, or not to act: that's the SMA's question. Curr. Opin. Behav. Sci. 8, 14–21. doi: 10.1016/j.cobeha.2016.01.003
Davie, C. A. (2008). A review of Parkinson's disease. Br. Med. Bull. 86, 109–127. doi: 10.1093/bmb/ldn013
Doumas, M., Praamstra, P., and Wing, A. M. (2005). Low frequency rTMS effects on sensorimotor synchronization. Exp. Brain Res. 167, 238–245. doi: 10.1007/s00221-005-0029-7
Ernst, M. O., and Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429. doi: 10.1038/415429a
Fitch, W. (2016). Dance, music, meter and groove: a forgotten partnership. Front. Hum. Neurosci. 10:64. doi: 10.3389/fnhum.2016.00064
Fujioka, T., Ross, B., and Trainor, L. J. (2015). Beta-band oscillations represent auditory beat and its metrical hierarchy in perception and imagery. J. Neurosci. 35, 15187–15198. doi: 10.1523/JNEUROSCI.2397-15.2015
Fujioka, T., Trainor, L. J., Large, E. W., and Ross, B. (2009). Beta and gamma rhythms in human auditory cortex during musical beat processing. Ann. N.Y. Acad. Sci. 1169, 89–92. doi: 10.1111/j.1749-6632.2009.04779.x
Fujioka, T., Trainor, L. J., Large, E. W., and Ross, B. (2012). Internalized timing of isochronous sounds is represented in neuromagnetic beta oscillations. J. Neurosci. 32, 1791–1802. doi: 10.1523/JNEUROSCI.4107-11.2012
Gan, L., Huang, Y., Zhou, L., Qian, C., and Wu, X. (2015). Synchronization to a bouncing ball with a realistic motion trajectory. Sci. Rep. 5:11974. doi: 10.1038/srep11974
Grahn, J. A., Henry, M. J., and McAuley, J. D. (2011). FMRI investigation of cross-modal interactions in beat perception: audition primes vision, but not vice versa. Neuroimage 54, 1231–1243. doi: 10.1016/j.neuroimage.2010.09.033
Grahn, J. A., and Rowe, J. B. (2009). Feeling the beat: premotor and striatal interactions in musicians and nonmusicians during beat perception. J. Neurosci. 29, 7540–7548. doi: 10.1523/JNEUROSCI.2018-08.2009
Gu, B. M., van Rijn, H., and Meck, W. H. (2015). Oscillatory multiplexing of neural population codes for interval timing and working memory. Neurosci. Biobehav. Rev. 48, 160–185. doi: 10.1016/j.neubiorev.2014.10.008
Gupta, D. S., and Chen, L. (2016). Brain oscillations in perception, timing and action. Curr. Opin. Behav. Sci. 8, 161–166. doi: 10.1016/j.cobeha.2016.02.021
Hagen, E. H., and Bryant, G. A. (2003). Music and dance as a coalition signaling system. Hum. Nat. 14, 21–51. doi: 10.1007/s12110-003-1015-z
Hass, J., and Durstewitz, D. (2016). Time at the center, or time at the side? Assessing current models of time perception. Curr. Opin. Behav. Sci. 8, 238–244. doi: 10.1016/j.cobeha.2016.02.030
Hove, M. J., Balasubramaniam, R., and Keller, P. E. (2014). The time course of phase correction: a kinematic investigation of motor adjustment to timing perturbations during sensorimotor synchronization. J. Exp. Psychol. Hum. Percept. Perform. 40:2243. doi: 10.1037/a0037826
Hove, M. J., Fairhurst, M. T., Kotz, S. A., and Keller, P. E. (2013b). Synchronizing with auditory and visual rhythms: an fMRI assessment of modality differences and modality appropriateness. Neuroimage 67, 313–321. doi: 10.1016/j.neuroimage.2012.11.032
Hove, M. J., Iversen, J. R., Zhang, A., and Repp, B. H. (2013a). Synchronization with competing visual and auditory rhythms: bouncing ball meets metronome. Psychol. Res. 77, 388–398. doi: 10.1007/s00426-012-0441-0
Hove, M. J., and Keller, P. E. (2010). Spatiotemporal relations and movement trajectories in visuomotor synchronization. Music Percept. Interdisc. J. 28, 15–26. doi: 10.1525/mp.2010.28.1.15
Hove, M. J., Spivey, M. J., and Krumhansl, C. L. (2010). Compatibility of motion facilitates visuomotor synchronization. J. Exper. Psychol. Hum. Percept. Perform. 36:1525. doi: 10.1037/a0019059
Hyman, J. M., Ma, L., Balaguer-Ballester, E., Durstewitz, D., and Seamans, J. K. (2012). Contextual encoding by ensembles of medial prefrontal cortex neurons. Proc. Natl. Acad. Sci.U.S.A. 109, 5086–5091. doi: 10.1073/pnas.1114415109
Iversen, J. R. (2016). “21 In the beginning was the beat: evolutionary origins of musical rhythm in humans,” in The Cambridge Companion to Percussion, eds R. Hartenberger (Cambridge, MA: Cambridge University Press), 281–295.
Iversen, J. R., and Balasubramaniam, R. (2016). Synchronization and temporal processing. Curr. Opin. Behav. Sci. 8, 175–180. doi: 10.1016/j.cobeha.2016.02.027
Iversen, J. R., Patel, A. D., Nicodemus, B., and Emmorey, K. (2015). Synchronization to auditory and visual rhythms in hearing and deaf individuals. Cognition 134, 232–244. doi: 10.1016/j.cognition.2014.10.018
Iversen, J. R., Repp, B. H., and Patel, A. D. (2009). Top-down control of rhythm perception modulates early auditory responses. Ann. N.Y. Acad. Sci. 1169, 58– doi: 10.1111/j.1749-6632.2009.04579.x
Janata, P., Tomic, S. T., and Haberman, J. M. (2012). Sensorimotor coupling in music and the psychology of the groove. J. Exp. Psychol. Gen. 141:54. doi: 10.1037/a0024208
Jäncke, L., Loose, R., Lutz, K., Specht, K., and Shah, N. J. (2000). Cortical activations during paced finger-tapping applying visual and auditory pacing stimuli. Cogn. Brain Res. 10, 51–66. doi: 10.1016/S0926-6410(00)00022-7
Jang, J., Jones, M., Milne, E., Wilson, D., and Lee, K. H. (2016). Contingent negative variation (CNV) associated with sensorimotor timing error correction. Neuroimage 127, 58–66. doi: 10.1016/j.neuroimage.2015.11.071
Jantzen, K. J., Ratcliff, B. R., and Jantzen, M. G. (2018). Cortical Networks for correcting errors in sensorimotor synchronization depend on the direction of asynchrony. J. Mot. Behav. 50, 235–248. doi: 10.1080/00222895.2017.1327414
Jantzen, K. J., Steinberg, F. L., and Kelso, J. A. (2005). Functional MRI reveals the existence of modality and coordination-dependent timing networks. Neuroimage 25, 1031–1042. doi: 10.1016/j.neuroimage.2004.12.029
Kotz, S. A., Brown, R. M., and Schwartze, M. (2016). Cortico-striatal circuits and the timing of action and perception. Curr. Opin. Behav. Sci. 8, 42–45. doi: 10.1016/j.cobeha.2016.01.010
Laland, K., Wilkins, C., and Clayton, N. (2016). The evolution of dance. Curr. Biol. 26, R5–R9. doi: 10.1016/j.cub.2015.11.031
Large, E. W., and Gray, P. M. (2015). Spontaneous tempo and rhythmic entrainment in a bonobo (Pan paniscus). J. Comp. Psychol. 129:317. doi: 10.1037/com0000011
Large, E. W., and Snyder, J. S. (2009). Pulse and meter as neural resonance. Ann. N.Y. Acad. Sci. 1169, 46–57. doi: 10.1111/j.1749-6632.2009.04550.x
Lorås, H., Sigmundsson, H., Talcott, J. B., Öhberg, F., and Stensdotter, A. K. (2012). Timing continuous or discontinuous movements across effectors specified by different pacing modalities and intervals. Exp. Brain Res. 220, 335–347. doi: 10.1007/s00221-012-3142-4
Lusk, N. A., Petter, E. A., MacDonald, C. J., and Meck, W. H. (2016). Cerebellar, hippocampal, and striatal time cells. Curr. Opin. Behav. Sci. 8, 186–192. doi: 10.1016/j.cobeha.2016.02.020
MacDonald, C. J., Lepage, K. Q., Eden, U. T., and Eichenbaum, H. (2011). Hippocampal “time cells” bridge the gap in memory for discontiguous events. Neuron 71, 737–749. doi: 10.1016/j.neuron.2011.07.012
Matell, M. S., and Meck, W. H. (2004). Cortico-striatal circuits and interval timing: coincidence detection of oscillatory processes. Cogn. Brain Res. 21, 139–170. doi: 10.1016/j.cogbrainres.2004.06.012
Merchant, H., and Yarrow, K. (2016). How the motor system both encodes and influences our sense of time. Curr. Opin. Behav. Sci. 8, 22–27. doi: 10.1016/j.cobeha.2016.01.006
Merker, B., Morley, I., and Zuidema, W. (2015). Five fundamental constraints on theories of the origins of music. Phil. Trans. R. Soc. B, 370:20140095. doi: 10.1098/rstb.2014.0095
Morein-Zamir, S., Soto-Faraco, S., and Kingstone, A. (2003). Auditory capture of vision: examining temporal ventriloquism. Cogn. Brain Res. 17, 154–163. doi: 10.1016/S0926-6410(03)00089-2
Nozaradan, S., Peretz, I., Missal, M., and Mouraux, A. (2011). Tagging the neuronal entrainment to beat and meter. J. Neurosci. 31, 10234–10240. doi: 10.1523/JNEUROSCI.0411-11.2011
Okawa, H., Suefusa, K., and Tanaka, T. (2017). Neural Entrainment to Auditory Imagery of Rhythms. Front. Hum. Neurosci. 11:493. doi: 10.3389/fnhum.2017.00493
Parise, C., and Spence, C. (2008). Synesthetic congruency modulates the temporal ventriloquism effect. Neurosci. Lett. 442, 257–261. doi: 10.1016/j.neulet.2008.07.010
Patel, A. D. (2006). Musical rhythm, linguistic rhythm, and human evolution. Music Percept. Interdisc. J. 24, 99–104. doi: 10.1525/mp.2006.24.1.99
Patel, A. D., and Iversen, J. R. (2014). The evolutionary neuroscience of musical beat perception: the Action Simulation for Auditory Prediction (ASAP) hypothesis. Front. Sys. Neurosci. 8:57. doi: 10.3389/fnsys.2014.00057
Patel, A. D., Iversen, J. R., Bregman, M. R., and Schulz, I. (2009). Experimental evidence for synchronization to a musical beat in a nonhuman animal. Curr. Biol. 19, 827–830. doi: 10.1016/j.cub.2009.03.038
Phillips-Silver, J., and Trainor, L. J. (2005). Feeling the beat: movement influences infant rhythm perception. Science 308, 1430–1430. doi: 10.1126/science.1110922
Phillips-Silver, J., and Trainor, L. J. (2007). Hearing what the body feels: auditory encoding of rhythmic movement. Cognition 105, 533–546. doi: 10.1016/j.cognition.2006.11.006
Phillips-Silver, J., and Trainor, L. J. (2008). Vestibular influence on auditory metrical interpretation. Brain Cogn. 67, 94–102. doi: 10.1016/j.bandc.2007.11.007
Praamstra, P., Turgeon, M., Hesse, C. W., Wing, A. M., and Perryer, L. (2003). Neurophysiological correlates of error correction in sensorimotor-synchronization. Neuroimage 20, 1283–1297. doi: 10.1016/S1053-8119(03)00351-3
Provasi, J., Anderson, D. I., and Barbu-Roth, M. (2014). Rhythm perception, production, and synchronization during the perinatal period. Front. Psychol. 5:1048. doi: 10.3389/fpsyg.2014.01048
Rauschecker, J. P. (2017). Where, When, and How: are they all Sensorimotor? Towards a unified view of the dorsal pathway in vision and audition. Cortex 98, 262–268. doi: 10.1016/j.cortex.2017.10.020
Ravignani, A., and Cook, P. F. (2016). The evolutionary biology of dance without frills. Curr. Biol. 26, R878–R879. doi: 10.1016/j.cub.2016.07.076
Repp, B. H. (2000). Compensation for subliminal timing perturbations in perceptual-motor synchronization. Psychol. Res. 63, 106–128. doi: 10.1007/PL00008170
Repp, B. H. (2005). Sensorimotor synchronization: a review of the tapping literature. Psychon. Bull. Rev. 12, 969–992. doi: 10.3758/BF03206433
Repp, B. H. (2001a). Phase correction, phase resetting, and phase shifts after subliminal timing perturbations in sensorimotor synchronization. J. Exp. Psychol. Hum. Percept. Perform. 27:600. doi: 10.1037/0096-1523.27.3.600
Repp, B. H. (2001b). Processes underlying adaptation to tempo changes in sensorimotor synchronization. Hum. Mov. Sci. 20, 277–312. doi: 10.1016/S0167-9457(01)00049-5
Repp, B. H. (2003). Rate limits in sensorimotor synchronization with auditory and visual sequences: the synchronization threshold and the benefits and costs of interval subdivision. J. Motor Behav. 35, 355–370. doi: 10.1080/00222890309603156
Repp, B. H., and Keller, P. E. (2004). Adaptation to tempo changes in sensorimotor synchronization: effects of intention, attention, and awareness. Q. J. Exp. Psychol A, 57, 499–521. doi: 10.1080/02724980343000369
Repp, B. H., and Penel, A. (2002). Auditory dominance in temporal processing: New evidence from synchronization with simultaneous visual and auditory sequences. J. Exp. Psychology. Hum. Percept. Perform. 28, 1085–1099.
Repp, B. H., and Penel, A. (2004). Rhythmic movement is attracted more strongly to auditory than to visual rhythms. Psychol. Res. 68, 252–270. doi: 10.1007/s00426-003-0143-8
Repp, B. H., and Su, Y. H. (2013). Sensorimotor synchronization: a review of recent research (2006–2012). Psychon. Bull. Rev. 20, 403–452. doi: 10.3758/s13423-012-0371-2
Rochester, L., Hetherington, V., Jones, D., Nieuwboer, A., Willems, A. M., Kwakkel, G., et al. (2005). The effect of external rhythmic cues (auditory and visual) on walking during a functional task in homes of people with Parkinson's disease. Arch. Phys. Med. Rehabilit. 86, 999–1006. doi: 10.1016/j.apmr.2004.10.040
Ross, J. M., Iversen, J. R., and Balasubramaniam, R. (2016a). Motor simulation theories of musical beat perception. Neurocase 22, 558–565. doi: 10.1080/13554794.2016.1242756
Ross, J. M., Iversen, J. R., and Balasubramaniam, R. (2018). The role of posterior parietal cortex in beat-based timing perception: a continuous theta burst stimulation study. J. Cogn. Neurosci. 30, 634–643. doi: 10.1162/jocn_a_01237
Ross, J. M., Warlaumont, A. S., Abney, D. H., Rigoli, L. M., and Balasubramaniam, R. (2016b). Influence of musical groove on postural sway. J. Exp. Psychol. Hum. Percept. Perform. 42:308. doi: 10.1037/xhp0000198
Rouse, A. A., Cook, P. F., Large, E. W., and Reichmuth, C. (2016). Beat keeping in a sea lion as coupled oscillation: implications for comparative understanding of human rhythm. Front. Neurosci. 10:257. doi: 10.3389/fnins.2016.00257
Sarnthein, J., Petsche, H., Rappelsberger, P., Shaw, G. L., and von Stein, A. (1998). Synchronization between prefrontal and posterior association cortex during human working memory. Proc. Natl. Acad. Sci. U.S.A. 95, 7092–7096. doi: 10.1073/pnas.95.12.7092
Schwartze, M., Rothermich, K., and Kotz, S. A. (2012). Functional dissociation of pre-SMA and SMA-proper in temporal processing. Neuroimage 60, 290–298. doi: 10.1016/j.neuroimage.2011.11.089
Shuler, M. G. (2016). Timing in the visual cortex and its investigation. Curr. Opin. Behav. Sci. 8, 73–77. doi: 10.1016/j.cobeha.2016.02.006
Silva, S., and Castro, S. L. (2016). Moving stimuli facilitate synchronization but not temporal perception. Front. Psychol. 7:1798. doi: 10.3389/fpsyg.2016.01798
Takeya, R., Kameda, M., Patel, A. D., and Tanaka, M. (2017). Predictive and tempo-flexible synchronization to a visual metronome in monkeys. Sci. Reports 7:6127. doi: 10.1038/s41598-017-06417-3
Teki, S., Grube, M., Kumar, S., and Griffiths, T. D. (2011). Distinct neural substrates of duration-based and beat-based auditory timing. J. Neurosci. 31, 3805–3812. doi: 10.1523/JNEUROSCI.5561-10.2011
Todd, N. P., and Lee, C. S. (2015). The sensory-motor theory of rhythm and beat induction 20 years on: a new synthesis and future perspectives. Front. Hum. Neurosci. 9:444. doi: 10.3389/fnhum.2015.00444
Trainor, L. J., Gao, X., Lei, J. J., Lehtovaara, K., and Harris, L. R. (2009). The primal role of the vestibular system in determining musical rhythm. Cortex 45, 35–43. doi: 10.1016/j.cortex.2007.10.014
von Stein, A., and Sarnthein, J. (2000). Different frequencies for different scales of cortical integration: from local gamma to long range alpha/theta synchronization. Int. J. Psychophysiol. 38, 301–313. doi: 10.1016/S0167-8760(00)00172-0
Vroomen, J., Bertelson, P., and de Gelder, B. (2001). Directing spatial attention towards the illusory location of a ventriloquized sound. Acta Psychol. 108, 21–33. doi: 10.1016/S0001-6918(00)00068-8
Wiener, M., and Kanai, R. (2016). Frequency tuning for temporal perception and prediction. Curr. Opin. Behav. Sci. 8, 1–6. doi: 10.1016/j.cobeha.2016.01.001
Wiener, M., Turkeltaub, P., and Coslett, H. B. (2010). The image of time: a voxel-wise meta-analysis. Neuroimage 49, 1728–1740. doi: 10.1016/j.neuroimage.2009.09.064
Williams, P. E., Mechler, F., Gordon, J., Shapley, R., and Hawken, M. J. (2004). Entrainment to video displays in primary visual cortex of macaque and humans. J. Neurosci. 24, 8278–8288. doi: 10.1523/JNEUROSCI.2716-04.2004
Wilson, M., and Cook, P. F. (2016). Rhythmic entrainment: why humans want to, fireflies can't help it, pet birds try, and sea lions have to be bribed. Psychon. Bull. Rev. 23, 1647–1659. doi: 10.3758/s13423-016-1013-x
Wing, A. M., and Kristofferson, A. (1973). The timing of interresponse intervals. Percept. Psychophys. 13, 455–460. doi: 10.3758/bf03205802
Zatorre, R. J., Chen, J. L., and Penhune, V. B. (2007). When the brain plays music: auditory–motor interactions in music perception and production. Nat. Rev. Neurosci. 8:547. doi: 10.1038/nrn2152
Keywords: sensorimotor synchronization, timing, rhythm, visual perception, auditory perception
Citation: Comstock DC, Hove MJ and Balasubramaniam R (2018) Sensorimotor Synchronization With Auditory and Visual Modalities: Behavioral and Neural Differences. Front. Comput. Neurosci. 12:53. doi: 10.3389/fncom.2018.00053
Received: 05 March 2018; Accepted: 19 June 2018;
Published: 18 July 2018.
Edited by:
Daya Shankar Gupta, Camden County College, United StatesReviewed by:
Edward W. Large, University of Connecticut, United StatesDavid Ian Anderson, San Francisco State University, United States
Todd Troyer, University of Texas at San Antonio, United States
Copyright © 2018 Comstock, Hove and Balasubramaniam. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Ramesh Balasubramaniam, cmFtZXNoQHVjbWVyY2VkLmVkdQ==