Abstract
Recently, researchers have measured hand movements en route to choices on a screen to understand the dynamics of a broad range of psychological processes. We review this growing body of research and explain how manual action exposes the real-time unfolding of underlying cognitive processing. We describe how simple hand motions may be used to continuously index participants’ tentative commitments to different choice alternatives during the evolution of a behavioral response. As such, hand-tracking can provide unusually high-fidelity, real-time motor traces of the mind. These motor traces cast novel theoretical and empirical light onto a wide range of phenomena and serve as a potential bridge between far-reaching areas of psychological science – from language, to high-level cognition and learning, to social cognitive processes.
Traditional theories once viewed the mind's cognitive and motor systems as functionally independent. A motor movement, therefore, was thought to be the uninteresting end-result of cognitive processing. It is becoming increasingly clear, however, that the dynamics of action do not simply reside in the aftermath of cognition. Rather, they are part and parcel with cognition, and the cognitive and motor systems are far more coextensive than previously imagined. The implication, then, is that the rich dynamics of a simple bodily movement, such as a hand movement, can leave powerful traces of internal cognitive processes.
The existence of such motor traces of the mind would represent an important advance because researchers have too often been forced to view the “black box” of cognition through off-line, indirect observation. By and large, we use inferences from stimulus characteristics and discrete behavioral outcomes, such as reaction times and errors, to generate rich accounts of mental phenomena. Although quite valuable, a major drawback to this approach is that behavioral outcomes provide relatively little information about how cognitive processing evolves over time and how multiple processes may temporally converge to drive ultimate responses. This presents a serious quandary because competing theoretical accounts in many domains make fine-grained predictions about what ought to be taking place across the time-course of mental activity – from perception, to language and high-level cognition, to social cognitive processes.
Movements of the hand, however, offer continuous streams of output that can reveal ongoing dynamics of processing, potentially capturing the mind in motion with fine-grained temporal sensitivity. Tracking eye movements has long been used for this purpose (e.g., Tanenhaus et al., ), but eye-tracking and hand-tracking each have their advantages (see Magnuson, ). For example, hand movements offer continuous streams of motor output whereas eye movements are typically comprised of discrete saccades. Here, we explain how continuous hand movements provide a powerful – yet cheap and readily obtainable – index of underlying cognitive processing as it unfolds across time. Unlike eye movements, hand movements are also able to occupy graded locations amidst multiple response locations at the same time. These fortuitous qualities of hand movements led Magnuson (, p. 9996), in a commentary on the seminal article by Spivey et al. (), to portend that such movements could be able to “address…some of the biggest questions facing cognitive science.”
The Link between Manual and Mental Dynamics
Classically, motor responses were seen as the end-result of a one-way route from perception → cognition → action; thus, they were thought to reveal little about cognitive processing (see Rosenbaum, , for discussion). However, substantial behavioral evidence now indicates that cognition does not discretely collapse its processing onto movement execution. Instead, movement is continually updated by cognitive processing over time (Goodale et al., ). This is buttressed by neurophysiological evidence. Studies in humans, for example, suggest that the process of categorizing a visual stimulus immediately shares its ongoing results with the motor cortex to continuously guide a hand-movement response over time (e.g., Freeman et al., ). Further, human reaching movements suggest that multiple motor plans are prepared in parallel, and that these cascade over time into visually guided action (Song and Nakayama, , ). Monkey studies show that the hand's position and velocity are tightly yoked to ongoing changes in the firing of population codes within motor cortex (Paninski et al., ), and when a monkey must generate a hand movement based on a perceptual decision, these motor-cortical population codes are yoked to the evolution of the decision process (Cisek and Kalaska, ). These findings suggest that manual dynamics are intimately coextensive with mental dynamics, and therefore, hand movements that are sampled fast enough could potentially provide real-time read-outs of internal cognitive processing.
The recent tracking of hand movement en route to choices on a screen has opened up new avenues of investigation into a wide range of psychological phenomena. Hand trajectories recorded by a computer mouse, or sometimes a wireless remote or position sensor, have proven able to sensitively track participants’ tentative commitments to various choice alternatives over hundreds of milliseconds. As such, the emergence of a behavioral response can be sampled anywhere from 40 to 120 times per second. This permits insights into fine-grained temporal dynamics, providing unusually high-fidelity, real-time motor traces of the mind. Its sensitivity to these dynamics can reveal “hidden” cognitive states that are otherwise not availed by traditional measures (Song and Nakayama, , , ). More broadly, these real-time dynamics may be used to assess how multiple processes temporally interact and how multiple information sources weigh in on processing across hundreds of milliseconds, thereby resolving critical empirical questions and theoretical impasses across psychology (e.g., see Spivey, ).
Doing Psychology by Hand
Language
In language research, debate continues as to whether the language system is a discrete-symbolic architecture or whether it integrates multiple sources of information in parallel to continually constrain its operation. Considerable evidence for the parallel, integrative approach came from eye movements, but still some researchers argued that the discontinuity of eye-movement patterns could be accommodated by discrete-symbolic accounts. The link between manual and mental dynamics has recently served as compelling evidence that language processing is indeed, at multiple levels, gradient and parallel.
Spivey et al. () asked participants to move the computer mouse from the bottom-center of the screen to the top-left or top-right corners. For instance, a picture of a candle appeared in the top-left and a picture of a candy appeared in the top-right while a spoken phrase was heard (e.g., “click the candle”). When the distractor object corresponded to a word whose initial set of phonemes overlapped with the spoken word (e.g., a picture of candy, for the spoken word, candle), hand movements showed a continuous attraction toward the distractor before settling into the correct alternative. Thus, the ongoing accrual of the spoken word partially activated multiple lexical representations (e.g., both candle and candy), causing movements to partially gravitate toward candy before settling into candle. This showed that the language system continually integrates the acoustic-phonetic input of a spoken word, triggering parallel competing representations that resolve over time. This permitted a discrimination among theoretical models previously not possible.
In some cases, however, comprehending spoken language is more difficult and would benefit from help of the visual system. Consider the following syntactically ambiguous sentence:
Put the apple on the towel in the box.
Typically, listeners resolve such ambiguities by incorporating available visual cues (e.g., is there an apple on a towel or an apple by itself?). In a series of studies, participants were given several objects on a screen while they heard a sentence instructing them to move one object to another. For example, hearing the above sentence, participants would use a computer mouse to click on an image displaying an apple on a towel, and to then drag it onto an image of a box. Before reaching the box, however, participants’ hand movements exhibited a continuous attraction toward the object associated with the incorrect syntactic parse (in this case, an empty towel). Consistent with analogous eye-tracking studies (e.g., Tanenhaus et al., ), the attraction effect was not seen when a second relevant object was present (e.g., an apple on a napkin). This provided strong support for constraint-based accounts of language processing by demonstrating partially active syntactic alternatives competing over time with a weighing-in of visual, contextual, and linguistic factors (Farmer et al., ,). Importantly, hand movements were able to reveal competition on a trial-by-trial basis, thereby evading longstanding concerns with analogous eye-movement results that were limited by discrete oculomotor output.
In some contexts, however, sentence processing may not flow so smoothly. For instance, Dale and Duran () had participants verify simple statements by clicking on a “true” or “false” response. Statements involving a negation (e.g., “cats are not dogs”) led hand movements to exhibit sharp shifts in direction, initially pursuing one response and then rapidly redirecting toward the other, especially for negated true statements. Thus, during sentence processing, participants experienced an abrupt shift in their evolving decision. Classic work suggests that negation is an operator that rapidly reverses the truth/falsity of an interpretation and predicts shifts in cognitive dynamics, consistent with other work showing that hand movement may index such rapid transitions (e.g., Resulaj et al., ). These rapid cognitive shifts, however, could not be evidenced by traditional measures.
Social cognition
Hand movements have also been critical in uncovering the temporal dynamics of social cognition. For example, they have provided new insights into social categorization. In one series of studies, participants were presented with sex-typical (e.g., masculine male) and sex-atypical (e.g., feminine male) faces and asked to categorize the target's sex by clicking on a “male” or “female” response on the screen. During categorization of sex-atypical faces, hand trajectories were continuously attracted to the opposite sex-category before settling into the correct response. This was evidence that social categorization involves dynamic competition, where partially active category representations (male and female) continuously compete over time to stabilize onto one categorical outcome (Freeman et al., ; also see Dale et al., ). These findings led to a new model of social categorization, proposing that all available facial, vocal, and bodily cues (among other constraints) simultaneously weigh in on multiple partially active category representations, and that these dynamically evolve over time into stable categorical perceptions (Freeman and Ambady, ).
This model was supported by further evidence from a similar paradigm with race categorization. Hand movements again showed continuous-attraction effects for atypical targets, but also revealed important dissociations in how the processing of White-specifying and Black-specifying cues evolves across the categorization time-course. Moreover, as the amount of racial ambiguity increased, hand trajectories showed corresponding increases in attraction toward the unselected race-category and increases in motor complexity. This showed that the competition inherent to social categorization sensitively increases and decreases with the amount of ambiguity challenging the perceptual system (Freeman et al., ). Further, hand movements have revealed not only that multiple facial cues are dynamically integrated, but also that cues across sensory modalities (facial and vocal cues) are dynamically integrated as well (Freeman and Ambady, ). See Figure 1. Hand movements have also been used to temporally dissociate the integration of specific cues, such as facial shape and pigmentation (Freeman and Ambady, ).
Figure 1
The competition during categorization can also bear downstream effects. For instance, Freeman and Ambady (
High-level cognition and learning
Even high-level decision-making processes can be revealed in manual dynamics. McKinstry et al. (
Duran et al. (
Figure 2

Hand movements provide insights into deception. Duran et al. (
Finally, at an even longer timescale than decisions, Dale et al. (
Further Questions and Conclusions
The work discussed above suggests that the intimate link between manual and mental dynamics may be used to provide critical insights into psychological phenomena. It also emphasizes an integrative approach to these phenomena by demonstrating a deep coextension of processes typically investigated independently, such as action, perception, and both basic and social cognition. Insights from simple manual actions may therefore serve as an unforeseen bridge between far-reaching areas of psychological science.
This action-dynamics approach to the mind holds particular promise given the practicality of measuring hand movements. There now exists freely available, user-friendly software to run and analyze data from computer mouse-tracking experiments (Freeman and Ambady,
In this action-dynamics approach, researchers are afforded many ways to measure and analyze hand trajectories. The computer mouse is by far the most practical, but other options include a wireless remote (e.g., Dale et al.,
Despite the headway made by the work reviewed here, however, the moving hand still has much to reveal about the mind. For example, in a recent critique of the method and its interpretation, van der Wel et al. (
There is also not yet a clear understanding of the particular cognitive processes directly indexed by hand movements. For instance, several of the studies described here reported effects of motor competition, but one might wonder whether the competition may be mediated by attentional mechanisms (e.g., see Rizzolatti et al.,
Figure 3

Parallel competition vs. strategic response switching. Farmer et al. (
Another issue is to relate this methodological approach to the long-established precedent of reaction times and reaction time distributions (e.g., Van Zandt,
A final pressing question is how fine-grained mental dynamics provided by hand movements relate to higher-level phenomena, such as behavior, educational and clinical outcomes, or social interaction. More work is needed to tie together data across these multiple time scales. Moreover, future research would benefit from combining hand movements with eye-movement and electroencephalography (EEG) data for a more complete characterization of the time-course of perception, cognition, and action, and their many interactions. Ultimately, this work could be used to bridge diverse domains of psychological science and help form truly integrative accounts of the mind.
Statements
Acknowledgments
This work was supported by NIH fellowship (F31-MH092000) to Jonathan Freeman and NSF research grants (BCS-0826825 and BCS-0720322) to Rick Dale. The authors thank Nalini Ambady, Ken Nakayama, and Phillip Holcomb for helpful insights.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
1
AndersonS.E.FarmerT.A.GoldsteinM.SchwadeJ.SpiveyM. (in press). “Individual differences in measures of linguistic experience account for variability in the sentence processing skill of five-year-olds,” in How children make linguistic generalizations: Experience and variation in first language acquisition, eds ArnonI.ClarkE. V. (Amsterdam: Benjamins).
2
CisekP.KalaskaJ. F. (2005). Neural Correlates of Reaching Decisions in Dorsal Premotor Cortex: specification of multiple direction choices and final selection of action. Neuron45, 801–814.10.1016/j.neuron.2005.01.027
3
DaleR.DuranN. D. (2011). The cognitive dynamics of negated sentence verification. Cogn. Sci.10.1111/j.1551-6709.2010.01164.x
4
DaleR.KehoeC.SpiveyM. J. (2007). Graded motor responses in the time course of categorizing atypical exemplars. Mem. Cognit.35, 15–28.
5
DaleR.RocheJ.SnyderK.McCallR. (2008). Exploring action dynamics as an index of paired-associate learning. PLoS ONE3, e1728.10.1371/journal.pone.0001728
6
DuranN. D.DaleR.McNamaraD. S. (2010). The action dynamics of overcoming the truth. Psychon. Bull. Rev.17, 486–491.
7
FarmerT. A.AndersonS. E.SpiveyM. J. (2007a). Gradiency and visual context in syntactic garden-paths. J. Mem. Lang.57, 570–595.10.1016/j.jml.2007.04.003
8
FarmerT. A.CargillS.HindyN.DaleR.SpiveyM. J. (2007b). Tracking the continuity of language comprehension: computer-mouse trajectories suggest parallel syntactic processing. Cogn. Sci.31, 889–909.10.1080/03640210701530797
9
FreemanJ. B.AmbadyN. (2009). Motions of the hand expose the partial and parallel activation of stereotypes. Psychological Science20, 1183–1188.10.1111/j.1467-9280.2009.02422.x
10
FreemanJ. B.AmbadyN. (2010). MouseTracker: software for studying real-time mental processing using a computer mouse-tracking method. Behav. Res. Methods42, 226–241.
11
FreemanJ. B.AmbadyN. (2011a). When two become one: temporally dynamic integration of the face and voice. J. Exp. Soc. Psychol.47, 259–263.10.1016/j.jesp.2010.08.018
12
FreemanJ. B.AmbadyN. (2011b). A dynamic interactive theory of person construal. Psychol. Rev.118, 247–279.10.1037/a0022327
13
FreemanJ. B.AmbadyN. (in press). Hand movements reveal the time-course of shape and pigmentation processing in face categorization. Psychonomic Bulletin and Rev.10.3758/s13423-011-0097-6.
14
FreemanJ. B.AmbadyN.MidgleyK. J.HolcombP. J. (2011). The real-time link between person perception and action: brain potential evidence for dynamic continuity. Soc. Neurosci.6, 139–155.
15
FreemanJ. B.AmbadyN.RuleN. O.JohnsonK. L. (2008). Will a category cue attract you? Motor output reveals dynamic competition across person construal. J. Exp. Psychol. Gen.137, 673–690.
16
FreemanJ. B.PaukerK.ApfelbaumE. P.AmbadyN. (2010). Continuous dynamics in the real-time perception of race. J. Exp. Soc. Psychol.46, 179–185.
17
Goldin-MeadowS.AlibaliM. W. (2002). “Looking at the hands through time: a microgenetic perspective on learning and instruction,” in Microdevelopment: Transition Processes in Development and Learning, eds GranottN.ParzialeJ. (Cambridge: Cambridge University Press), 80–105.
18
GoodaleM. A.PelissonD.PrablancC. (1986). Large adjustments in visually guided reaching do not depend on vision of the hand or perception of target displacement. Nature320, 748–750.10.1038/320748a0
19
MagnusonJ. S. (2005). Moving hand reveals dynamics of thought. Proc. Natl. Acad. Sci. U.S.A.102, 9995–9996.
20
McKinstryC.DaleR.SpiveyM. J. (2008). Action dynamics reveal parallel competition in decision making. Psychol. Sci.19, 22–24.
21
MilesL. K.BetkaE.PendryL. F.MacraeC. N. (2010). Mapping temporal constructs: actions reveal that time is a place. Q. J. Exp. Psychol. (Colchester)63, 2113–2119.10.1080/17470218.2010.524932
22
PaninskiL.FellowsM. R.HatsopoulosN. G.DonoghueJ. P. (2004). Spatiotemporal tuning of motor cortical neurons for hand position and velocity. J. Neurophysiol.91, 515–532.10.1152/jn.00587.2002
23
RatcliffR.Van ZandtT.McKoonG. (1999). Connectionist and diffusion models of reaction times. Psychol. Rev.106, 261–300.
24
ResulajA.KianiR.WolpertD. M.ShadlenM. N. (2009). Changes of mind in decision-making. Nature461, 263–266.10.1038/nature08275
25
RizzolattiG.RiggioL.DascolaI.UmiltaC. (1987). Reorienting attention across the horizontal and vertical meridians: evidence in favor of a premotor theory of attention. Neuropsychologia25, 31–40.10.1016/0028-3932(87)90041-8
26
RosenbaumD. A. (2005). The Cinderella of psychology: the neglect of motor control in the science of mental life and behavior. Am. Psychol.60, 308–317.
27
SongJ. H.NakayamaK. (2006). Role of focal attention on latencies and trajectories of visually guided manual pointing. J. Vis.6, 982–995.
28
SongJ. H.NakayamaK. (2008). Target selection in visual search as revealed by movement trajectories. Vision Res.48, 853–861.
29
SongJ. H.NakayamaK. (2009). Hidden cognitive states revealed in choice reaching tasks. Trends Cogn. Sci. (Regul. Ed.)13, 360–366.
30
SpiveyM. J. (2007). The Continuity of Mind. New York: Oxford University Press.
31
SpiveyM. J.DaleR.KnoblichG.GrosjeanM. (2010). Do curved reaching movements emerge from competing perceptions? A reply to van der Wel et al. (2009). J. Exp. Psychol. Hum. Percept. Perform. 36, 251–254.
32
SpiveyM. J.GrosjeanM.KnoblichG. (2005). Continuous attraction toward phonological competitors. Proc. Natl. Acad. Sci.102, 10393–10398.
33
TanenhausM. K.Spivey-KnowltonM. J.EberhardK. M.SedivyJ. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science268, 1632–1634.10.1126/science.7777863
34
van der WelR. P. R. D.EderJ. R.MitchelA. D.WalshM. W.RosenbaumD. A. (2009). Trajectories emerging from discrete versus continuous processing models in phonological competitor tasks: a commentary on Spivey, Grosjean, and Knoblich (2005). J. Exp. Psychol. Hum. Percept. Perform.35, 588–594.
35
Van ZandtT. (2000). How to fit a response time distribution. Psychon. Bull. Rev.7, 424–465.
36
WojnowiczM. T.FergusonM. J.DaleR.SpiveyM. J. (2009). The self-organization of explicit attitudes. Psychol. Sci.20, 1428–1435.
Summary
Keywords
hand, motor, real-time, temporal dynamics, time-course, mouse-tracking
Citation
Freeman JB, Dale R and Farmer TA (2011) Hand in Motion Reveals Mind in Motion. Front. Psychology 2:59. doi: 10.3389/fpsyg.2011.00059
Received
08 February 2011
Accepted
24 March 2011
Published
20 April 2011
Volume
2 - 2011
Edited by
Anna M. Borghi, University of Bologna, Italy
Reviewed by
Ruud Custers, Utrecht University, Netherlands; Michael Kaschak, Florida State University, USA
Copyright
© 2011 Freeman, Dale and Farmer.
This is an open-access article subject to a non-exclusive license between the authors and Frontiers Media SA, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and other Frontiers conditions are complied with.
*Correspondence: Jonathan B. Freeman, Department of Psychology, Tufts University, 490 Boston Avenue, Medford, MA 02155, USA. e-mail: jon.freeman@tufts.edu
This article was submitted to Frontiers in Cognition, a specialty of Frontiers in Psychology.
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.