A commentary has been posted on this article:
Extending the reach of mousetracking in numerical cognition: a comment on Fischer and Hartmann (2014)
Martin H. Fischer
Matthias Hartmann- Cognitive Science, University of Potsdam, Potsdam, Germany
Freely available software has popularized “mousetracking” to study cognitive processing; this involves the on-line recording of cursor positions while participants move a computer mouse to indicate their choice. Movement trajectories of the cursor can then be reconstructed off-line to assess the efficiency of responding in time and across space. Here we focus on the process of selecting among alternative numerical responses. Several studies have recently measured the mathematical mind with cursor movements while people decided about number magnitude or parity, computed sums or differences, or simply located numbers on a number line. After some general methodological considerations about mouse tracking we discuss several conceptual concerns that become particularly evident when “mousing” the mathematical mind.
Today, cognitive scientists no longer study higher-level cognition separate from sensory and motor processes, even when investigating supposedly abstract knowledge domains such as language comprehension or numerical cognition. The “embodied turn” over the last two decades (Varela et al., 1991; Wilson, 2002; Glenberg et al., 2013) has raised interest in dynamic responses that presumably reflect underlying conceptual competition in real time.
Freely available software (Freeman and Ambady, 2010) has popularized “mousetracking” to study cognitive processing; this involves the on-line recording of cursor positions while participants move a computer mouse to indicate their choice (e.g., Spivey, 2007). Movement trajectories of the cursor can then be reconstructed off-line to assess the efficiency of responding in time and across space. Here we focus on the process of selecting among alternative numerical responses. Several studies have recently measured the mathematical mind with cursor movements while people decided about number magnitude or parity (Weaver and Arrington, 2013; Faulkenberry, 2014), computed sums or differences (Marghetis et al., 2014), or simply located numbers on a number line (Dotan and Dehaene, 2013). After some general methodological considerations about mousetracking we discuss several conceptual concerns that become particularly evident when “mousing” the mathematical mind.
Methodological Considerations about Mousetracking
Hard- and Software Issues
In contrast to established kinematic motion tracking, a computer mouse does not record three-dimensional position but position changes in two dimensions along an uncalibrated part of space that changes whenever we lift the mouse off its surface. Moreover, the temporal recording of mouse coordinates relies on the computer's operating system, which introduces limitations in sampling rate and temporal uncertainties. Despite these limitations, a formal comparison reveals reasonable recording quality if users exert some cautions (O'Reilly and Plamondon, 2011; see Box 1).
Box 1. A checklist for conducting mousetracking studies.
Checklist for conducting a mousetracking study
□ Reduce the participant's degrees of freedom
Constrain the yaw (rotation around the vertical axis) of the mouse-pad to prevent hand rotations which are not adequately captured in the cursor trajectory, e.g., by wearing a wrist band.
□ Change default mouse settings
Disable the default mouse acceleration option in the control panel of your operating system (“dynamic acceleration option” as labeled in Windows XP or “Enhance pointer precision” as labeled in Windows 7. Note that for Windows 7, additional effort is required to disable the acceleration function completely, for example by using a more sophisticated “gaming” mouse; for Macintosh users, type “defaults write .GlobalPreferences com.apple.mouse.scaling -1” into the Terminal (mouse acceleration cannot be disabled directly in the Mac control panel).
Also lower the default speed of the mouse to a reasonable range (e.g., second value from the left in the control panel) to capture cognitive effects in the trajectory measures.
□ Report mouse settings
Report mouse settings as selected in the control panel and also report the resulting hand-to-cursor movement ratio (e.g., 1 cm hand movement results in x pixels mouse cursor displacement).
□ Report exact task instructions
Instructing participants to begin the mouse movement at the beginning of the trial (before response selection has finished) helps to capture cognitive effects in the trajectory measures.
□ Data analysis
Control for bimodality (compute bimodality coefficients or Hartigan's dip statistic, or/and show probability plots of mouse trajectories).
Depending on the mouse settings in the computer's control panel, experienced users can displace the cursor by quick pivoting movements of the wrist instead of displacing the hand smoothly across the desk, so that there is no linear relationship between hand displacement and cursor displacement. In mousetracking studies, mouse settings should therefore be selected carefully to prevent scaling of mouse cursor displacement. This includes disabling the “dynamic acceleration option” which is enabled by default, and lowering the speed of the mouse (see Box 1). Because these mouse settings play a crucial role, we advise to report the exact settings in the Method section, along with the display resolution, mouse sensitivity and resulting displacement ratio (see Bruhn et al., 2014, for an example). To date, the majority of studies do not report this information.
Tracking Different Phases of the Cognitive Process
A motor response generally consists of two phases: planning and execution. Movement planning begins when the target is (at least partially) known and ends when there is a physical displacement that initiates movement execution. When using classical response time paradigms, the response movement consists merely of a finger twitch on a button and scientists therefore rarely care about movement characteristics. With hand displacements, however, the early part of movement execution will largely reflect motor planning. Due to cognitive processing times, as well as afferent and efferent neural delays, only the later part of the movement will be sensitive to new information about the current distance of the hand (or cursor) to the target.
Mousetracking allows researchers to push cognitive processing into movement execution and thereby makes features of the trajectory itself diagnostic. To this end, it is crucial to instruct participants to start moving their hands at the beginning of a trial, before the decision-related cognitive process is completed. In order to enhance such a behavior, a minimal displacement requirement shortly after target onset has been defined in some studies, and participants are reminded to start moving earlier when the requirement was not fulfilled in the previous trial (e.g., Freeman and Ambady, 2009; Scherbaum et al., 2010; Dshemuchadse et al., 2013; Faulkenberry, 2014; Marghetis et al., 2014). Some studies even require participants to move the hand before the target information in each trial is released (e.g., Dotan and Dehaene, 2013; Bruhn et al., 2014). However, some studies do not emphasize early movement onsets, inviting participants to complete decision-related cognitive processes before initiating their response, thus making initiation time (the time until movement onset) a more diagnostic measure (e.g., Weaver and Arrington, 2013). Since this trade-off between reaction time and movement time strongly depends on task instructions, we recommend reporting exact task instructions (see Box 1).
Interpreting Mouse Trajectories
Mousetracking typically involves moving the mouse cursor from the central start box at the bottom of a display to either the left or right target box at the top of the display. There are two types of resulting trajectories: those where incongruent response mappings induce crossing over into the wrong hemispace before returning into the correct hemispace (e.g., Weaver and Arrington, 2013), and those where even under the incongruent mapping all trajectories remain in the correct hemispace and merely have differentially strong curvatures (e.g., Faulkenberry, 2014; Marghetis et al., 2014). Both types of results are currently interpreted as attraction by the competing distracting stimulus, due to the theoretical framework of dynamic competition (Spivey, 2007). However, in our opinion, only the former case, where trajectories actually verge into the distractor's hemifield, can be interpreted as evidence for attraction by the competing distractor. In the other case there is no spatial bias away from the correct target and curvature might simply reflect the earlier or later occurrence of the participants' decisions, due to increased task difficulty (cf. Faulkenberry, 2014). Moreover, even in the case where mean trajectories verge into the distractor's hemifield, this cannot automatically be taken as evidence for a continuous competitive cognitive process. Such a pattern can instead be the result of a small subset of trials in which participants incorrectly aimed for the wrong solution and corrected their trajectory during the motion. The latter case results in a bivariate variance distribution. It is therefore crucial to test variance distributions, for example by computing bimodality coefficients (cf. Spivey et al., 2005), or by using Hartigan's dip statistic (cf. Freeman and Dale, 2013; Faulkenberry, 2014). Given that this procedure tests the null hypothesis of uni-modal distributions, p-values that are only slightly larger than 0.05 should not be interpreted as evidence for a uni-modal distribution (null-hypothesis tests can yield p-values greater than 0.05 even when the tested assumption is violated to a degree that significantly affects the results of classic parametric tests; see Erceg-Hurn and Mirosevich, 2008). In case the researcher is interested to maintain the null hypothesis, it has been suggested to increase the conventional significance level α from 0.05 to 0.1 or 0.2 (Bortz and Schuster, 2010, p. 128). An alternative (or complementary) way to illustrate whether the average curve is representative for task performance is to present probability plots of mouse trajectories (see Figure 4 in Dshemuchadse et al., 2013 or Figure 2 in Scherbaum et al., 2010, for nice examples).
May we Mouse the Mathematical Mind? Some Conceptual Concerns
Most conceptual domains can convey spatial meanings (e.g., the words “left” or “right”; or a directed gaze). However, none exhibits the rich and obligatory association of semantic features with space that characterizes number concepts. First, small and large magnitudes are associated to left/lower and right/upper space, respectively, leading to systematic biases in spatial behavior for single digit processing (the SNARC effect; Dehaene et al., 1993) as well as for mental arithmetic (the Operational Momentum effect; McCrink et al., 2007). For recent review of both effects see Fischer and Shaki (2014). Second, odd and even numbers are associated with left and right space, respectively, probably reflecting linguistic markedness of the associated labels (MARC effect; Nuerk et al., 2004). Third, each digit presentation requires a particular font size or auditory frequency that activates spatial associations indirectly, triggering the size congruity effect (SiCE; Henik and Tzelgov, 1982) for vision and the spatial-musical association of response codes for audition (SMARC effect; Rusconi et al., 2005; Fischer et al., 2013). Finally, in the case of multi-digit strings the relative position of each digit in the string determines its meaning via the place-value system (Nuerk et al., 2011, for review). This up to 6-fold association between space and number meaning(s) makes the interpretation of mouse trajectories in numerical tasks quite challenging: We need to know when the magnitude meaning of a number is known relative to its other spatially associated features, such as its parity, its decimal structure or its perceived intensity. An interpretation of typical trajectory-based measures, such as divergence points, area under the curve, or maximal deviation, is constrained by these uncertainties (for a detailed evaluation of trajectory biases from different features of number representation, see Dotan and Dehaene, 2013).
Moreover, the spatial nature of number concepts raises concerns about the validity of the mousetracking task itself, which requires movements in the horizontal plane in order to displace a cursor in the vertical plane. This task requirement raises two concerns: First, this visuo-motor mapping is non-intuitive and requires considerable mental effort to coordinate actions in one plane and their effects in another plane (e.g., Cunningham and Pavel, 1991). This non-intuitive transformation and the fact that the data reflect changes in cursor position, and not veridical hand position, make it implausible to assume that we obtain a valid proxy for “a record of the mental trajectory traversed” (Spivey et al., 2005, p. 10,398). Ideally, mousetracking users should constrain the yaw (rotation around the vertical axis) of the mouse-pad to prevent hand rotations (see Box 1). More suitable (and still relatively inexpensive) might be the direct recording of two dimensional hand position with digitizing tablets or even three-dimensional body position with Kinect© technology (e.g., Festman et al., 2013).
More importantly, the continuous forward movement of the hand, as well as the continuous upward movement of the cursor, both induce systematic biases into the activation of number concepts. Additionally, the mouse itself is typically located in the participant's right hemi-space and operated with the preferred (right) hand. Together, these four factors (the two movement directions and the two right spatial codes) are all associated with larger numbers. For example, turning right activates larger numbers (Loetscher et al., 2008; Hartmann et al., 2012; Shaki and Fischer, 2014), addition is easier when moving one's hand upward (Wiemers et al., 2014), and also forward and backward motion does interact with number processing (Fischer and Campens, 2009; Seno et al., 2011; Marghetis and Youngstrom, 2014). These inherent biases make number task as “special case” for mousetracking investigations. For number studies, we propose to move away from the standard paradigm (starting in the middle of the lower screen and move to the top left vs. top right) that does not allow researchers to capture adequately the various spatial-numerical associations. Instead, it may be helpful to incorporate additional spatial manipulations, such as starting at the top, placing the mouse in the center or the left side of the screen, or reversing the forward-upward-translation between mouse and visual motion. These manipulations might help to capture the various spatial-numerical association and to advance the understanding of their dynamic influence on cognition.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
Martin H. Fischer's work is funded by ESF grant EW12-114 “From Numbers To Knowledge—20 Years Of Spatial-Numerical Associations” and by DFG grant 1915/2 on “manumerical cognition.” Matthias Hartmann was funded by Swiss national Science Foundation (P2BEP1_152104).
References
Bortz, J., and Schuster, C. (2010). Statistik für Human- und Sozialwissenschaftler, 7th Edn. [Statistics for human and social scientists]. Berlin: Springer. doi: 10.1007/978-3-642-12770-0
Bruhn, P., Huette, S., and Spivey, M. (2014). Degrees of certainty modulates anticipatory processes in real time. J. Exp. Psychol. Hum. Percept. Perform. 40, 525–538. doi: 10.1037/a0034365
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Cunningham, H. A., and Pavel, M. (1991). “Target axis effects under transformed visual-motor mappings,” in Pictorial Communication in Virtual and Real Environments, ed S. R. Ellis (London: Taylor & Francis), 283–294.
Dehaene, S., Bossini, S., and Giraux, P. (1993). The mental representation of parity and number magnitude. J. Exp. Psychol. Gen. 122, 371–396. doi: 10.1037/0096-3445.122.3.371
Dotan, D., and Dehaene, S. (2013). How do we convert a number into a finger trajectory? Cognition 129, 512–529. doi: 10.1016/j.cognition.2013.07.007
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Dshemuchadse, M., Scherbaum, S., and Goschke, T. (2013). How decisions emerge: action dynamics in intertemporal decision making. J. Exp. Psychol. Gen. 142, 93–100. doi: 10.1037/a0028499
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Erceg-Hurn, D. M., and Mirosevich, V. M. (2008). Modern robust statistical methods: an easy way to maximize the accuracy and power of your research. Am. Psychol. 63, 591–601. doi: 10.1037/0003-066X.63.7.591
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Faulkenberry, T. J. (2014). Hand movements reflect competitive processing in numerical cognition. Can. J. Exp. Psychol. 68, 147–151. doi: 10.1037/cep0000021
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Festman, Y., Lindemann, O., and Fischer, M. H. (2013). “Using Xbox Kinect to explore spatial-numerical association of arm movements in parity judgments,” in Poster presented at Embodied and Situated Language Processing (Potsdam).
Fischer, M. H., and Campens, H. (2009). Pointing to numbers and grasping magnitudes. Exp. Brain Res. 192, 149–153. doi: 10.1007/s00221-008-1622-3
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Fischer, M. H., Riello, M., Giordano, B. L., and Rusconi, E. (2013). Singing numbers… in cognitive space. Topics Cogn. Sci. 5, 354–366. doi: 10.1111/tops.12017
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Fischer, M. H., and Shaki, S. (2014). Spatial associations in numerical cognition: from single digits to arithmetic. Q. J. Exp. Psychol. 67, 1461–1483. doi: 10.1080/17470218.2014.927515
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Freeman, J. B., and Ambady, N. (2009). Motions of the hand expose the partial and parallel activation of stereotypes. Psychol. Sci. 20, 1183–1188. doi: 10.1111/j.1467-9280.2009.02422.x
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Freeman, J. B., and Ambady, N. (2010). MouseTracker: software for studying real-time mental processing using a computer mouse-tracking method. Behav. Res. Methods 42, 226–241. doi: 10.3758/BRM.42.1.226
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Freeman, J. B., and Dale, R. (2013). Assessing bimodality to detect the presence of a dual cognitive process. Behav. Res. Methods 45, 83–97. doi: 10.3758/s13428-012-0225-x
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Glenberg, A. M., Witt, J. K., and Metcalfe, J. (2013). From the revolution to embodiment: 25 years of cognitive psychology. Perspect. Psychol. Sci. 8, 573–585. doi: 10.1177/1745691613498098
Hartmann, M., Grabherr, L., and Mast, F. W. (2012). Moving along the mental number line: interactions between whole-body motion and numerical cognition. J. Exp. Psychol. Hum. Percept. Perform. 38, 1416–1427. doi: 10.1037/a0026706
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Henik, A., and Tzelgov, J. (1982). Is three greater than five: the relation between physical and semantic size in comparison tasks. Mem. Cognit. 10, 389–395. doi: 10.3758/BF03202431
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Loetscher, T., Schwarz, U., Schubiger, M., and Brugger, P. (2008). Head turns bias the brain's internal random generator. Curr. Biol. 18, R60–R62. doi: 10.1016/j.cub.2007.11.015
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Marghetis, T., Núñez, R., and Bergen, B. (2014). Doing arithmetic by hand: Hand movements during exact arithmetic reveal systematic, dynamic spatial processing. Q. J. Exp. Psychol. 67, 1579–1596. doi: 10.1080/17470218.2014.897359
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Marghetis, T., and Youngstrom, K. (2014). “Pierced by the number-line: integers are associated with back-to-front sagittal space,” in Proceedings of the 36th Annual Conference of the Cognitive Science Society (Austin, TX: Cognitive Science Society).
McCrink, K., Dehaene, S., and Dehaene-Lambertz, G. (2007). Moving along the number line: operational momentum in nonsymbolic arithmetic. Percept. Psychophys. 69, 1324–1333. doi: 10.3758/BF03192949
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Nuerk, H. C., Iversen, W., and Willmes, K. (2004). Notational modulation of the SNARC and the MARC (linguistic markedness of response codes) effect. Q. J. Exp. Psychol. 57A, 835–863. doi: 10.1080/02724980343000512
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Nuerk, H.-C., Moeller, K., Klein, E., Willmes, K., and Fischer, M. H. (2011). Extending the mental number line: a review of multi-digit number processing. Z. Psychol. 219, 3–22. doi: 10.1027/2151-2604/a000041
O'Reilly, C., and Plamondon, R. (2011). Can computer mice be used as low-cost devices for the acquisition of planar human movement velocity signals? Behav. Res. 43, 229–238. doi: 10.3758/s13428-010-0030-3
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Rusconi, E., Kwan, B., Giordano, B., Umiltà, C., and Butterworth, B. (2005). Spatial representation of pitch height: the SMARC effect. Cognition 99, 113–129. doi: 10.1016/j.cognition.2005.01.004
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Scherbaum, S., Dshemuchadse, M., Fischer, R., and Goschke, T. (2010). How decisions evolve: the temporal dynamics of action selection. Cognition 115, 407–416. doi: 10.1016/j.cognition.2010.02.004
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Seno, T., Taya, S., Ito, H., and Sunaga, S. (2011). The mental number line in depth revealed by vection. Perception 40, 1241–1244. doi: 10.1068/p6965
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Shaki, S., and Fischer, M. H. (2014). Random walks on the number line. Exp. Brain Res. 211, 73–85. doi: 10.1007/s00221-013-3718-7
Spivey, M. J., Grosjean, M., and Knoblich, G. (2005). Continuous attraction toward phonological competitors. Proc. Natl. Acad. Sci. U.S.A. 102, 10393–10398. doi: 10.1073/pnas.0503903102
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Varela, F. J., Thompson, E., and Rosch, E. (1991). The Embodied Mind: Cognitive Science and Human Experience. Cambridge, MA: MIT Press.
Weaver, S. M., and Arrington, C. M. (2013). Tracking the multitasking mind. Z. Psychol. 221, 51–60. doi: 10.1027/2151-2604/a000130
Wiemers, M., Bekkering, H., and Lindemann, O. (2014). Spatial interferences in mental arithmetic: evidence from the motion-arithmetic compatibility effect. Q. J. Exp. Psychol. 67, 1557–1570. doi: 10.1080/17470218.2014.889180
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Wilson, M. (2002). Six views of embodied cognition. Psychon. Bull. Rev. 9, 625–636. doi: 10.3758/BF03196322
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Keywords: mousetracking, numerical cognition, SNARC effect, trajectories, on-line processing
Citation: Fischer MH and Hartmann M (2014) Pushing forward in embodied cognition: may we mouse the mathematical mind? Front. Psychol. 5:1315. doi: 10.3389/fpsyg.2014.01315
Received: 13 October 2014; Paper pending published: 27 October 2014;
Accepted: 29 October 2014; Published online: 20 November 2014.
Edited by:
Guillaume T. Vallet, Centre de Recherche de l'Institut Universitaire de Gériatrie de Montréal, CanadaReviewed by:
Thomas J. Faulkenberry, Tarleton State University, USAAmandine E. Rey, Lyon 2 University, France
Copyright © 2014 Fischer and Hartmann. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Martin H. Fischer, Division of Cognitive Science, University of Potsdam, Karl-Liebknecht-Str. 24-25, House 14, D14476 Potsdam OT Golm, Germany e-mail: martinf@uni-potsdam.de