Spatially Localized Time Shifts of the Perceptual Stream

Visual events trigger representations in different locations and times in the brain. In experience, however, these various neural responses refer to a single unified cause. To investigate how representations might be brought into temporal alignment, we attempted to locally manipulate neural processing in such a way that identical, simultaneous sequences would appear temporally misaligned. After adaptation to a 20 Hz sequentially expanding and contracting concentric grating, a running clock presented in the adapted region of the visual field appeared advanced relative to an identical clock presented simultaneously in an unadapted region. No such effect was observed following 5-Hz adaptation. Clock time reports following an exogenous cue showed the same effect of adaptation on perceived time, demonstrating that the apparent temporal misalignment was not mediated by differences in target selection or allocation of attention. This effect was not mediated by the apparent speed of the adapted clock: a clock in a 20-Hz-adapted spatial location appeared slower than a clock in a 5-Hz-adapted location, rather than faster. Furthermore, reaction times for a clock-hand orientation discrimination task were the same following 5- and 20-Hz adaptation, indicating that neural processing latencies were not differentially affected. Altogether, these findings suggest that the fragmented perceptual stream might be actively brought into temporal alignment through adaptive local mechanisms operating in spatially segregated regions of the visual field.

changes in color (Moutoussis and Zeki, 1997), suggesting temporal mechanisms are at the mercy of delays in neural representation. However, this experimental paradigm is cognitively complex, requiring comparison across submodalities and different types of transient temporal features. Furthermore, it may tax visual routines for selection, individuation and comparison (Nishida and Johnston, 2002;Holcombe, 2009;Nishida and Johnston, 2010). Adaptation paradigms are more readily interpretable. Adaptation to audiovisual delays can result in a reduction in apparent temporal asynchrony and a similar change to the distribution of the stream-bounce illusion, in which colliding objects appear to bounce rather than pass through each other depending on the relative timing of a tone and the collision (Sekuler et al., 1997). A shift in this distribution suggests a perceptual adaptation rather than a cognitive bias (Fujisaki et al., 2004). However, in this approach temporal realignment is implied, rather than directly measured.
If adaptive regulatory mechanisms are responsible for co-aligning neural information and referring the result to a specific point in time, then it might be possible to modify perceptual processing such that two similar visual streams appear shifted in time. Johnston et al. (2006) showed that adaptation to a high temporal frequency (20 Hz) appeared to reduce the duration of a subsequent test stimulus. No such effect was observed for otherwise identical 5-Hz adaptation. Furthermore, this temporal distortion was tightly tuned to the spatial location of the adaptor (Ayhan et al., 2009) and independent of the apparent onset and offset of the test stimulus (Johnston et al., 2006), suggesting the presence of explicit local temporal information which might be used to bring visual processes into alignment.

IntroductIon
The visual brain is composed of many separate modular systems, each of which processes specific aspects of the incoming visual image. Inevitably, due to differences in transmission delays and processing requirements, the results of these separate computations will become available at different times (Bullier, 2001;Bartels and Zeki, 2005;Zeki, 2005). As a result, the various features of any given visual event are represented in different locations in the brain at different times (Johnston and Nishida, 2001). Since we consciously experience a temporally coherent visual world, the question is how the brain puts the visual image back together after it is distributed over both time and space.
This asynchronous architecture presents a significant computational problem: if the representations available in different modules at a given moment do not correspond to the same point in time in the outside world, how does the brain determine a property of the visual image that depends on output from multiple modules? (Fujisaki et al., 2004). Ideally, the inputs to a computation across modules should be co-aligned with respect to the time at which we perceive an event to have occurred. Only then will the result of the computation (and by extension, visual perception) accurately reflect the co-occurrence of external events.
A number of situations have been reported in which discrete visual events appear temporally distorted. Saccadic eye movements (Morrone et al., 2005) or superimposed dynamic visual noise (Terao et al., 2008) can distort the perceived timing of visual events to such a degree that the order of two sequential flashes can appear reversed. For frequencies greater than around 4 Hz, reversals in motion direction appear significantly delayed relative to physically simultaneous Before each block of each experiment, half of the clock locations were adapted to full-contrast concentric sinusoidal luminance gratings (spatial frequency 2 cpd) alternately drifting inwards and outwards (switching discretely) for periods of 1 s each, for a total of 12 s. The temporal frequency of the grating was either 5 or 20 Hz, in separate blocks. Odd and even locations were adapted in separate blocks. Before each trial, the same adapters were presented again in the same locations for 6 s to provide top-up adaptation (Figure 1).
The speed of the drifting gratings was therefore 2.5°/s in the 5-Hz condition and 10°/s in the 20-Hz condition. In Experiments 1-3, clock hands rotated at 1 Hz (on average in Experiment 2), such that the tangential speed of the clock hand spanned a comparable range: roughly ∼2-9°/s. Similarly, the thickness of the clock hand was chosen to be roughly 0.3°, such that its spatial frequency content (although necessarily broadband) was comparable to the adaptor. Although the clock stimulus is therefore a relatively complex stimulus, the spatiotemporal parameters were chosen such that spatiotemporal parameters of the adaptor and test stimuli had approximately the same order of magnitude.

ExpErImEnt 1
In Experiment 1 we determined the temporal offset at which an adapted and an unadapted clock appear to show the same time.
Here we tested whether adapting to a high temporal frequency introduces an apparent temporal realignment in a paradigm that required judgments about stimuli in continuous motion that could not be made by referring to offset or onset transients. We show that one of two rotating clocks can appear ahead of the other by a small but consistent amount, if the spatial location of that clock has previously been adapted to oscillating motion. This forms the first clear evidence that two synchronized, continuous displays can appear shifted in time, and provides a mechanism by which asynchronous visual computations can be integrated into a temporally coherent visual experience.

GEnEral mEthod
Five observers took part in each of four experiments. All observers reported normal or corrected-to-normal vision.
Stimuli were presented on a calibrated 22′′ Mitsubishi Diamondtron CRT monitor at a viewing distance of 80 cm (120 Hz refresh) controlled by a PC running Matlab (The Mathworks, Inc.) with PsychToolbox extensions. Stimuli consisted of an array of clocks, each subtending 3.3° of visual angle, presented on an imaginary circle 6.7° from a central fixation point. Eight clocks were presented in Experiments 1 and 2 and 10 clocks in Experiments 3 and 4. For a detailed introduction of the clocks stimulus, please see Carlson et al. (2006). Experiment 2 was similar to Experiment 1 with the exception that we manipulated the speeds, rather than positions, of the clock hands, in order to determine the speed difference at which adapted and unadapted clocks appear to run at the same speed. The initial clock hand positions were randomized on all clocks, and clock hands rotated at random rates drawn from a uniform distribution between 0.75 and 1.33 Hz. There were two target clocks on each trial, one in an adapted location and one in an unadapted location. The target clock in the adapted location always ran at 1 Hz, and the speed of the target clock in the unadapted location was systematically varied from 0.75 to 1.33 Hz. Observers reported which of the two target clocks appeared to run more quickly.
Observers completed 128 trials in each of eight blocks. Cumulative Gaussian psychometric functions were again fitted to observer responses. For each observer and each adaptation frequency, we calculated the speed at which an unadapted clock appeared to run at the same speed as the adapted clock, along with bootstrapped 95% confidence intervals for this estimate.

rEsults
Clocks presented in regions adapted to 20 Hz in fact appeared to run more slowly, and clocks in regions adapted to 5 Hz more rapidly, than clocks presented in unadapted regions (t(4) = −7.68, p = 0.002 and t(4) = 4.34, p = 0.012, respectively) (Figure 3). The effect of 20-Hz adaptation on perceived speed is therefore in the opposite direction to what would be required to explain the direction of the temporal decoupling of the synchronous clocks observed in Experiment 1. Moreover, the effect of 5-Hz adaptation is dissociated in Experiments 1 and 2. There is a clear effect on perceived speed, but no accompanying temporal decoupling. It is therefore unlikely that the temporal misalignment seen in Experiment 1 resulted from observers inaccurately extrapolating the positions of the moving clock hands.
Rather, it supports the conclusion that the desynchronization observed in Experiment 1 is in fact a temporal effect. An alternative, spatial interpretation, in which adaptation would cause moving objects to appear shifted forward in the direction of motion, would predict that those shifts be accompanied by increases in perceived speed. Although observers are not asked to make a temporal judgment per se, the task does require establishing simultaneity. Additionally, the precision with which a moving object can be localized has been shown to have a constant temporal, rather than spatial, limit (Linares et al., 2009).

ExpErImEnt 3
Attended items can appear to occur earlier than identical unattended items (Spence et al., 2001;Spence and Parise, 2010), and attentional access has been argued to require the serial allocation of limited attentional resources (Hogendoorn et al., 2010). The difference in apparent time observed in Experiment 1 might therefore have resulted from a spurious but systematic bias in the division of attention over adapted and unadapted clocks. In Experiment 3, we eliminated this possibility by using a task in which attention was not divided. mEthod All clock hands rotated at a constant speed of 1 Hz. The clocks were presented on the screen for 0.5 s, after which six were removed, leaving one target clock in an adapted location and one target clock in an unadapted location. These clocks continued to run for 1 s, after which they were replaced by a mask comprised of a clock with 10 randomly oriented static hands. The starting orientation of the hands on the six non-target clocks and one of the target clocks was randomized on each trial. The relative offset between the orientations of the clock hands on the two target clocks was systematically varied between −25.2° and 25.2° (corresponding to −70 and 70 ms). Observers compared the time shown on the two continuously running target clocks, reporting which appeared advanced relative to the other.
Observers completed 128 trials in each of eight blocks. Cumulative Gaussian psychometric functions were fitted to observer responses, using the psignifit MATLAB toolbox (Wichmann and Hill, 2001a,b), for each observer and each adaptation frequency. We subsequently calculated the relative temporal offset at which the adapted clock appeared synchronous with the unadapted clock (the point of subjective equality or PSE) along with bootstrapped 95% confidence intervals.

rEsults
After 20-Hz adaptation, the clock in the adapted location appeared shifted forward in time by almost 10 ms against the baseline no adaptation locations (t(4) = −4.88, p = 0.008). No such effect was observed following 5-Hz adaptation (t(4) = 1.91, p = 0.128; Figure 2). This pattern of results is similar to Johnston et al. (2006), who showed strong apparent duration compression after 20 Hz and weak or absent duration compression after 5-Hz adaptation. This temporal decoupling of event time (perceived time) and world time suggests that adaptation locally modifies temporal processing, directly influencing the mental representation of the time course of events.

ExpErImEnt 2
In Experiment 2, we addressed the possibility that the result of Experiment 1 was an artifact of the perceived speed of the clock hands, rather than a temporal shift. Adapting to motion can influence the perceived speed of stimuli presented in the adapted region (Thompson, 1981). In Experiment 1, the direction of motion of the test stimulus was orthogonal to the adaptor and would therefore not be expected to induce a static motion after effect (Shioiri and Matsumiya, 2009). However, an orthogonal motion aftereffect can be observed for dynamic test patterns (Grunewald and Lankheet, 1996), and non-directional adaptation effects on perceived velocity have been reported (Stocker and Simoncelli, 2007). Although in principle neither model would predict that a clock adapted to a high temporal frequency would appear to move faster than a clock adapted to a low temporal frequency, we wanted to eliminate the possibility that 20-Hz adapted clocks might appear to move more quickly, and as a consequence be seen as being ahead of unadapted clocks. To test this hypothesis, in Experiment 2 we measured the perceived speed of the clock hands after adaptation. of its rim from black to bright red for 20 ms. The clocks then continued to run for the remaining period, allowing attention to be allocated to the clock while it was still continuously running. Subsequently, the observer was presented with a single clock, the hand of which could be adjusted by a key press. The observer's task was to report what time he or she perceived on the cued clock The reported time was advanced relative to world time reflecting the well known flash-lag effect (Nijhawan, 1994;Eagleman and Sejnowski, 2000;Whitney et al., 2000). More importantly at the moment it was cued. Observers completed 100 trials in each of eight blocks. The mean difference between reported and veridical hand orientation was calculated separately for adapted and unadapted locations. In Experiment 2, observers reported which of two clocks appeared to move faster. A 20-Hz adaptation caused a clock in an adapted region to appear to move more slowly than a clock in an unadapted region, and 5-Hz adaptation caused a clock to appear to move more quickly. Ten clocks were presented after 5-and 20-Hz adaptation. All clock hands were stationary, with half oriented to the 3 o'clock position and half oriented to the 9 o'clock position. At a random time between 1 and 2 s after onset, one of the clocks was cued by changing the color of its rim from black to bright red. Observers' task was to report the orientation of the hand on the cued clock (left or right) with a corresponding button press as quickly as possible. Mean reaction times were calculated for correct responses separately for adapted and unadapted locations. Observers completed 100 trials in each of eight blocks.

rEsults
Adaptation to both 20-and 5-Hz flicker increased reaction time, but to the same degree (t(4) = −0.133, p = 0.901) (Figure 4). This indicates that the temporal decoupling caused by 20-Hz adaptation for our purposes, however, observers consistently reported the hand orientation to be 15 ms more advanced for clocks presented in regions adapted to 20 Hz than for clocks presented in unadapted regions (t(4) = 4.58, p = 0.01). Again, no such effect was observed for 5-Hz adaptation (t(4) = 1.24, p = 0.28). This result therefore corroborates the main finding reported in Experiment 1. However, temporal decoupling might simply reflect a change in the latency of the establishment of a neural representation of the clock hand after adaptation. It has previously been shown that the perceived timing of visual events can be affected by the latency with which neural processes start and finish (Arnold and Wilcock, 2007). To test this, in Experiment 4, observers carried out a reaction time task to establish whether adaptation influences the time at which a neural representation of the clock is formed. is not mediated by a change in the speed with which the neural representation of the hand on the clock-face is formed. Therefore, 20-Hz flicker induces a spatially localized shift in event time (the time at which the event is perceived to have occurred) without a corresponding shift in neural processing time.
The task in Experiment 4 did not require the same fine orientation judgment that was required in Experiment 1. One might therefore argue that it remains possible that a neural latency difference might be introduced by this fine orientation judgment. However, this is unlikely in light of the corroborating evidence from Experiment 3. In Experiment 3, the perceived orientation of a single clock hand was reported at the time of a cue. In this paradigm, changes in neural processing latency would not be expected to affect observer reports (Carlson et al., 2006;Hogendoorn et al., 2010). Nonetheless, the same dissociation in perceived timing was observed as in Experiment 1.

dIscussIon
We have shown that adaptation to high drift rates induces a spatially localized temporal shift in the perceptual stream of events. The shift is dissociated from processing time differences and is not a consequence of the dynamics of attentional allocation. The perceptual shift therefore reflects a shift in the representation of the time course of events itself, rather than latency differences in establishing visual representations in the brain. Following adaptation, synchronized, concurrent stimuli, with comparable processing latencies, can appear shifted in time.
The important implication of this finding is the converse: the same mechanism responsible for shifting concurrent stimuli out of alignment can shift the representations of events that have different neural processing latencies into alignment. Note that what shifts is the explicit representation of when (in the manner of a postmark on a letter), not the actual latencies with which particular neural processes complete (c.q. the day the letter arrives). Local representation of event time thereby provides a solution to the computational problem presented by the visual brain's asynchronous processing architecture.
The distinction between a representation of time and the time of representing has previously been noted by Dennett in his multiple drafts model of consciousness (Dennett and Kinsbourne, 1992). However, aside from an algorithm which attempts to maximize correspondence, no mechanism by which multiple perceptual streams might be brought into alignment was proposed. Our findings provide direct empirical support for temporal alignment based on explicit, local neural representations of event time. Furthermore, because the mechanism by which perceptual streams are brought into register is sensitive to adaptation, it follows that the local determination of event time is content dependent.
There are numerous schools of thought regarding the neural underpinnings of time in the brain. Multiple lines of evidence suggest that discrete neural mechanisms subserve time perception on microsecond (Grothe, 2003), millisecond (Pöppel, 1988), second (Treisman et al., 1990), and multi-second timescales (Czeisler et al., 1999). Importantly, although adaptation to high temporal frequency flicker has also been shown to affect perceived duration (Johnston et al., 2006), the time shift demonstrated in the experiments reported here is in the temporal alignment of visual events, rather than in the conscious perception of time per se. Accordingly, the conclusions which can be drawn from the present study pertain particularly to the resolution of the temporal binding problem resulting from the visual brain's asynchronous architecture.
We observe adaptation effects with test stimuli orthogonal to adapting stimuli, suggesting an early locus for the adaptation effect, possibly the lateral geniculate nucleus (LGN). The LGN has previously been put forward as a possible candidate for adaptation-induced temporal distortions (Johnston et al., 2006), based on the observations that such adaptation effects are observed at very high, cortically invisible temporal frequencies (Johnston et al., 2008), and that they are highly spatially localized (Ayhan et al., 2009). The present result therefore provides converging evidence that the LGN plays an important role in establishing a visual temporal signal.
It is perhaps striking that a running clock in an adapted location can continuously appear to run both ahead of and slower than a similar running clock in an unadapted location. In the physical world, the faster moving clock would overtake the other clock. However, the perceptual processing of velocity and position takes place with some degree of independence (c.q. the motion without movement illusion; Freeman and Adelson, 1991). Although at first sight paradoxical, there is therefore no contradiction in the finding that one clock hand can appear to move faster than another without overtaking it.
Altogether, these findings indicate that the time at which an event appears to occur can be locally decoupled from both the sequence of events in the physical world and the time course of the processing of these events in the brain. They indicate a process of explicit temporal realignment, and suggest that the neural computations involved are content-dependent and sensitive to high temporal frequency adaptation. This perspective also implies that different components of this neural system may need to learn and keep a record of their neural latencies, continuously recalibrating relative to other components of the network with which they need to cooperate. acknowlEdGmEnt This research was supported by a Netherlands Organization for Scientific Research (NWO Pionier) grant to Frans A.J. Verstraten.