Explicit behavioral detection of visual changes develops without their implicit neurophysiological detectability

All material supplied via JYX is protected by copyright and other intellectual property rights, and duplication or sale of all or part of any of the repository collections is not permitted, except that material may be duplicated by you for your research use or educational purposes in electronic or print form. You must obtain permission for any other use. Electronic or print copies may not be offered, whether for sale or otherwise to anyone who is not an authorised user. Explicit behavioral detection of visual changes develops without their implicit neurophysiological detectability Lyyra, Pessi; Wikgren, Jan; Ruusuvirta, Timo; Astikainen, Piia


INTRODUCTION
The human visual system is equipped with an automatic mechanism for detecting sudden changes in the environment. Abrupt stimulus appearance or motion captures attention in a bottomup way, even if these are not targets of visual search (Yantis and Jonides, 1984). However, this mechanism is prone to error if the load of attention is increased (inattentional blindness, see Mack and Rock, 1998) or if the change occurs simultaneously with some interruptive events. Massive changes may go easily unnoticed by this way, a phenomenon usually known as "change blindness" (for reviews, see Simons and Levin, 1997;Rensink, 2002). Change blindness can be observed as a failure to detect a considerable and otherwise easily detectable change when a flash of light, blink, saccade, or a large transient stimulus immediately precedes the change. The flicker paradigm, where a brief blank screen (or interstimulus interval, ISI) separates the two images across which a change is introduced, is a method that has firmly established the phenomenon of change blindness. It is also the most frequently used paradigm to induce the effect experimentally (Rensink et al., 1997).
Change blindness, as observed through failures in tasks to voluntarily (explicitly) report on the occurrences of the changes, is held to entirely reflect errors in top-down guided focal attention (Rensink et al., 1997;O'Regan and Noë, 2001). However, there is also accumulating behavioral (Hayhoe et al., 1998;Fernandez-Duque and Thornton, 2000;Hollingworth et al., 2001;Koivisto and Revonsuo, 2003;Laloyaux et al., 2006) and brain imaging (Beck et al., 2001;Huettel et al., 2001;Niedeggen et al., 2001;Pessoa and Ungerleider, 2004;Eimer and Mazza, 2005;Wascher, 2007, 2008;Kimura et al., 2008;Khittl et al., 2009;Lyyra et al., 2010) evidence of implicit change detection by the visual system during change blindness. It has been suggested that changes are initially detected implicitly to guide focal attention to the locus of the changes and, thereby, to eventually allow the explicit detection and identification of changes supported by focal attention (cf. Chun and Nakayama, 2000;Niedeggen et al., 2001). Similarly, there is both behavioral (Watanabe, 2003) and electrophysiological evidence that changes are implicitly localized before their identification already during change blindness Wascher, 2007, 2008) or "sensing" the presence of change without identification (Busch et al., 2010a,b) suggesting that change localization and identification follow each other sequentially in the same visual processing stream. This view regards implicit change detection as a pre-requisite for explicit change detection. However, the dissociation between neural mechanisms for the (explicit) identification and (implicit) localization of a visual object (Goodale and Milner, 1992) also leaves open a possibility that explicit and implicit modes of visual change detection may recruit distinct and mutually independent neural mechanisms. This possibility is also indicated by distinct ERP-markers found for explicit and implicit change detection Kimura et al., 2008. Furthermore, while explicit change detection is associated with attentional visual short-term memory with a limited capacity (e.g., Rensink, 2000), it is possible that the neural traces of implicit change detection is held to be solely supported by visual sensory memory with limited time span and reportability of its contents (Sperling, 1960). The observation of distinct markers cannot, however, yield direct evidence in support of independent functioning between explicit behavioral change detection and implicit change detection. To substantiate this possibility, it should be shown that explicit change detection could occur with and without the presence of implicit change detection.
To this end, we investigated whether explicit detection of visual changes, as indexed by voluntary behavioral responses to them, can only develop when implicit detection of these changes, as reflected by the visual mismatch negativity (vMMN) of eventrelated potentials (ERPs) of the brain (for a review, see Pazo-Alvarez et al., 2003;Czigler, 2007;Kimura et al., 2011), also takes place. vMMN is a neurophysiological index of pre-attentive detection of visual changes on the basis of visual sensory memory (e.g., Astikainen et al., 2008). Its pre-attentive nature is reflected by the fact that, unlike with explicit change detection, voluntary attention need not be directed toward the changes to detect them. Its implicit nature is, in turn, reflected by no need to be declaratively aware of the changes. vMMN is elicited at the posterior electrode sites at around 150-300 ms following change (e.g., Czigler, 2007). Although explicit detection of visual changes is relatively insensitive to the duration of the blank interval separating the changed and unchanged images , vMMN, in contrast, is known to diminish when ISI is prolonged (Astikainen et al., 2008), reflecting most probably the time span of visual sensory memory (Sperling, 1960). A behavioral index of explicit detection of visual changes and vMMN as an index of their implicit detection is, therefore, a perfect combination to investigate whether explicit detection could develop without implicit one. Previously, in change blindness studies using short intervals or change-occluding stimuli between images (30-100 ms, Wascher, 2007, 2008) change-related electrophysiological responses have been observed, but not in similar studies using longer intervals (500 ms: Eimer and Mazza, 2005; 900 ms: Henderson and Orbach, 2006). We thus tested the implicit detection of the changes as indexed by vMMN to an image with change as interspersed with a repeated image with no change. To vary the elicitability of vMMN, two durations of blank intervals (ISIs) between the images delivering the change were used: a short interval (100 ms ISI) expected to lead to vMMN, and a long interval (500 ms) unlikely to produce vMMN. ERPs were recorded in trials prior to the first explicitly detectable image with change. We hypothesized that if the explicit detection of the changes is conditional upon their implicit detection, the explicit detection of the changes should fail if vMMN fails to be generated by images with change (with longer ISIs). If, on the other hand, the explicit detection of the changes can bypass visual sensory memory and directly recruit attentional memory (Rensink et al., 1997), the explicit detection should emerge even when vMMN remains unobservable in preceding trials.

PARTICIPANTS
Twenty-two volunteers, 14 female and eight male with age distribution of 19-33 years (mean age 23.2 years, s.e. 4.03) participated in the study. All of them had normal or corrected-to-normal vision. Before the experimental treatment, the participants were informed about the nature and purpose of the study, and a written consent was obtained from them approved by the ethical committee of University of Jyväskylä. The study conforms to The Code of Ethics of the World Medical Association (Declaration of Helsinki).

STIMULI AND PROCEDURE
Participants viewed the stimuli while they were seated in a chair in a darkened room. The stimuli appeared on a 17 monitor at a distance of approximately one meter from the participant.
We used the same images of complex natural scenes as stimuli as in (Lyyra et al., 2010) (see Figure 1). There were a total of 10 original images and their modified versions with different kinds of changes, whose presentation order was randomized across participants. Changes across the images referred to appearances or disappearances of objects, or changes in their position or color. The direction of the changes (disappearance vs. appearances of objects) was counterbalanced across participants. Each of the images was 500 ms in duration and blank screen, i.e., ISI, of 100 ms in duration in one condition, and 500 ms in another condition separated consecutive image pairs. Ten stimulus blocks with different 50 image pairs were presented. In five of the blocks, a 100 ms ISI was used, and 500 ms ISI in the other five blocks. The order of the blocks was counterbalanced across the participants.
An image with change was infrequently and pseudo-randomly interspersed (p = 0.2) with its frequently presented pair without change (p = 0.8) (oddball condition). There were up to seven but no less than three image presentations with no change separating image presentations with change.
The participants were instructed to search for a change in the images in each stimulus sequence, and to report of noticing the change for the first time by pressing a button. They were also instructed to silently count the number of the remaining observable change occurrences in the sequence. Change detection performance was measured for each subject as the average number of presentation cycles of the changed images in one stimulus sequence required for the explicit behavioral report of change detection (button press). Occasional button presses to misses and false alarms (less than 3%) were used to distinguish them from change blindness, and successful detection was corrected on the basis of the counting task.

Electroencephalography (EEG) was recorded with Brain Vision
Recorder software (Brain Products GmbH, Munich, Germany) Frontiers in Human Neuroscience www.frontiersin.org from 29 channels of the international 10/20 system (FP1, FP2, Fz, F3, F4, F7, F8, FC1, FC2, FC5, FC6, Cz, C3, C4, CP1, CP2, CP5, CP6, T7, T8, Pz, P3, P4, P7, P8, Oz, O1, and O2) using an elastic cap (Electrocap) with Ag/AgCl electrodes. Average reference was used in the recordings and the extraction of ERPs. An electrode on the forehead was used as a ground electrode. The original EEG signals were sampled continuously at 1000 Hz, amplified and band-pass-filtered online with 0.1 to 100 Hz. The data were further processed with Brain Vision Analyzer 2.0 (Brain Products GmbH, Munich, Germany). The independent component analysis (ICA) function of the Brain Vision Analyzer 2.0 software was used to identify and correct for blinks and eye movements in the EEG. Channels with excessive muscular artifacts were removed. Data with voltage gradient over 80 µV/200 ms and maximum and minimum voltages exceeding ± 80 µV at any channel were marked as containing artifacts. The signals were off-line band-pass filtered from 0.1 to 30 Hz. Sweeps containing artifacts were discarded for all channels, the average rejection rate being 12.4% (0 to 65%, s.e. 20.89). Only subjects with more than 15 sweeps in the averaging after the artifact rejection were included in the ERP-averages (mean number of sweeps 57.7 for the 100 ms ISI condition and 74.1 for the 500 ms ISI condition). The rejection was done for each condition and all subjects, and this procedure lead to 18 subjects in each ISI condition (100 and 500 ms) that were, however, not the same for both conditions. As we were interested in implicit change processing, only data during change blindness, from the time period before the button press in the stimulus sequence were used for the ERP analysis. We excluded one presentation of change immediately preceding the button press not to confound implicit change processing with initial explicit change detection. For the ERP analyses, sweeps of the electrophysiological responses to the changed images and to the unchanged images immediately preceding them were picked in a time window of 100 ms before and 300 ms after stimulus onset. These responses were averaged Frontiers in Human Neuroscience www.frontiersin.org across each subject. This way the sweeps were from responses to image pairs close to each other in the stimulus sequence, and a similar amount of responses to changed and original images included in the averages. A 100 ms pre-stimulus period served as a baseline, against which the ERPs were corrected in both conditions. A prominent differential ERP for the unnoticed changes of negative polarity is observable in the grand averaged waveforms (see Figures 2 and 3) in the posterior and posterio-temporal electrode sites (P7, P8, Oz, O1, O2), especially in the condition of 100 ms ISI. These differential ERPs resembled in their latency, scalp topography, and stimulus and attentive conditions the visual analog of the mismatch negativity (MMN) of ERPs (Pazo-Alvarez et al., 2003;Czigler, 2007). For assessing response amplitudes, a time window of 200-260 ms (for vMMN, see e.g., Astikainen et al., 2008;Pazo-Alvarez et al., 2003; for change blindness studies, see Wascher, 2007, 2008;Kimura et al., 2008) was selected for which mean values from each electrode were extracted. The resultant values were submitted to a repeated measures multivariate analysis of variance (MANOVA), Electrode site (P7, P8, Oz, O1, O2) and Stimulus type (No change, Change) as within-subject factors in each ISI condition (100 ms and 500 ms). This is due to two things. First, a response to the presentation of the blank screen is visible in the pre-stimulus period of the ERPs of the 100 ms ISI condition suppressing their amplitudes (see Figure 3), which makes the conditions differ slightly in functional terms with respect to each other. Second, the subjects are only partly same in both conditions, so they cannot be incorporated in the same statistical model. Only results involving Change as a factor are reported here.
Explicit change detection performance was measured by the average number of presentations of images containing a change in a stimulus sequence required for explicit report of initial change detection (button press). The performance in the distinct ISI-conditions was compared by paired t-tests.    (Figures 2 and 3).

DISCUSSION
We assessed explicit detection, as reflected by voluntary behavioral responses, and implicit detection, as reflected by ERPs of the brain, of a rare pair of images with change as interspersed with a repeated pair of images with no change. There was a blank interval of either 100 ms or 500 ms between the paired images across which the changes were produced. Both intervals allowed the explicit detection of the images with changes. The detection developed slightly faster across the trials with the 100 ms (after presentation of 11 changed images) than with the 500 ms ISI (after presentation of 14 changed images). Prior their explicit detectability, i.e., during change blindness, images with changes were detected implicitly, as reflected by the posterior electrical brain responses 200-260 ms after the change onset (vMMN) but only with the 100 ms ISI.
With the shorter ISI of 100 ms, we found vMMN to unnoticed images with change. Similarly, traces of implicit change detection, as reflected by differential electrophysiological responses at 200-300 ms post-stimulus, have been observed by Schankin and Wascher in a couple of studies using a short ISI (30 ms) or transient change-occluding stimuli (50 ms in Schankin and Wascher, 2007;100 ms in Schankin and Wascher, 2008;cf. Fernandez-Duque et al., 2003;Kimura et al., 2008). More specifically, Schankin and Wascher observed a contralateral modulation of the N2pc-component at posterio-temporal electrode sites 220-260 ms (Schankin and Wascher, 2007), and 245-295 ms after stimulus onset (Schankin and Wascher, 2008) elicited by changes presented in the right or left visual hemifield, reflecting the location of the unnoticed change in the display.
Despite the longer ISI of 500 ms, in comparison to 100 ms ISI, only slightly delayed the explicit detectability of images with changes, we failed to observe a robust vMMN to these images when 500 ms ISI was applied. This finding was expected since it is known that vMMN is dependent on active sensory memory trace of the standards (for the memory comparison hypothesis of MMN, see Näätänen, 1992). Indeed, vMMN to even perceptually simpler changes (in orientation of a bar) in repeated visual stimuli has been found to disappear along with the lengthening of blank ISIs from 400 ms to 1100 ms (Astikainen et al., 2008). Previous studies with longer ISIs between the images delivering the change (900 ms in Henderson and Orbach, 2006;500 ms in Eimer and Mazza, 2005) have failed to observe vMMN or other deflection in the N2-latency in response to unnoticed images with change. Indeed, Schankin and Wascher (2007) associated these negative findings with the extinction of the memory trace of the first image of a pair and, thereby, with the crash down of the whole process of comparison between the second and the first image of a pair. This account could well also explain our negative finding with the longer, 500 ms ISI and the positive one with the 100 ms ISI.
Our finding of the explicit detection of implicitly undetectable images with change questions the view that implicit change detection is needed for the development of the ability to detect changes explicitly. Furthermore, given the previous findings of implicit without explicit change detection Eimer and Mazza, 2005;Wascher, 2007, 2008;Kimura et al., 2008;Khittl et al., 2009;Lyyra et al., 2010), our finding of an opposite dissociation may even suggest that the implicit and explicit modes of change detection may recruit functionally independent neural mechanisms.
vMMN to images with change with the 100 ms intervals between the images of a pair may also have been modulated by other temporally overlapping ERP components linked to specific aspects of change, such as its target status or mere occurring probability (vMMN and N2pb, see Luck, Hillyard, 1994;Henderson and Orbach, 2006), its spatial location (N2pc, see Eimer and Mazza, 2005;Wascher, 2007, 2008), or visual awareness of changes (visual awareness negativity, VAN, Koivisto and Revonsuo, 2003;Busch et al., 2010a,b). The N2pb has been associated with explicitly detected changes (Luck, Hillyard, 1994). Like N2pb, VAN has been linked to explicit change detection (Koivisto and Revonsuo, 2003) and also to "sensing" of presence of changes before their detection (Busch et al., 2010a,b). In the present study, a possibility of explicit detection of images with change was minimized. The threshold for the participants' responses to these images was set as low as possible. No identification of the changes was required and we excluded the last explicitly undetectable change in stimulus sequence from analyses and ERP-extraction, and it is for these change presentations that sensing or localization has been found to occur (Niedeggen et al., 2001;Mitroff et al., 2002). Accordingly, the N2pc component related to spatial attention has been observed in change blindness experiments specifically in the S1-S2 paradigm during change blindness Wascher, 2007, 2008) and "sensing" of the changes (Busch et al., 2010a,b). For these reasons, we consider that the differential ERP-response we observed for unnoticed changes was most likely vMMN in particular. Therefore, our study also leaves open the possibility that a shift of spatial attention to the changes is necessary, even if not sufficient, for explicit identification (see Watanabe, 2003;Wascher, 2007, 2008;Busch et al., 2010a,b), and that vMMN reflects another type of implicit change detection that is not Frontiers in Human Neuroscience www.frontiersin.org even necessary for explicit change detection. However, if it is unlikely that the vMMN we observed reflects localization of the changes or sensing of changes, it is not clear what kind of processing the vMMN we observed reflects in our study. Implicit change detection may only consist of registration of the features of the changing objects (Lyyra et al., 2010), although there is also interesting behavioral evidence suggesting that information about the identity of the changing objects or their features could be implicitly processed (Angelone et al., 2003). vMMN itself is a component that is elicited by single features but also for complexes of features (e.g., Müller et al., 2010). Moreover, it is even possible that vMMN does not reflect a unitary process throughout the period of change blindness but rather progressing (implicit) change detection (cf. Rensink, 2002). A study showing, e.g., distinct brain responses to implicit detection of random groups of single features versus organized combinations of the same features could shed more light on the issue of how far the changes are processed implicitly. Some theorists have challenged the existence of implicit change detection (Mitroff et al., 2002). Their criticism lies in the reliability of behavioral measures of change awareness. Unconfidence of the subjects about the content of their perceptual experience and inconservative response criterions of explicit awareness may allow residual explicit awareness to affect indirect behavioral measures (see, however, Fernandez-Duque and Thornton, 2003;Laloyaux et al., 2006). In the present study, we used a fairly liberal response criterion, that is, button press at the first explicitly detectable occurrence of change not requiring identification. Nevertheless, even if we excluded the change occurrence immediately preceding explicit change detection from ERP-extraction, the subjects might have been marginally aware of or "sensed" the change even at some point before the button press. Therefore, the implicit nature of the detection stands in our study in the failure of the declarative consciousness to access the changes to allow the changes to be reported behaviorally. It is thereby possible that implicit changes may have made a difference in the phenomenal subjective experience of the subjects, even though they have not been able to access this information to report on it. Nevertheless, we think that our interpretation of the independence of the pre-attentive vMMN from explicit behavioral detection requiring focal attention holds.
Manipulating the duration of the blank interval led to a couple of incompatibilities with regard to the ERPs between the two ISI conditions. The amplitudes in the ISI100 condition are decreased and their polarity even partly reversed due to the ERP-response to the blank interval (cf. Wascher, 2007, 2008) as compared to the ISI500 condition in which the response to the blank screen has dissipated. We see this incompatibility as no threat to the present results, since the two ISI conditions were not compared to each other, only ERPs to changed and unchanged images within each ISI condition. For each ISI condition, the stimulus conditions are identical, and the differences can thus only be due to the unnoticed changes in the stimuli. Also because of surprisingly good change detection performance together with artifact rejection, the sweep numbers for ERP extraction were relatively low for some participants in either ISI condition. However, most sweep numbers were close to the average (57.7 for the 100 ms ISI condition and 74.1 for the 500 ms ISI condition), and we used methods such as ICA in ocular correction to enhance the signal to noise ratio. Moreover, the trial numbers were lower for the ISI100 condition than the ISI500 condition. Low numbers of trials rather occludes potential effects, but still the effect of change in ERPs reached significance in the ISI100 condition, and the negative result in ISI500 condition was not due to its poorer signal to noise ratio. To further investigate the reliability of our results in this respect, we reanalyzed the results using only the data on the 12 participants with more than 30 trials in ERP-averages in both conditions. The results were similar to the original ones; the ERPs differed significantly in response to unnoticed changes in the ISI100 condition (p < 0.05), but not in the ISI500 condition (p = 0.09).
In the ERPs of the anterior electrode sites, there is a positive change-related amplitude difference visible at the same latency as the vMMN (see Figure 3). Positive differential ERPs have been reported accompanying vMMN in a number of studies (see Czigler, 2007 for a review). However, a similar ANOVA for the frontal electrode sites (F3, F4, Fz, FC1, FC2, Cz) as for the posterior electrode sites did not reveal any significant change-related amplitude differences.
The development of explicit change as a function of repeated occurrences of change without the guidance of implicit change detection would be in line with the view that a major factor in overcoming change blindness is top-down visual search strategy (Sampanes et al., 2008). In future studies, selectively affecting this strategy (attention to single objects as opposed to sets of objects) could provide a means to further explore on the aspects that dissociate explicit from implicit change detection, the latter of which rather utilizes elementary visual attributes. Nevertheless, we cannot exclude the possibility that some aspects of implicitly processed information about changes could affect their explicit detection. For example, it has been shown that socially relevant changes are detected faster than neutral ones (David et al., 2006) and that information about the emotional content of visual stimuli could be processed implicitly (Pasley et al., 2004). ERPs related to emotional processing could be used to investigate implicit processing of emotional contents of the changes, and their relation to change detection performance in the change blindness condition.
In sum, our results suggest that, despite implicit detection of visual changes remains unobservable, the changes still can build up an ability to detect the changes explicitly. Namely, despite the implicit detection of the visual changes fails due to long stimulation intervals beyond the temporal span of a memory system involved, the visual information, on which such detection is based, may still be successfully stored for these periods and used to prepare the visual system for the explicit detection of changes. Together with previous findings of implicit despite the absence of explicit change detection, our finding of the opposite dissociation suggests that implicit and explicit modes of change detection may recruit functionally distinct memory-based mechanisms of the brain.