Original Research ARTICLE
Relational dynamics in perception: impacts on trial-to-trial variation
- Network Biology Research Laboratories, Technion – Israel Institute of Technology, Haifa, Israel
We show that trial-to-trial variability in sensory detection of a weak visual stimulus is dramatically diminished when rather than presenting a fixed stimulus contrast, fluctuations in a subject’s judgment are matched by fluctuations in stimulus contrast. This attenuation of fluctuations does not involve a change in the subject’s psychometric function. The result is consistent with the interpretation of trial-to-trial variability in this sensory detection task being a high-level meta-cognitive control process that explores for something that our brains are so used to: subject–object relational dynamics.
Trial-to-trial variation in responses to repeated presentations of the same weak sensory object is noticeable in practically every cognitive modality. These fluctuations, which have been labeled “internal,” “unexplained,” or “inherent” noise, are correlated over extended timescales (Werthheimer, 1953; Gilden, 2001; Monto et al., 2008) and inversely related to the degree of stimulus-determination (Conklin and Sampson, 1956)1. Over the years since the inception of psychophysics, there has been a shift in how the source of trial-to-trial variation at threshold is explained. At present-time, the concept of noisy neural response dynamics that “poses a fundamental problem for information processing” is dominant (Faisal et al., 2008).
Yet there is something very un-natural in the way traditional psychophysical studies of sensory detection – studies that expose extensive trial-to-trial variation at threshold – are set up. In real-life situations, when encountering a weak sensory stimulus that deserves attention, we try to “do something about it.” Consider the set of operations performed by the average man over 50 confronted with a barely detectable printed text: tilting the page, exposing it to enhanced light conditions, etc. The stimulus itself becomes dynamic. If the barely detected stimulus originates from another subject, we (for instance) might lean forward or ask that other person to raise his voice or to present the object in a more favorable manner. Again, the stimulus itself becomes dynamic. Indeed, in real-life situation, our attempts to “do something about” the barely detected stimulus impacts on the stimulus dynamics, although not necessarily on our capacity to detect it. Thus, natural perception involves an expectation of the perceiver for an ongoing coupling between his actions and the threshold-level stimulus dynamics. In that sense, natural perception is relational. This is real-life, but in standard psychophysical experiments the situation is different: in these experiments much effort is invested by the experimentalist to control the conditions so that the threshold-level stimulus remains static. The subject might actively explore various features of the stimulus (“active sensing”), but any given stimulus feature in these standard psychophysical designs, remains the same regardless of the subject’s behavior. Hence no feedback between the subject’s actions and the stimulus dynamics is involved, and perception becomes non-relational.
In view of the above, and encouraged by old and recent analyses that reveal rich temporal structure and non-independence in response fluctuations at threshold over extended timescales (Conklin and Sampson, 1956; Gilden, 2001; Monto et al., 2008; Marom, 2010, and references therein), we turned to examine the possibility that trial-to-trial variation in responses to repeated presentations of the same weak sensory object do not reflect an inherent noise that constrains sensory acuity and information processing. Rather, we hypothesize that most of the observed variability in responses to weak stimuli is due to an active cognitive exploratory process, seeking for a coupling between the stimulus dynamics and subject’s behavior. To test this hypothesis we have used a generic feedback loop control algorithm, endowing a visual stimulus in a detection task the capacity to on-line match its contrast to the subject’s performance, while “clamping” the performance at a predefined (mostly 0.5) probability of detection. We show that once such relations are established (i.e., as long as the control algorithm is active), trial-to-trial variability is dramatically diminished, breaking the apparent limits of inherent noise, while keeping detection threshold and sensitivity (as reflected in the psychometric function) unchanged. This result points at the possibility of trial-to-trial variability in sensory detection of weak stimuli being a high-level meta-cognitive control process that explores for something that life trained us to expect: subject–object relational (or, coupled) dynamics.
Materials and Methods
All the experiments and their analyses were performed within a Wolfram’s Mathematica 7.0 environment; the software package is available on request from Shimon Marom.
Psychophysical Detection Task
Fourteen healthy volunteers (six females), graduate students and post-docs at the age of 27–40 year, were the subjects of this study. Unless indicated otherwise, the basic visual detection task is as follows (see Figure 1): A random 500 × 500 background raster of black and white pixels, occupying 135 mm × 135 mm, was presented in the center of a flat Apple 24′′ screen. A single session was composed of 500 presentation trials of the raster, randomized in each trial. The raster remained on screen for half a second in each trial. A smaller foreground raster of 70 × 70 randomized grayscale pixels was embedded in the background raster area. Within the foreground, the gray-level (denoted x) of the i-th pixel in the n-th trial was determined by a uniformly distributed random number (0 ≤ ri,n ≤ 1) such that
where Cn, referred to as “contrast,” was calculated on a trial-by-trial basis as described in the next section. With this procedure the general pattern of background black and white scatter is present also within the foreground, while the range [0.5, Cn] of foreground grayscale serves as the control variable. Of the 500 trials in a session, 50 randomly introduced sham trials did not include any foreground object. The position of the foreground object in each trial was randomized. After a trial, subjects were asked to press one of two keys, signifying whether they detected or not the foreground object. Each trial immediately followed the subject’s response to the preceding trial; no time limit was set for the subject to produce an answer.
Figure 1. Detection task. Three examples of single trial presentations: Background raster only (left panel), a barely detectable foreground object indicated in the top-left field of the middle panel, and an obvious foreground object (right panel). A circle at the center of the image appeared in all trials of all sessions; the subjects were instructed to try to fixate on that circle at the beginning of each trial.
Stimulus Control and Estimation of Detection Probability
The experimental design was adopted from a recently introduced Response Clamp methodology for analysis of neural fluctuations (Wallach et al., 2011), with modifications that enabled its application to the present behavioral setting2.
Response probability was estimated on-line as follows: Let sn be an indicator function, so that sn = 1 if the subject detected the n-th foreground stimulus and sn = 0 otherwise. We define π(n) as the probability of the subject to detect a foreground stimulus at trial n. We can estimate this probability using all past responses by integrating them with an exponential kernel,
where τ is the kernel’s decay-constant. To compute this on-line, we used the recursive formula:
setting τ = 10 trials, and
A proportional–integral–derivative (PID) controller was realized in Wolfram’s Mathematica 7.0 environment. The input to the controller is the error signal,
where and are the desired and actual detection probabilities (calculated as explained below) at the n-th trial, respectively. The output of the controller is generally composed of three expressions,
where gP, gI, and gD are the proportional, integral, and derivative gains, respectively; gP was set to 1.0, gI to 0.02, and gD to either 0.02 or 0 (with no appreciable effect). Our experience has been that the integral component is essential: omitting that component results in stabilization around an arbitrary and time-variant value; for an extensive discussion on this issue see Figure 3D in Wallach et al. (2011), a methodology paper that introduces the concept of Response Clamp in the neural context.
Finally, the contrast Cn equals the controller’s output plus some baseline:
To quantify correlations in a given sequence of binary responses, the latter was divided to non-overlapping bins of size K. A count sequence ZK was constructed by registering the number of positive responses (i.e., trials in which the visual object was detected) within each of the bins. This process was repeated for all bin sizes used. For each count sequence thus obtained, the Fano-factor (variance to mean ratio) is defined as F(K) = var(ZK)/mean(ZK) and plotted as a function of the bin size, K. As extensively established by Teich et al. (1997), this quantity provides a way to extract correlations in a sequence of binary events: An uncorrelated white process (e.g., Homogeneous Poisson Process) yields a constant F(K) for all bin sizes; regularity (e.g., low-jitter oscillation) would be reflected in a decreasing F(K) within the relevant range of bin sizes; and, finally, a complex temporal structure, exhibiting long-term correlations (e.g., fractal point process), would produce an ever increasing F(K), following a power-law relation to bin size.
Closed-Loop, Replay, and Fixed Contrast Modes
In the basic design, each subject was exposed to three experimental sessions denoted closed-loop, replay, and fixed. The first session was always a closed-loop session, whereas the second and third were replay and fixed sessions, introduced in an alternating order to different subjects. A 10-min break was given after the first and second sessions.
In the closed-loop session, the desired response probability () was kept constant ( unless indicated otherwise in the main text) and the control algorithm operated as explained above, updating the contrast (Cn) of the foreground object from one trial to the next based on the error signal (en). The series of 450 Cn values produced in this closed-loop session (500 trials minus the 50 sham trials), served for the generation of both dynamically and statistically identical foreground objects in the replay session. Thus, in the replay session the control algorithm was disconnected, yet we were able to record the responses of the subject to exactly the same series of contrasts presented in the closed-loop session, but now in an open-loop context, detached from the trial-by-trial coupled observer’s–observed dynamics. In the fixed session, the average contrast calculated from the series of above mentioned contrasts was used for all presentations, thus omitting stimulus variance altogether. This fixed session allowed us to estimate the impact of stimulus fluctuations on response dynamics.
The nature of the detection task is demonstrated in Figure 1: The left panel shows a background raster only. The middle panel shows a barely detectable foreground object (indicated in the top-left field). The righthand panel demonstrates an obvious foreground object. The probability of false positive detection, calculated from responses of all subjects to the 50 sham trials was negligible (0.017, SD = 0.029, n = 8), indicating that the subjects did not tend to report detection when they did not really see something. The average response time was around 1 s per trial, slightly longer in the closed-loop session (1.03 s, SD = 1.3) compared to replay and fixed sessions (0.89 s, SD = 1.6 and 0.81 s, SD = 1.0, respectively). Response time distributions in all three sessions had a long right tail (coefficient of skewness: 11, 17, and 19, respectively).
The main observation is shown in Figure 2, where the responses of one subject (left column) and the group of eight subjects (right column) that were tested in closed-loop (dark blue, top row), replay (purple, middle row), and fixed (a kind of yellow, bottom row) sessions are plotted. Let us start with Figure 2A, describing the results obtained in the closed-loop session of one individual. The desired detection probability (see Materials and Methods) was set to 0.5, and the control algorithm updated the contrast Cn, trial-by-trial, as indicated by black dots (righthand y-axis of Figure 2A). The resulting estimated detection probability ( dark blue, lefthand y-axis) of that individual gradually approached the desired value, albeit fluctuating about it. Also note the expected anti-correlation of Cn and Figure 2B shows the average performance (and SD) of all eight subjects that participated in such a closed-loop session, showing that the controller converges within ca. 100 trials, and succeeds in “clamping” the detection probability at around 0.5, as preset.
Figure 2. Detection probability in closed-loop, replay, and fixed sessions. Data obtained from an experiment on one individual (left column) and the summation of observations from the eight different subjects that were tested in this protocol (right column). In all cases, the fluctuations around mean detection probability are significantly smaller in the closed-loop session. The initial decline in detection probability, which is most apparent in the closed-loop and replay sessions, reflects the initial setting of both and Cn to 0.5. Error bars in the righthand column depict SD across all subjects.
Figure 2C shows the performance of the same individual whose data are shown in Figure 2A, but now in the replay session, where the Cn series obtained in the closed-loop session (black dots) is “replayed,” regardless of the subject’s responses. Under these conditions the control algorithm is shut down and stimulus contrast is completely decoupled from the subject’s behavior. Note the emergence of large slow fluctuations around the preset 0.5 detection probability; despite the fact that the stimulus series is practically identical to that shown in Figure 2A, the performance in Figure 2C is very different. Figure 2D shows the average performance (and SD) of all eight subjects that participated in the replay session.
And finally, as demonstrated in Figure 2E, when the average Cn of the subject – whose data is shown in Figures 2A,C – is used in the fixed session, where all 450 trials have identical contrast (black line of Figure 2E), large slow fluctuations and drift (which might reflect perceptual fatigue) are observed. Figure 2F shows the average performance (and SD) of all eight subjects that participated in these fixed session.
As mentioned in the introductory comments to this study, psychophysical data exhibits correlations over extended timescales in responses to near-threshold stimuli (Werthheimer, 1953; Gilden, 2001; Monto et al., 2008). Such long-term correlations are apparent in the open-loop data of Figures 2C,E. Figures 3A,B demonstrates one way to quantify the statistics of these correlations, using Fano-factor analyses (see Materials and Methods). It shows that both for the case of an individual subject (Figure 3A) as well as the lumped data of all subjects (Figure 3B), response fluctuations in the open-loop modes are indeed dominated by long-range correlations, whereas responses in the closed-loop (controlled) mode exhibit regularity, reflected as a decreasing Fano-factor within the observed bin sizes. Autocorrelation or, equivalently, power spectrum analyses also show substantial lessening of slow frequency components in the closed-loop mode (data not shown).
Figure 3. Temporal structure and psychometrics. (A,B) Fano-factor analysis (see Materials and Methods) for a single subject (A) and the average over eight subjects (B). Long-term correlations in the temporal fluctuations of responses in both the replay and fixed modes (solid lines, purple, and yellow, respectively) are revealed. A solid black line in (B) depicts power-law relations (exponent power of 0.4), for reference. Under the closed-loop conditions (solid dark blue line), the Fano-factor decreases as a function of bin size, indicative of a highly regular process. Random shuffling of the response sequences eliminates all temporal correlations, yielding (as expected) a constant Fano-factor (dashed lines). (C) Histograms of detection probability (), calculated from data of all subjects, in closed-loop (top), replay (middle), and fixed session. (D) Both threshold and sensitivity are practically identical in closed-loop (dark blue) and replay (purple) sessions. The curves were calculated by averaging the responses (1’s and 0’s), for each of the eight subjects, in different contrast (Cn) bins; bin size = 0.025. A minimum of five occurrences of a given contrast per bin was set as a requirement for inclusion in the calculation. The Average Response (y-axis) and its SD among subjects for each contrast bin are shown in the plot. Note that average response thus calculated is not the same quantity as detection probability (the latter takes into account the temporal order of responses).
The group statistics of Figures 2B,D,F are summarized in Figure 3C. Clearly, the best performance is obtained when relational dynamics are allowed between (the performance of the observer) and Cn (the contrast series). One possible explanation to this result is that under these different experimental conditions there is a change in the sensitivity of the subject to the stimulus. Figure 3D shows the psychometric functions for closed-loop and replay sessions, calculated by averaging the responses (1s and 0s), for each of the eight subjects, in different contrast (Cn) bins. (Note that average response thus calculated is not the same quantity as detection probability the latter takes into account the temporal order of responses.) Clearly, these psychometric functions show that both threshold and sensitivity extracted from the responses of all the subjects, are practically identical in closed-loop and replay. However, the richness of the dynamics and the marked differences between closed-loop and replay modes seen in Figures 2 and 3B, are practically averaged out when the data are collapsed to standard psychometric functions of the kind shown in Figure 3D.
The importance of instantaneous coupling between the observer’s behavior and the stimulus dynamics is demonstrated in Figure 4A, where a closed-loop mode is instantly switched to a fixed mode, by disconnecting the controller and using a constant contrast value (average of the Cn series over a time segment depicted by a black bar). Figure 4A shows data of a single subject, in three different preset values (0.25, 0.5, and 0.75). As soon as the coupling of observer’s–observed dynamics is disconnected, the variance of detection probability markedly increases and slow correlations seem to emerge, at all tested. As shown in the averaged histograms of Figure 4B, within the fixed phase of the experiment of Figure 4A the detection probabilities are skewed, exhibiting a “binary” preference toward 1 or 0 (for the cases of and respectively). This preference stands in contrast to the symmetric case of (e.g., Figure 3C).
Figure 4. Responses of subjects to sudden transition from closed-loop to fixed modes. (A) A closed-loop mode is instantly switched to a fixed mode, by disconnecting the controller and using a constant contrast value (average of the Cn series over a time segment depicted by a black bar). Data obtained from a single subject in three different preset values (0.25, 0.5, and 0.75). (B) Average distributions of detection probability (n = 4 subjects) for and sessions as shown in (A); obtained separately from the closed-loop (blue) and fixed phases.
Two basic observations are presented here. The first is that trial-to-trial variability in sensory detection of a weak visual stimulus is dramatically diminished when rather than presenting a stimulus contrast that is independent of the subject’s ongoing actions, the fluctuations in stimulus contrast were matched to the fluctuations in a subject’s judgment. Clearly, this result reaffirms that trial-to-trial fluctuations are not “noise” in the strict sense of being independent of each other. Moreover, the significant difference, between features of fluctuations measured when dynamic observer–observed relations exist, and those measured in the absence of such coupled dynamics, calls for re-examination of the way psychophysical experiments are conducted. Indeed, measuring temporal fluctuations of a psychophysical function under open-loop conditions, where there is no relation between subject performances and sensory object contrast dynamics, is a most un-natural setting. Here we implemented an adaptive algorithm (PID) borrowed from control theory in order to couple the observer–observed dynamics. The PID control algorithm has theoretical advantages in the present context by being simple, and probably the most extensively used and theoretically studied in control engineering (Levine, 1996). Having said that, there exist many other adaptive psychophysical procedures (Treutwein, 1995) that are, actually, in use when experimentalists attempt to identify points of interest on psychometric functions. We propose to substantially extend their use in order to expose the dynamics of perception under more natural experimental conditions. The second basic observation is that the above diminishing of trial-to-trial fluctuations by coupling between observer–observed dynamics, is not accompanied by a change in sensory sensitivity to the input. Taken together, the two basic observations suggest that trial-to-trial variability in sensory detection of weak stimuli might reflect a high-level control process.
As pointed out by Werthheimer (1953), trial-to-trial variation at threshold was generally attributed, in the early days of psychophysics, to uncontrolled experimental conditions, with the assumption that the subject is stable. Response fluctuations, however, were soon shown to be non-independent (Verplanck et al., 1952); that is – successive responses to repeated presentations of the same threshold stimulus (in auditory, visual, and somatosensory modalities) are correlated over timescales ranging from seconds to days (Werthheimer, 1953). These long-range trial-to-trial response correlations were then interpreted as reflecting modulation of an a priori detection probability by recent subjective experience, a meta-cognitive guessing process that is active where there is no possibility for stimulus-determination; the assumption being that “guesses are more likely to be influenced by preceding responses (success and failures) than are sensory judgments” (Conklin and Sampson, 1956). Later years brought with them a more reductionistic focus on neural sources of “noise” and “short-term plasticity” that may account for observed trial-to-trial response variability (Faisal et al., 2008). Viewed from this historical angle, the results presented here pull the pendulum back to the meta-cognitive pole, offering an interpretation according to which – when facing a weak stimulus – subjects vary their response patterns, seeking to establish (predictive? see Rosen, 1985; Creutzig et al., 2009) relations between their actions and the dynamics of stimulus features. This interpretation, in its broader sense, goes far beyond psychophysics of weak stimulus detection; it touches upon what psychologists try to tell us over the past 50 years on the developing mind (e.g., Stern, 1985), when we care to listen.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
- ^Stimulus-determination is a term used by Conklin and Sampson (1956) as well as others to indicate the extent to which a subject’s response is determined by features of the stimulus itself, rather than stimulus-independent factors.
- ^For an extensive discussion of parameter choices, the reader is encouraged to consult Wallach et al. (2011).
Keywords: psychophysics, threshold, fluctuations, control
Citation: Marom S and Wallach A (2011) Relational dynamics in perception: impacts on trial-to-trial variation. Front. Comput. Neurosci. 5:16. doi: 10.3389/fncom.2011.00016
Received: 11 January 2011; Accepted: 26 March 2011;
Published online: 09 April 2011.
Edited by:David Hansel, University of Paris, France
Reviewed by:Paolo Del Giudice, Italian National Institute of Health, Italy
Emilio Salinas, Wake Forest University, USA
Copyright: © 2011 Marom and Wallach. This is an open-access article subject to a non-exclusive license between the authors and Frontiers Media SA, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and other Frontiers conditions are complied with.
*Correspondence: Shimon Marom, Network Biology Research Laboratories, Technion – Israel Institute of Technology, Technion, Haifa 32000, Israel. e-mail: firstname.lastname@example.org