Relational Dynamics in Perception: Impacts on Trial-to-trial Variation

We show that trial-to-trial variability in sensory detection of a weak visual stimulus is dramatically diminished when rather than presenting a fixed stimulus contrast, fluctuations in a subject's judgment are matched by fluctuations in stimulus contrast. This attenuation of fluctuations does not involve a change in the subject's psychometric function. The result is consistent with the interpretation of trial-to-trial variability in this sensory detection task being a high-level meta-cognitive control process that explores for something that our brains are so used to: subject–object relational dynamics.


IntroductIon
Trial-to-trial variation in responses to repeated presentations of the same weak sensory object is noticeable in practically every cognitive modality.These fluctuations, which have been labeled "internal," "unexplained," or "inherent" noise, are correlated over extended timescales (Werthheimer, 1953;Gilden, 2001;Monto et al., 2008) and inversely related to the degree of stimulus-determination (Conklin and Sampson, 1956) 1 .Over the years since the inception of psychophysics, there has been a shift in how the source of trial-to-trial variation at threshold is explained.At present-time, the concept of noisy neural response dynamics that "poses a fundamental problem for information processing" is dominant (Faisal et al., 2008).
Yet there is something very un-natural in the way traditional psychophysical studies of sensory detection -studies that expose extensive trial-to-trial variation at threshold -are set up.In reallife situations, when encountering a weak sensory stimulus that deserves attention, we try to "do something about it."Consider the set of operations performed by the average man over 50 confronted with a barely detectable printed text: tilting the page, exposing it to enhanced light conditions, etc.The stimulus itself becomes dynamic.If the barely detected stimulus originates from another subject, we (for instance) might lean forward or ask that other person to raise his voice or to present the object in a more favorable manner.Again, the stimulus itself becomes dynamic.Indeed, in real-life situation, our attempts to "do something about" the barely detected stimulus impacts on the stimulus dynamics, although not necessarily on our capacity to detect it.Thus, natural perception involves an expectation of the perceiver for an ongoing coupling between his actions and the threshold-level stimulus dynamics.

MaterIals and Methods
All the experiments and their analyses were performed within a Wolfram's Mathematica 7.0 environment; the software package is available on request from Shimon Marom.

PsychoPhysIcal detectIon task
Fourteen healthy volunteers (six females), graduate students and post-docs at the age of 27-40 year, were the subjects of this study.Unless indicated otherwise, the basic visual detection task is as follows (see Figure 1): A random 500 × 500 background raster of black and white pixels, occupying 135 mm × 135 mm, was presented in the center of a flat Apple 24′′ screen.A single session was composed of 500 presentation trials of the raster, randomized in each trial.The raster remained on screen for half a second in each trial.A smaller foreground raster of 70 × 70 randomized grayscale pixels was embedded in the background raster area.Within the foreground, the graylevel (denoted x) of the i-th pixel in the n-th trial was determined by a uniformly distributed random number (0 ≤ r i,n ≤ 1) such that where C n , referred to as "contrast," was calculated on a trial-bytrial basis as described in the next section.With this procedure the general pattern of background black and white scatter is present also within the foreground, while the range [0.5, C n ] of foreground grayscale serves as the control variable.Of the 500 trials in a session, 50 randomly introduced sham trials did not include any foreground object.The position of the foreground object in each trial was randomized.After a trial, subjects were asked to press one of two keys, signifying whether they detected or not the foreground object.Each trial immediately followed the subject's response to the preceding trial; no time limit was set for the subject to produce an answer.

stIMulus control and estIMatIon of detectIon ProbabIlIty
The experimental design was adopted from a recently introduced Response Clamp methodology for analysis of neural fluctuations (Wallach et al., 2011), with modifications that enabled its application to the present behavioral setting2 .
Response probability was estimated on-line as follows: Let s n be an indicator function, so that s n = 1 if the subject detected the n-th foreground stimulus and s n = 0 otherwise.We define π(n) as the probability of the subject to detect a foreground stimulus at trial n.We can estimate this probability using all past responses { } , by integrating them with an exponential kernel, where t is the kernel's decay-constant.To compute this on-line, we used the recursive formula: where g P , g I , and g D are the proportional, integral, and derivative gains, respectively; g P was set to 1.0, g I to 0.02, and g D to either 0.02 or 0 (with no appreciable effect).Our experience has been that the integral component is essential: omitting that component results in stabilization around an arbitrary and time-variant value; for an extensive discussion on this issue see Figure 3D in Wallach et al. (2011), a methodology paper that introduces the concept of Response Clamp in the neural context.Finally, the contrast C n equals the controller's output plus some baseline: where C baseline = 0 5 . .The main observation is shown in Figure 2, where the responses of one subject (left column) and the group of eight subjects (right column) that were tested in closed-loop (dark blue, top row), replay (purple, middle row), and fixed (a kind of yellow, bottom row) sessions are plotted.Let us start with Figure 2A, describing the results obtained in the closed-loop session of one individual.The desired detection probability P n * (see Materials and Methods) was set to 0.5, and the control algorithm updated the contrast C n , trial-by-trial, as indicated by black dots (righthand y-axis of Figure 2A).The resulting estimated detection probability ( P n , dark blue, lefthand y-axis) of that individual gradually approached the desired value, albeit fluctuating about it.Also note the expected anti-correlation of C n and P n .Figure 2B shows the average performance (and SD) of all eight subjects that participated in such a closed-loop session, showing that the controller converges within ca. 100 trials, and succeeds in "clamping" the detection probability at around 0.5, as preset.
Figure 2C shows the performance of the same individual whose data are shown in Figure 2A, but now in the replay session, where the C n series obtained in the closed-loop session (black dots) is "replayed," regardless of the subject's responses.Under these conditions the control algorithm is shut down and stimulus contrast is completely decoupled from the subject's behavior.Note the emergence of large slow fluctuations around the preset 0.5 detection probability; despite the fact that the stimulus series is practically identical to that shown in Figure 2A, the performance in Figure 2C is very different.Figure 2D shows the average performance (and SD) of all eight subjects that participated in the replay session.
And finally, as demonstrated in Figure 2E, when the average C n of the subject -whose data is shown in Figures 2A,C -is used in the fixed session, where all 450 trials have identical contrast (black line of Figure 2E), large slow fluctuations and drift (which might reflect perceptual fatigue) are observed.Figure 2F shows the average performance (and SD) of all eight subjects that participated in these fixed session.
As mentioned in the introductory comments to this study, psychophysical data exhibits correlations over extended timescales in responses to near-threshold stimuli (Werthheimer, 1953;Gilden, 2001;Monto et al., 2008).Such long-term correlations are apparent in the open-loop data of Figures 2C,E.Figures 3A,B demonstrates one way to quantify the statistics of these correlations, using Fanofactor analyses (see Materials and Methods).It shows that both for the case of an individual subject (Figure 3A) as well as the lumped data of all subjects (Figure 3B), response fluctuations in the open-loop modes are indeed dominated by long-range correlations, whereas responses in the closed-loop (controlled) mode exhibit regularity, reflected as a decreasing Fano-factor within the observed bin sizes.Autocorrelation or, equivalently, power spectrum analyses also show substantial lessening of slow frequency components in the closed-loop mode (data not shown).
The group statistics of Figures 2B,D,F are summarized in Figure 3C.Clearly, the best performance is obtained when relational dynamics are allowed between P n (the performance of the observer) and C n (the contrast series).One possible explanation to this result is that under these different experimental conditions there is a change in the sensitivity of the subject to the stimulus.Figure 3D shows the psychometric functions for closed-loop and replay sessions, calculated by averaging the responses (1s and 0s),

fano-factor analysIs
To quantify correlations in a given sequence of binary responses, the latter was divided to non-overlapping bins of size K.A count sequence Z K was constructed by registering the number of positive responses (i.e., trials in which the visual object was detected) within each of the bins.This process was repeated for all bin sizes used.For each count sequence thus obtained, the Fanofactor (variance to mean ratio) is defined as F(K) = var(Z K )/ mean(Z K ) and plotted as a function of the bin size, K.As extensively established by Teich et al. (1997), this quantity provides a way to extract correlations in a sequence of binary events: An uncorrelated white process (e.g., Homogeneous Poisson Process) yields a constant F(K) for all bin sizes; regularity (e.g., low-jitter oscillation) would be reflected in a decreasing F(K) within the relevant range of bin sizes; and, finally, a complex temporal structure, exhibiting long-term correlations (e.g., fractal point process), would produce an ever increasing F(K), following a power-law relation to bin size.

closed-looP, rePlay, and fIxed contrast Modes
In the basic design, each subject was exposed to three experimental sessions denoted closed-loop, replay, and fixed.The first session was always a closed-loop session, whereas the second and third were replay and fixed sessions, introduced in an alternating order to different subjects.A 10-min break was given after the first and second sessions.
In the closed-loop session, the desired response probability (P n * ) was kept constant (P n * = 0 5 ., unless indicated otherwise in the main text) and the control algorithm operated as explained above, updating the contrast (C n ) of the foreground object from one trial to the next based on the error signal (e n ).The series of 450 C n values produced in this closed-loop session (500 trials minus the 50 sham trials), served for the generation of both dynamically and statistically identical foreground objects in the replay session.Thus, in the replay session the control algorithm was disconnected, yet we were able to record the responses of the subject to exactly the same series of contrasts presented in the closed-loop session, but now in an open-loop context, detached from the trial-by-trial coupled observer's-observed dynamics.In the fixed session, the average contrast calculated from the series of above mentioned contrasts was used for all presentations, thus omitting stimulus variance altogether.This fixed session allowed us to estimate the impact of stimulus fluctuations on response dynamics.

results
The nature of the detection task is demonstrated in Figure 1: The left panel shows a background raster only.The middle panel shows a barely detectable foreground object (indicated in the top-left field).The righthand panel demonstrates an obvious foreground object.The probability of false positive detection, calculated from responses of all subjects to the 50 sham trials was negligible (0.017, SD = 0.029, n = 8), indicating that the subjects did not tend to report detection when they did not really see something.The average response time was around 1 s per trial, slightly longer in the closed-loop session (1.03 s, SD = 1.3) compared to replay and fixed sessions (0.89 s, SD = 1.6 and 0.81 s, SD = 1.0, respectively).Response time distributions in all three sessions had a long right tail (coefficient of skewness: 11, 17, and 19, respectively).the coupling of observer's-observed dynamics is disconnected, the variance of detection probability markedly increases and slow correlations seem to emerge, at all P n * tested.As shown in the averaged histograms of Figure 4B, within the fixed phase of the experiment of Figure 4A the detection probabilities are skewed, exhibiting a "binary" preference toward 1 or 0 (for the cases of P n * = 0 75 .and P n * = 0 25 ., respectively).This preference stands in contrast to the symmetric case of P n * = 0 5 .(e.g., Figure 3C).

conclusIon
Two basic observations are presented here.The first is that trialto-trial variability in sensory detection of a weak visual stimulus is dramatically diminished when rather than presenting a stimulus contrast that is independent of the subject's ongoing actions, the fluctuations in stimulus contrast were matched to the fluctuations in a subject's judgment.Clearly, this result reaffirms that trial-to-trial fluctuations are not "noise" in the strict sense of being independent for each of the eight subjects, in different contrast (C n ) bins.(Note that average response thus calculated is not the same quantity as detection probability P n ; the latter takes into account the temporal order of responses.)Clearly, these psychometric functions show that both threshold and sensitivity extracted from the responses of all the subjects, are practically identical in closed-loop and replay.However, the richness of the dynamics and the marked differences between closed-loop and replay modes seen in Figures 2 and 3B, are practically averaged out when the data are collapsed to standard psychometric functions the kind shown in Figure 3D.
The importance of instantaneous coupling between the observer's behavior and the stimulus dynamics is demonstrated in Figure 4A, where a closed-loop mode is instantly switched to a fixed mode, by disconnecting the controller and using a constant contrast value (average of the C n series over a time segment depicted by a black bar).Figure 4A shows data of a single subject, in three different preset P n * values (0.25, 0.5, and 0.75).As soon as of each other.Moreover, the significant difference, between features of fluctuations measured when dynamic observer-observed relations exist, and those measured in the absence of such coupled dynamics, calls for re-examination of the way psychophysical experiments are conducted.Indeed, measuring temporal fluctuations of a psychophysical function under open-loop conditions, where there is no relation between subject performances and sensory object contrast dynamics, is a most un-natural setting.Here we threshold stimulus (in auditory, visual, and somatosensory modalities) are correlated over timescales ranging from seconds to days (Werthheimer, 1953).These long-range trial-to-trial response correlations were then interpreted as reflecting modulation of an a priori detection probability by recent subjective experience, a meta-cognitive guessing process that is active where there is no possibility for stimulus-determination; the assumption being that "guesses are more likely to be influenced by preceding responses (success and failures) than are sensory judgments" (Conklin and Sampson, 1956).Later years brought with them a more reductionistic focus on neural sources of "noise" and "short-term plasticity" that may account for observed trial-to-trial response variability (Faisal et al., 2008).Viewed from this historical angle, the results presented here pull the pendulum back to the meta-cognitive pole, offering an interpretation according to which -when facing a weak stimulus -subjects vary their response patterns, seeking to establish (predictive?see Rosen, 1985;Creutzig et al., 2009) relations between their actions and the dynamics of stimulus features.This interpretation, in its broader sense, goes far beyond psychophysics of weak stimulus detection; it touches upon what psychologists try to tell us over the past 50 years on the developing mind (e.g., Stern, 1985), when we care to listen.implemented an adaptive algorithm (PID) borrowed from control theory in order to couple the observer-observed dynamics.The PID control algorithm has theoretical advantages in the present context by being simple, and probably the most extensively used and theoretically studied in control engineering (Levine, 1996).Having said that, there exist many other adaptive psychophysical procedures (Treutwein, 1995) that are, actually, in use when experimentalists attempt to identify points of interest on psychometric functions.We propose to substantially extend their use in order to expose the dynamics of perception under more natural experimental conditions.The second basic observation is that the above diminishing of trial-to-trial fluctuations by coupling between observer-observed dynamics, is not accompanied by a change in sensory sensitivity to the input.Taken together, the two basic observations suggest that trial-to-trial variability in sensory detection of weak stimuli might reflect a high-level control process.
As pointed out by Werthheimer (1953), trial-to-trial variation at threshold was generally attributed, in the early days of psychophysics, to uncontrolled experimental conditions, with the assumption that the subject is stable.Response fluctuations, however, were soon shown to be non-independent (Verplanck et al., 1952); that is -successive responses to repeated presentations of the same integral-derivative (PID) controller was realized in Wolfram's Mathematica 7.0 environment.The input to the controller is the error signal, * and P n are the desired and actual detection probabilities (calculated as explained below) at the n-th trial, respectively.The output of the controller is generally composed of three expressions,

Figure 1 |
Figure 1 | Detection task.Three examples of single trial presentations: Background raster only (left panel), a barely detectable foreground object indicated in the top-left field of the middle panel, and an obvious foreground

Figure 2 |Figure 3 |Figure 4 |
Figure 2 | Detection probability in closed-loop, replay, and fixed sessions.Data obtained from an experiment on one individual (left column) and the summation of observations from the eight different subjects that were tested in this protocol (right column).In all cases, the fluctuations around mean detection probability are significantly smaller in the closed-loop session.The initial decline in detection probability, which is most apparent in the closed-loop and replay sessions, reflects the initial setting of both P n and C n to 0.5.Error bars in the righthand column depict SD across all subjects.