We also performed a natural image analysis to test our hypothesis that human detection thresholds are linked to the frequency with which different plaid configurations occur in natural scenes. For this purpose, we quantified whether plaids occur more or less often than predicted from the likelihoods of their constituting single patches. This statistics was derived from analyzing how similar local image regions are to the four different oriented gratings used in our experiments (Figure 4). Mathematical details of the procedures described below can be found in the Supplementary Material.

Image processing consisted of the following basic steps, hereby making use of the convolution theorem to realize whitening and Gabor filtering in a numerically efficient way:

Conversion from RGB color space to grayscale.

In natural scenes, lower spatial frequencies typically occur with higher amplitudes than higher spatial frequencies (van der Schaaf and van Hateren, 1996). To compensate for this effect, whitening is used to equalize the average spectral composition of image ensembles, allowing us to separate the actual probabilities of occurrence of the different grating plaid configurations from the typical intensity with which they are present.

The grating patch – image patch overlaps O_p statistically quantify how well an image patch is explained by the presence of a single grating with parameters p (e.g., comprising orientation and spatial frequency, detailed explanation see Supplementary Material). Consequently, it also allows assessing the presence or absence of full plaid configurations C comprising a combination of four grating patches. To quantify how often configuration C is encountered in an image ensemble E, we computed the ratio Λ(C) between the joint likelihood to observe C and the likelihood to independently observe the single grating patches c_i of C:

For example, a value of Λ(C) = 2 would mean that plaid C occurs twice as frequently as expected from the probability of occurrence of its single grating patches c_i.

As image ensembles E, we used two different data bases: first, the Corel Image Database [Corel Mega Gallery (add-on to CorelDraw version 6), Corel Corporation (1996)] with about 68,000 images of size 384 × 256 pixels in JPEG-compression, and the McGill Color Image Database with about 820 color-calibrated images of size 576 × 768 pixels without compression (Olmos and Kingdom, 2004). JPEG compression is known to introduce artifacts in cardinal orientations. However, since we were only interested in oblique orientations this putative confounding factor was of negligible concern for our investigations.

The threshold contrast results are summarized in Figure 5B for the small distance grid (1°, left panel) and the large distance grid (2°, right panel). The results patterns for small and larger inter-patch distance were remarkably different. To substantiate different effects of alignment and spatial frequency for the two grid sizes we analyzed the threshold contrast data with ANOVA, and tested against the assumption of probability summation among the four grid positions with a confidence interval test.

For assessing how well a model with a particular set of adapted parameters fits the experimental data, we first counted the number of conditions N_outside in which the model prediction fell outside the confidence intervals around the measured thresholds. The lower the minimum N_outside achieved over the full set of simulations, the better the fit between model and experiment. In total, there were 19 conditions: 9 plaid configurations for each distance plus one probability summation threshold which we required the model to reproduce if all interactions were set to zero (see Supplementary Material). Second, we computed the mean quadratic error E₂ between predicted and measured thresholds.

Parameter optimization of model A was performed starting from 500 different initial conditions. After convergence of the stochastic gradient descent method, 457 parameter sets yielded thresholds for which N_outside = 0, giving an average E₂ of 1.68 · 10⁻⁶ ± 4.72 · 10⁻⁶. The best performing model with lowest quadratic distance E₂ = 6.6 · 10⁻⁸ is shown in Figure 6, with its interactions displayed in panel Figure 6A, and the corresponding linking function in Figure 6B. Detection thresholds are shown in panel Figure 6C, demonstrating a perfect fit between model and experiment. In particular, this fit is much closer than the human response variability in the experiment expressed by the confidence intervals. Typically, such a perfect match indicates that a model contains too many free parameters (“overfitting”) and thus will not generalize well to other experimental situations. For different initializations, the parameters after convergence were very similar, which we demonstrate by also showing the nine linking functions for the next best matches of model to experiment (Figure 6B, black lines). All linking functions have exponents around 1.17–3.35 and exhibit a similar shape (concave down).

For model B, we also performed simulations from 500 initial conditions. Since the lower number of parameters restricts the degrees-of-freedom in the model's dynamics, the 8 best models yielded a minimum N_outside = 2, with an average E₂ of 2.54 · 10⁻⁴ ± 4.64 · 10⁻⁵. The best performing model with lowest quadratic distance E₂ is shown in Figure 7, with its interactions displayed in panels Figure 7A, and the corresponding linking function in Figure 7B. Detection thresholds shown in panels Figure 7C confirm that the fit of model to experiment is now less accurate. However, the number of parameters is about three times lower and thus a perfect fit is less likely than for model A. Again, for different initializations, the parameters after convergence are very similar. For example, the exponent of the linking function now varies between 0.99 and 1.37 (see Figure 7B, black lines, for examples of linking functions). An advantage of model B is that the parametric definition of the interactions as distance-dependent Gaussian functions allows one to predict interaction strengths also for other plaid configurations not used in this particular experiment.

Although individual weight values were different between models A and B, the general pattern which emerged after learning was very similar and thus confirmed the consistency of our approach. Comparing interaction weights for models A and B we find very similar structures that provide an intuitive explanation for the empirical results:

First, we computed the mean over all interactions as displayed in Figures 6A,B for d = 1° and d = 2°. Averaged over the 457 (8) best models of type A (type B), we obtained 〈w〉 = −0.14 ± 0.07 for 1° and 〈w〉 = −0.03 ± 0.02 for 2° (〈w〉 = −0.28 ± 0.10 for 1° and 〈w〉 = −0.01 ± 0.05 for 2°), respectively. Clearly, for the smaller patch distance interactions must be more inhibitory, thus explaining the higher detection thresholds.

The image analysis was performed for all 256 possible patch configurations, including plaids never used in the experiment. For comparison with the psychophysical data, the corresponding likelihood ratios Λ(C) for the 22 patch configurations C used were extracted and sorted into the 9 categories defined by alignment (0, 1, or 2 alignments) and spatial frequency (only low SFs, only high SFs, or both SFs), identically to the presentation of the experimental data in Figure 5. In Figure 8A, the results are shown for both data bases for a patch distance of d = 2°. In particular, we plotted 1/Λ since our hypothesis is that the more likely a patch, the lower will be the corresponding detection threshold. Compared to the experimental data for the same distance shown in Figure 8B, it turns out that the general result pattern is well reproduced, in particular for configurations with two alignments. Two exceptions are the inverse likelihood ratios for high SF configurations: they are much lower than in the corresponding psychophysical data.

Since natural images can be observed from different viewing angles, one degree of visual angle can correspond to a varying numbers of pixels. To check how strongly results depend on this unknown variable, we analyzed image data using a range from 6 to 24 pixels/degree for the Corel data base, and 12–48 pixels/degree for the McGill data base, over a range of spatial distances from d = 0.5° to d = 4°. Although results varied quantitatively, the general pattern as displayed in Figure 8 remained unchanged (not shown). There was also no conspicuous change when we varied the threshold used for reducing noise. While this finding means that the psychophysical data for d = 1° has no apparent relation to natural image statistics, it nevertheless confirms the well-known fact that natural image statistics is in many aspects scale-invariant (Ruderman, 1997).

A single, common functional principle emerges when putting these observations into context with our knowledge about neural activation and detection thresholds if grating patches are smaller than 1°: typically, neural activation increases and detection thresholds decrease when the diameter of the patches becomes larger (Kretzberg and Ernst, 2013). This suggests an interaction profile resembling a Mexican hat: short-range excitation combined with medium-range inhibition. Now considering the orientation-specific interactions, a second Mexican-hat profile emerges in spatial frequency space: excitation if frequencies are close, and inhibition if frequencies are further apart. Functionally, Mexican-hat interactions are closely related to edge detection and image compression (e.g., see examples in Ernst, 2013): stimuli consisting of similar features are suppressed (low neural activity, Sillito et al., 1995; Levitt and Lund, 1997), while stimuli consisting of dissimilar features are enhanced (high neural activity, Sillito et al., 1995; Levitt and Lund, 1997). More complex interaction patterns (surrounds) which might be used by the brain to detect specifc patch configurations have also been reported from physiological studies (Walker et al., 1999). In our situation, the Mexican-hat in Cartesian space will suppress configurations with multiple, closely spaced patches (of any orientation and spatial frequency) in favor of configurations with more widely spaced, or isolated, patches. At the same time, the Mexican-hat in spatial frequency space will suppress configurations containing many spatial frequencies, while enhancing configurations with a single spatial frequency—provided that the patches have similar orientations, since this latter interaction is orientation (difference)-specific (Figure 9).

The finding of suppression in configurations with multiple, closely spaced patches resembles crowding phenomena, i.e. general detrimental effects of nearby probe stimuli on the perception of test stimulus attributes. Crowding effects were first observed in letter identification, but appear in a large variety of tasks (see Levi, 2008, for a comprehensive review). However, crowding effects mostly concern object feature identification and discrimination, but hardly object detection (Pelli et al., 2004). Particularly, crowding does not affect the apparent contrast of the test stimulus (Levi, 2008). Further, crowding effects are quite feature specific (Herzog et al., 2015). On the contrary, the inhibitory effects of close grating patch spacing reported here are effects on contrast detection, and were observed irrespective of orientation alignment and spatial frequency homogeneity. We therefore conclude that the inhibitory interaction for close spacings are better understood as lateral or surround masking effects (see Levi, 2008, for further aspects of distinguishing crowding from masking phenomena). Masking effects of surround gratings on central grating patches were extensively studied by Xing and Heeger (2000, 2001). Center (test) and surround (inducer) gratings were separated by a thin annulus, and the test contrast was matched to a reference grating of equal size, but without surrounding stimulus. The stimulus geometry in Xing's and Heeger's experiments compares to ours in the short (1°) spacing, since there the patches were separated by just 0.41° space of background luminance. Results showed that the test grating had lower perceived contrast than the reference grating, even when inducer contrasts were low. This result was practically independent of the spatial frequency of the gratings, but orientation difference of center and surround diminished the inhibitory contextual influence. In a similar center-surround arrangement Bruchmann et al. (2010) studied the temporal dynamics of the center-surround interaction, and consistently found evidence for inhibitory effects of the surround masker. Consistent with these results, we conclude that, despite some orientation selectivity, net contextual influence is inhibitory in the near surround.

Orientation-specific interactions are well known from electrophysiological studies on contour integration and are already observed in the form of firing rate enhancements for configurations of only two aligned edges (Ito and Gilbert, 1999). These effects increase in strength with an increasing length of a contour embedded into a randomly oriented background (Li et al., 2006). Optical imaging confirms these findings, and allows us to observe modulatory effects in a spatially extended manner (Gilad et al., 2013). Note that while our study locates all interactions within a single cortical layer, in the real brain different types of contextual interactions are typically located in different visual areas, as e.g., contour integration might be performed not in V1 but in V2 or in V4 (Chen et al., 2014). Moreover, psychophysical studies have shown that in contour integration, spatial frequency and orientation alignment information are combined to yield higher detection performances than expected from the additive combination of the single cues (Persike and Meinhardt, 2015a). Furthermore, if spatial frequencies in contour and background become homogeneous, detection performance is enhanced (Persike et al., 2009; Persike and Meinhardt, 2015b), similar to the effects observed with SF-homogeneous plaids in the 2° condition.

The functional role of the Mexican hat profile in spatial frequency is a potential advantage in reaching unique shape descriptions from single spatial scales. Studies on the detectability of simple global shapes have shown a detection advantage for shapes formed by parameter homogeneous Gabor patches, compared to heterogeneous Gabor elements (Saarinen et al., 1997; Saarinen and Levi, 2001). The advantage was found to be relatively independent of orientation alignment, and stressed the benefit of parameter homogeneity, in contrast to mixed configurations (Saarinen et al., 1997). These findings correspond to our finding of enhanced plaid detectability for orientation and spatial frequency homogeneous grating patches. However, evidence for suppressive interaction among different spatial scales is closely bound to contextual interactions in 2D configurations. Studying the interaction of different spatial scales at the same retinal location has revealed gradual decline of facilitation among grating patches when spatial scale difference is increased, until independence is reached for far apart local carrier frequencies of the grating stimuli (Watson, 1982). While the bandwidth of the psychophysical tuning function is close to the bandwidth of the grating patches, there is no evidence for inhibitory interactions among spatial frequency channels at one retinal location (see Meinhardt, 2001, p. 417). In their seminal psychophysical study on lateral grating patch interactions Polat and Sagi (1993) also studied contextual interactions for spatial scale differences of test and flankers. Results showed that the biphasic contextual response profile was attenuated for increasing spatial scale differences, but there was no change of the form of the profile, indicating no suppressive interactions for larger spatial scale differences. However, Polat and Sagi (1993) tested 1D contextual configurations of co-aligned stimuli, but not 2D configurations with collinear and collateral stimulus arrangements, as done here. Electrophysiological studies aiming at measuring a complete contextual interaction topography in 2D (Kapadia et al., 2000) have unfortunately not yet explored whether the contextual response field changes qualitatively if there is a spatial scale difference of central test and peripheral probe stimulus. More data are needed to settle the constraints for the relationship of object descriptions on different spatial scales and their neural underpinnings.

The observation that the statistics of plaid configurations in natural images is similar for a wide range of inter-patch distances is not surprising: many studies have shown (albeit sometimes w.r.t simpler features) that natural images have self-similar structures, i.e. that their statistics are invariant with respect to the particular observation scale (e.g., see Ruderman, 1997). For large inter-patch distances (in our case 2°), the visual system seems to realize interactions that enhance feature combinations with higher likelihoods to occur in “nature.” But why does this parallelism fail at 1°? Apart from the trivial explanation that there might be no reason at all, or that biophysical constraints prevent the brain from realizing the necessary neural couplings, there might be one functional explanation: Instead of just enhancing representation proportionally to their likelihood, our brain might rather be interested in enhancing representations only when they appear in contexts normally leading to reduced saliency. For example, in contour integration a continuous curve is much easier to follow and to integrate than a broken curve consisting of isolated, colinearly oriented line segments separated by “open” space: the larger the separation of elements, the less salient are contours (Mandon and Kreiter, 2005). Enhanced processing with larger distances would help to bridge the gaps and allow to detect the contour, in particular if these gaps would be filled with distractor elements. This example bears a close resemblance to a plaid consisting of four iso-oriented, frequency-homogeneous gratings, where we find suppression when the elements are very close (1°), but enhancement when the elements are well separated (2°). Here one might speculate that the visual system suppresses “trivial” while enhancing “surprising,” or more challenging feature conjunctions.

The two variants of our network model are extremely simplified versions of more complex approaches (e.g., Li, 1998; Ichida et al., 2006; Hansen and Neumann, 2008). This reductionist approach was taken for two reasons. First, for revealing computational mechanisms as succinctly as possible, and second, for reducing free parameters as far as possible. Even so, model A is still susceptible to overfitting, as indicated by its match to the experiment being far better than expected from the confidence intervals of the experimental results. Model B is superior in the sense of avoiding overfitting: except from one configuration, still all experimental results are explained within their confidence intervals.

Short-range excitatory couplings, which would have been implemented as self-interactions between orientation columns, were not included in our model. Mathematically, including these interactions is equivalent with re-scaling the feedforward and recurrent input strength without having such interactions. Consistent with previous results, our model also requires interaction ranges to depend on SF preference of the feature detectors involved. It turned out that this interaction has to scale less than exactly anti-proportionally to SF (Polat and Sagi, 1993). Furthermore, we also tested whether cross-SF interactions are required to be orientation-specific. Without this specificity, the match between experimental results and model predictions was far worse. A comparison of our interaction scheme to association fields obtained from studies on contour integration is, unfortunately, not possible. Since our visual stimuli sample orientation (difference) space only extremely sparsely, it is difficult to predict interactions for orientation preferences not being parallel or orthogonal to each other. In addition, we obtain the best fit of the model by not having different interaction strengths between parallel and aligned configurations with same orientation. This feature is in contrast to interactions predicted from contour integration studies where parallel configurations (“ladders”) are much harder to perceive than aligned configurations (“snakes”) (Bex et al., 2001; May and Hess, 2007; Vancleef and Wagemans, 2013).

Can our approach make predictions for even more complex plaids? Our interpretation of the interactions as one functional principle (Mexican hat) extending over multiple feature dimensions makes possible some “educated guesses.” For example, if a 2° configuration is “filled” by adding five patches in the spaces between the original plaid, we expect inhibition to kick in and raise detection thresholds. This would be consistent with our functional explanation that a closely spaced, 3 × 3 configuration of identical patches would be not surprising at all, but considered as a homogeneous texture possibly just being the background of much more interesting image features. It would also be interesting to restrict the image analysis to “informative” patch configurations, as e.g., has been done for oriented edge statistics by requiring human observers to label contours belonging to the same object (Geisler and Perry, 2009).

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.