Widespread Brain Areas Engaged during a Classical Auditory Streaming Task Revealed by Intracranial EEG

Dykstra, Andrew  R; Halgren, Eric; Thesen, Thomas; Carlson, Chad  E; Doyle, Werner; Madsen, Joseph  R; Eskandar, Emad  N; Cash, Sydney  S

doi:10.3389/fnhum.2011.00074

ORIGINAL RESEARCH article

Front. Hum. Neurosci., 03 August 2011

Sec. Brain Imaging and Stimulation

volume 5 - 2011 | https://doi.org/10.3389/fnhum.2011.00074

Widespread Brain Areas Engaged during a Classical Auditory Streaming Task Revealed by Intracranial EEG

AR
Andrew R. Dykstra ^1,2^*
EH
Eric Halgren ^3,4
TT
Thomas Thesen ⁵
CE
Chad E. Carlson ⁵
WD
Werner Doyle ⁵
JR
Joseph R. Madsen ⁶
EN
Emad N. Eskandar ⁷
SS
Sydney S. Cash ²

1. Program in Speech and Hearing Bioscience and Technology, Harvard-MIT Division of Health Sciences and Technology Cambridge, MA, USA
2. Cortical Physiology Laboratory, Department of Neurology, Massachusetts General Hospital and Harvard Medical School Boston, MA, USA
3. Department of Radiology, University of California San Diego San Diego, CA, USA
4. Department of Neurosciences, University of California San Diego San Diego, CA, USA
5. Comprehensive Epilepsy Center, New York University School of Medicine New York, NY, USA
6. Department of Neurosurgery, Brigham and Women's Hospital and Harvard Medical School Boston, MA, USA
7. Department of Neurosurgery, Massachusetts General Hospital and Harvard Medical School Boston, MA, USA

Abstract

The auditory system must constantly decompose the complex mixture of sound arriving at the ear into perceptually independent streams constituting accurate representations of individual sources in the acoustic environment. How the brain accomplishes this task is not well understood. The present study combined a classic behavioral paradigm with direct cortical recordings from neurosurgical patients with epilepsy in order to further describe the neural correlates of auditory streaming. Participants listened to sequences of pure tones alternating in frequency and indicated whether they heard one or two “streams.” The intracranial EEG was simultaneously recorded from sub-dural electrodes placed over temporal, frontal, and parietal cortex. Like healthy subjects, patients heard one stream when the frequency separation between tones was small and two when it was large. Robust evoked-potential correlates of frequency separation were observed over widespread brain areas. Waveform morphology was highly variable across individual electrode sites both within and across gross brain regions. Surprisingly, few evoked-potential correlates of perceptual organization were observed after controlling for physical stimulus differences. The results indicate that the cortical areas engaged during the streaming task are more complex and widespread than has been demonstrated by previous work, and that, by-and-large, correlates of bistability during streaming are probably located on a spatial scale not assessed – or in a brain area not examined – by the present study.

Introduction

The auditory system is constantly faced with the challenge of decomposing the complex mixture of sound arriving at the eardrums into an accurate representation of the acoustic environment. This decomposition, termed auditory scene analysis (ASA, Bregman, ), is critical for survival and communication and its failure is a common symptom reported by elderly individuals and those with sensorineural hearing loss. Despite its importance in daily life, the neural mechanisms of auditory scene analysis remain unclear (Carlyon, ; Micheyl et al., ; Snyder and Alain, ; Elhilali and Shamma, ; Nelken and Bar-Yosef, ; Bidet-Caulet and Bertrand, ; Winkler et al., ; Shamma and Micheyl, ; Shamma et al., ). One aspect of ASA – auditory streaming (the segregation of time-varying acoustic energy into distinct perceptual objects) – can be studied in a controlled setting using sequences of pure-tone triplets of the form ABA-ABA- (Miller and Heise, ; van Noorden, ; Bregman, ), where A and B denote tones of different frequencies separated by a silent gap (Figure 1A). Many psychophysical studies dating back to the 1950s have shown that when the frequency separation (ΔF) between the A and B tones is small, listeners hear the sequence as a single stream comprised of both A and B tones and that when ΔF is large, they hear the sequence as two isochronous streams, one of A tones and one of B tones (Miller and Heise, ; van Noorden, ; see http://web.mit.edu/∼adykstra/Public/streaming_demo.wav for a demo). Interestingly, percepts evoked by sequences with intermediate ΔF are bistable (i.e., can be heard as either one stream or two) and can switch between two stable states, either spontaneously or with effort (van Noorden, ; Anstis and Saida, ; Carlyon et al., ).

Figure 1

Recent interest in the neural underpinnings of auditory streaming has produced several studies using ABA tone sequences while recording from the auditory cortex in a variety of species including insects (Schul and Sheridan, ), fish (Fay, , ), bats (Kanwal et al., ), songbirds (Bee and Klump, , ; Itatani and Klump, , ; Bee et al., ), ferrets (Elhilali et al., ), non-human primates (Fishman et al., , ; Micheyl et al., ), and humans (Sussman et al., ; Deike et al., , ; Cusack, ; Gutschalk et al., , ; Snyder et al., ; Snyder and Alain, ; Wilson et al., ; Kondo and Kashino, ; Schadwinkel and Gutschalk, ,). A prevailing model from these studies posits that a two-stream percept will be evoked whenever the A and B tones excite non-overlapping populations of neurons (but see Elhilali et al., ). However, inherent limitations in previous work related to spatiotemporal resolution, sparsity of coverage, and lack of direct behavioral measures in experimental animals preclude straight-forward interpretation. A general extension of this model is schematized in Figure 1B. Specifically, a parametric variation of a given stimulus or stimulus feature could produce neural activity patterns which vary linearly or categorically as shown by the blue and red curves, respectively. Noise in the response of a population showing a linear relationship with the stimulus, when fed to a population showing a more categorical relationship, could engender sufficient trial-to-trial variability for bistable perception. While such activity patterns have been widely reported in vision (for reviews see Logothetis, ; Leopold and Logothetis, ; Sterzer et al., ), only limited evidence for such a mechanism exists in the auditory system (Cusack, ; Gutschalk et al., , ; Kondo and Kashino, ).

Here, we report the results from experiments in which direct cortical recordings were made from widespread brain areas of neurosurgical patients with epilepsy (Engel et al., ) while they participated in a classical auditory streaming paradigm. Our aims were to better characterize the neurophysiological correlates of auditory streaming, extend them into brain areas outside the auditory cortex and frequency regions less observable with non-invasive measure (Crone et al., ), and test the idea of neuronal variability as a mechanism for perceptual bistability in the auditory modality (Almonte et al., ; Moreno-Bote et al., ; Deco and Romo, ; Deco et al., ; Gigante et al., ; Shpiro et al., ) by comparing evoked responses to physically identical stimuli when they were perceived as one vs. two streams. Our participants listened to ABA tone sequences and indicated at the end of each sequence whether they were hearing one or two streams at the end of the sequence. For each electrode sampled in a given patient, we compared responses across ΔF conditions as well as perceptual report in an attempt to identify correlates of both during a classical auditory streaming task. We hypothesized that when a participant perceived one (two) stream(s), the evoked response would be similar to those conditions which consistently engender a one-stream (two-stream) percept. Responses from widespread brain areas showed robust correlates with ΔF but, surprisingly, rarely differed based on percept per se.

Materials and Methods

Ethics statement

All procedures were approved by the Institutional Review Boards at Partners Healthcare (MGH and BWH), the New York University (NYU) Langone Medical Center, and the Massachusetts Institute of Technology (MIT) in accordance with NIH guidelines. Written informed consent was obtained from all patients prior to their participation.

Listeners

Twelve patients with intractable epilepsy underwent invasive monitoring in order to localize the epileptogenic zone prior to its surgical removal. Each patient was implanted with an array of sub-dural platinum–iridium electrodes embedded in silastic sheets (2.3 mm exposed diameter, 10 mm center-to-center spacing; Ad-tech Medical, Racine, WI, USA) placed directly on the cortical surface. Prior to implantation, each patient underwent high-resolution T1-weighted MRI. Subsequent to implantation, patients implanted at Massachusetts General Hospital (MGH) and Brigham and Women's Hospital (BWH) underwent post-operative computerized tomography (CT); patients implanted at NYU underwent post-operative MRI. Electrode coordinates obtained from post-operative scans were co-registered with preoperative MRI and overlaid onto the patient's reconstructed cortical surface using FreeSurfer (Dale et al., ; Fischl et al., ) and custom MATLAB (The MathWorks, Framingham, MA, USA) scripts (Dykstra et al., under review; Wang et al., personal communication, Comprehensive Epilepsy Center, NYU School of Medicine). Electrode coordinates were then projected onto the FreeSurfer average brain using a spherical registration between the individual's cortical surface and that of the FreeSurfer average (Fischl et al., ). The data from three patients were excluded from analysis due to excessive noise caused by technical malfunction; the data reported here were from the remaining nine patients (Table 1).

Table 1

Patient	Hemisphere	Sex	Age	Stimulation	AF conditions	Seizure focus
S1	Left	M	21	Diotic	0,5,6,7,12	Left anterior temporal
S2	Left	M	22	Diotic	0,5,6,7,12	Left cingulate cortex
S3	Left	F	52	Diotic	0,5,6,7,12	Left anterior temporal
S4	Right	F	42	Diotic	0,5,6,7,12	Right anteromedial temporal
S5	Left	M	37	Diotic	0,5,6,7,12	Left anteromedial temporal
S6	Right	M	19	Diotic	0,2,4,6,8,10,12	Right parietal
S7	Left	F	38	Monaural (R)	0,2,4,6,8,10,12	Left mesial temporal
S8	Left	M	29	Diotic	0,2,4,6,8,10,12	Left anterior temporal/left parietal
S9	Left	M	25	Monaural (R)	0,2,4,6,8,10,12	Left anteromedial temporal

Patient information.

Stimuli and procedure

Stimuli were long sequences of pure-tone triplets of the form ABA-ABA-..., where A and B represent individual tones and the dash represents a silent gap (Figure 3A). Each tone was 100 ms in duration with 10 ms raised-cosine on- and off-ramps. The inter-stimulus interval (ISI) between the first A-tone and B-tone, as well as between the B-tone and second A-tone, was 25 ms; the ISI between the second A-tone and subsequent triplet was 150 ms. Stimulus onset asynchrony (SOA) between successive A tones was 250 ms; SOA between successive B tones was 500 ms; triplet onset asynchrony was also 500 ms. Total duration of each sequence varied between 6.5 and 10 s (13 and 20 triplets, respectively) depending on the listener (for P1–P5, duration varied between 6.5 and 7.5 s; for P6–P9, duration was 10 s). The B-tone frequency was fixed at 1 kHz. The A-tone frequency varied between 0 and 12 semitones below the B-tone. Listeners P1, P2, P3, P4, and P5 participated in conditions in which the frequency separation was 0, 5, 6, 7, or 12 semitones, where 1 semitone is an approximately 6% frequency difference. Listeners P6, P7, P8, and P9 participated in conditions in which the frequency separation was 0, 2, 4, 6, 8, 10, or 12 semitones. Each patient listened to between 200 and 378 triplets for a given frequency separation. All sounds were generated digitally in MATLAB, stored as .wav files, and converted to analog waveforms by the on-board soundcard of a laptop equipped with Presentation software (Neurobehavioral Systems, Albany, CA, USA). Stimuli were presented at a comfortable listening level via Etymotic ER-2 insert earphones (Etymotic Research, Inc., Elk Grove Village, IL, USA), diotically (when possible) or monaurally contralateral to the hemisphere of implantation. Patients were instructed to listen to the sounds and to indicate at the end of each sequence whether, at the end of the sequence, they were hearing a single “stream” comprised of all tones or two “streams,” one comprised of A tones and the other of B tones. Responses were made by button press with a response box (Cedrus Corporation, San Pedro, CA, USA) interfaced with Presentation via USB. Response windows were unconstrained, and the subsequent stimulus began 1 s after a response to the previous stimulus was entered.

Data acquisition

Intracranial EEG (iEEG) data at MGH and BWH were acquired with standard clinical EEG monitoring equipment (XLTEK, Natus Medical Inc., San Carlos, CA, USA) at a sampling rate of 250 Hz (P1) or 500 Hz (P2,P3,P6,P8). At NYU, iEEG data were acquired with a customized system at a sampling rate of 30 kHz (P4,P5,P7,P9). All data were subsequently re-sampled to 500 Hz for analysis. All data were referenced to either an inverted intracranial electrode (i.e., facing the inner skull table) remote from the electrodes of interest (P1,P2,P3,P6,P8) or a screw bolted to the skull (P4,P5,P7,P9). For each patient, clinically indicated, high-resolution T1-weighted structural MRI scans were acquired prior to surgery. High-resolution CT (P1,P2,P3,P6,P8) or structural MRI (P4,P5,P7,P9) scans were acquired subsequent to surgery for the purpose of electrode localization.

Data pre-processing

Intracranial EEG data were bandpass filtered offline between 1 and 190 Hz and notch filtered at 60 Hz and its harmonics using zero-phase shift FIR filters. Independent component analysis using the runica algorithm (Bell and Sejnowski, ) in EEGLAB (Delorme and Makeig, ) was performed on the “raw” data. Components dominated by large artifacts were identified and removed by inspection. The component data were then back-projected in order to remove the artifacts from the original data.

The iEEG was epoched relative to the onset of sound sequences (yielding long epochs encompassing the entire sequence) as well as to the onset of individual ABA triplets (yielding short epochs of 0.5 s) and binned with respect to either ΔF or perceptual report within a given ΔF. For triplet-locked epochs, the first triplet in each sequence was discarded. Epochs were baseline corrected with respect to either the 500-ms preceding sequence onset (for sequence-locked epochs) or the 50-ms preceding triplet onset (for triplet-locked epochs). Epochs containing large artifacts were rejected automatically using joint probability and kurtosis algorithms in EEGLAB (Delorme et al., ). Specifically, trials with joint probabilities or kurtosis values more than four and five SDs from the normalized mean of these measures, respectively, were rejected as artifact. Additional epochs found to contain large epileptiform activity were rejected by visual inspection.

Statistical analysis

A modified version of the cluster-based, non-parametric statistical procedure outlined by Maris and Oostenveld () was used to test for effects of ΔF and bistability on triplet-locked EP amplitude. Spearman (non-parametric) rank correlation (in the case of a multiple-level factor, e.g., ΔF) and unpaired t-test (in the case of two-level factors, e.g., percept) were used as the sample-level (i.e., individual time point within a single channel) statistics in order to evaluate possible effects of ΔF (five levels for P1–P5 and seven levels for P6–P9) and bistability (always two levels), respectively. Contiguous, statistically significant samples (defined as p < 0.05) within a single electrode were used to define the cluster-level statistic, which was computed by summing the sample-level statistics within a cluster. Statistical significance at the cluster-level was determined by computing a Monte Carlo estimate of the permutation distribution of cluster statistics using 1000 re-samples of the original data (Ernst, ). For multiple-level factors (ΔF), the estimate of the permutation distribution was performed by 1000 re-samples of the condition labels associated with each level in the factor. Within a single electrode, a cluster was taken to be significant if it fell outside the 95% confidence interval of the permutation distribution for that electrode. The determination of significant clusters was performed independently for each electrode. This method controls the overall false alarm rate within an electrode across time points; no correction for multiple comparisons was performed across electrodes.

Due to the known buildup effects of auditory streaming (i.e., 2-stream percepts become more likely as time since sequence onset increases and the fact that listeners only reported what they heard at the end of each stimulus sequence, two independent analyses were carried out. The first used only the data from the second half of each sequence while the second used all data after removing the onset response (0–0.5 s after stimulus onset). The method of analysis did not effect the results, and only the results from the second analysis are shown.

Dissimilarity index

In order to further evaluate possible effects of perceptual bistability on the evoked waveforms, we computed a dissimilarity index between waveforms from individual trials and a template waveform within individual channels in which significant EP–ΔF correlations were found. Qualitatively, this index is defined as the difference between the sum-squared error (SSE) computed for the condition of interest (i.e., a specific ΔF or percept) and the minimum SSE computed across all conditions, normalized by the difference between the maximum SSE and minimum SSE computed across all conditions. The index was computed by normalizing the average SSE between the trial and the template, as follows:

where X₀ is the template waveform and X_ij is the individual-trial waveform for trial i in condition j, t is the individual time point, and T is the overall number of significant time points in condition j. The average SSE for condition j was computed as:

where N is the number of trials.

The index was then defined as:

Except for trials from the 0-semitone condition, the template was defined as the average EP for the 0-semitone condition. The template to which individual trials from the 0-semitone condition were compared was the average EP from the 0-semitone condition including all waveforms but the one from the trial i (“leave one out”). This index provides a measure of how dissimilar two waveforms are from each other. Although this index is biased to show a significant correlation with ΔF, it provides a means to (i) collapse waveforms across individual electrode sites and patients into a single quantitative metric and (ii) quantitatively compare responses to one- vs. two-stream percepts in a way that circumvents variable latencies and durations of percept- or ΔF-based effects across sites.

High-gamma–power

Waveforms of high-gamma-power were constructed using the wavelet transforms built into EEGLAB (specifically, the newtimef function). Sequence-length (between 6.5 and 10 s) epochs were used to compute the event-related spectral perturbation (ERSP) which was baseline corrected to the 500-ms preceding stimulus onset. The number of wavelet cycles used varied logarithmically with respect to frequency from three cycles at the lowest frequency tested (5 Hz) to 10 at the highest (190 Hz), yielding approximate temporal resolution of <500 ms at 8 Hz and <125 ms in the gamma-band. High-gamma-power waveforms were constructed by summing the power in frequencies from 80–190 Hz for each time point in the full time–frequency representation. These waveforms were then baseline corrected by subtracting the mean power in each trial computed across the 500-ms preceding stimulus onset. Triplet-locked gamma-power epochs were constructed by time-locking with respect to each triplet onset and subsequently binned across the various ΔF and percept conditions in the same way as the evoked potentials. The same statistical procedures described above were applied to the high-gamma waveforms.

Results

Twelve patients with intractable epilepsy listened to sequences of alternating pure tones (Figure 1A) and indicated at the end of each sequence whether, at the end of the sequence, they were hearing the tones as grouped (“1 stream”) or segregated (“2 streams”) while we simultaneously recorded the intracranial EEG (Figure 2). Three patients were excluded from analysis for technical reasons (see Materials and Methods). Summed across the remaining nine patients (Table 1), we recorded from nearly 700 electrodes in the left hemisphere and 250 electrodes in the right hemisphere, mostly on lateral cortex of the temporal, frontal, and parietal lobes (Figure 2E).

Figure 2

Behavior

Figure 3 shows the probability of hearing two streams as a function of ΔF averaged across all nine patients included in the analysis. Patients reported hearing a single stream when the ΔF was small and two streams when ΔF was large. At intermediate ΔF, the percept was bistable, i.e., patients sometimes reported hearing one stream and sometimes reported hearing two streams. A Kruskal–Wallis test confirmed a main effect of ΔF (χ²(1,8) = 34.1; p < 0.0001).

Figure 3

Evoked potentials: ΔF

In order to assess putative correlates of streaming, we tested for correlations between triplet-locked evoked-potential (EP) amplitude and ΔF which, when parametrically varied, produced changes in how the sequences were perceptually organized. In light of the known effects of perceptual buildup in streaming tasks, two analyses were carried out: one using only the triplet-locked responses from the second half of each sequence and another using all the responses to all triplets save for the first (see Materials and Methods). The results did not differ based on which analysis was used, thus only the second analysis is reported here. Significant correlations were determined by cluster-based non-parametric permutation statistics (Materials and Methods). Figure 4 shows the average triplet-locked evoked responses across an 8 × 8 grid of electrodes for the different ΔF conditions from a single patient (P4). The positions of each electrode are overlaid onto the patient's cortical surface rendering. Examples from other patients can be found in Appendix (Figures A3 and A4). As can be seen, waveform morphology was complex and highly variable between different electrode sites, yet evoked responses in varying time windows and spatial positions robustly correlated with ΔF. The majority of sites which showed strong correlations with ΔF were over or adjacent to the posterior superior temporal gyrus (pSTG). However, several other sites also showed responses which correlated with ΔF. The sites which showed significant ΔF correlations across all nine patients included in the analysis are summarized in Figure 5, where electrode sites from each individual have been overlaid onto a template brain by spherical surface registration of each patient's pial surface with that of the FreeSurfer average (see Materials and Methods). Across patients, a widespread set of brain areas showing significant correlations with ΔF included pSTG (as was expected), middle temporal gyrus, pre- and post-central gyri (mainly ventrally), inferior and middle frontal gyri, and the supra-marginal gyrus.

Figure 4

Figure 5

Evoked potentials: Bistable perception

After having established significant correlations with a physical stimulus parameter (ΔF) known to produce changes in perceptual organization, we explicitly tested whether the same electrode sites showed significant triplet-locked EP differences based solely on how the sequences were perceptually organized (i.e., we compared EPs between sequences perceived as one stream vs. two streams within a given ΔF condition). For a given ΔF, responses were binned and averaged according to whether the listener reported hearing one or two streams. As for the analysis testing for effects of ΔF, two analyses were carried out; one using only the responses from the second half of each sequence and the other using responses from the entire sequence, expect for the first. Only the results from the second analysis are presented here. The results of this analysis for individual peri-STG sites across all nine patients are shown in Figure 6. The sites, overlaid onto each individual's pial surface as shown in the top row, were chosen based on the fact that each showed a significant correlation with ΔF and was the site with largest triplet-locked RMS power in the vicinity of the pSTG. Responses to sequences that were perceptually bistable [defined as: 0.3 ≤ P(2-stream percept) ≤ 0.7] are shown by the blue (1-stream percepts) and red (2-stream percepts) traces; otherwise, traces are black. As can be seen, EP morphology was highly variable across individual subjects. Waveforms changed significantly as a function of ΔF as determined by Monte Carlo permutations using Spearman rank correlation as the sample-level statistic (see Materials and Methods), but, surprisingly, did not show significant differences based on percept per se. Across all the channels in the study, there were individual channels which showed significant differences based on percept, but this effect was inconsistent across the multiple ΔF conditions for which a bistable percept was evoked. In summary, several brain areas both within and outside of the auditory cortex showed evoked responses that significantly correlated with ΔF but not conscious perceptual organization.

Figure 6

Dissimilarity analysis

In order to further evaluate whether sites showing significant EP-ΔF correlations also showed correlates of perceptual bistability, we carried out a dissimilarity analysis using the grand-average triplet-locked response to the 0-semitone condition as the template. Responses from each ΔF condition were binned according to percept as well as collapsed across them and compared to the template by SSE (see Materials and Methods). Our hypothesis was that responses from conditions with greater ΔF – as well as responses from trials in which the subject reported hearing two streams – would show a larger “dissimilarity index” computed from the SSE between the response of interest and the template. Figure 7 shows the results of this analysis. The value of the dissimilarity index increased as ΔF increased (Spearman's rho = 0.46, p < 0.0001) and, across all channels from all patients, showed a marginally significant difference based on percept alone (W+ = 3427, p = 0.097) in the expected direction (i.e., greater dissimilarity indices for 2- vs. 1-stream percepts), suggesting a propensity for activity during 2-stream percepts to be more similar to activity evoked by large ΔF conditions. However, a sufficient number of channels (23%) showed the opposite pattern so as to limit the statistical significance of the effect. Individually, across all sites which showed a significant correlation with ΔF (N = 44), four channels showed significant effects of percept on the dissimilarity index in the expected direction, while none showed a significant effect in the opposite direction. Three of those channels were from S4 [G30, G37, and a site over the left posterior STG (not shown)] whose data are shown in Figure 4, and the fourth was from a site over the inferior post-central gyrus in S1 (not shown). None of the four channels which showed significant percept-based differences in the dissimilarity index showed significant differences in the waveforms when evaluated directly.

Figure 7

A complementary analysis was carried out using the grand-average triplet-locked response collapsed across all conditions as the template (Figure A1 in Appendix). Averaged evoked responses from each ΔF condition were binned according to percept as well as collapsed across them and compared to the template by SSE. Using this analysis, the dissimilarity index increased as ΔF increased [χ²(8,46) = 200.47, p < 0.0001] but, across all channels from all patients, did not differ based on percept alone (W+ = 1076, p = 0.33), confirming a significant main effect of ΔF and lack of a significant main effect of percept.

Gamma-power analysis

Two sets of triplet-locked high-gamma (80–190 Hz) power waveforms were constructed using either (i) wavelet transforms or (ii) analytic signal methods (see Materials and Methods). These waveforms were subjected to the same Monte Carlo permutation statistics as the triplet-locked evoked potentials to test for effects of either ΔF or percept. No significant effects were found (Figure A2 in Appendix).

Discussion

Combining a classical behavioral paradigm using long sequences of tones alternating in frequency and direct cortical recordings in humans, the present results demonstrate a widespread set of brain areas – mainly in posterosuperior temporal and peri-rolandic cortex, but also extending to the middle temporal gyrus as well as inferior and middle frontal gyri – putatively involved in auditory streaming. EP amplitude tightly correlated with ΔF, but did not consistently differ based on perceptual organization alone. Waveform morphology was highly variable within and across brain areas, suggestive of their having different roles in auditory stream formation.

Complex meso-scale activity in the auditory cortex during streaming

Results from previous M/EEG (Gutschalk et al., , ; Snyder et al., ) and fMRI (Gutschalk et al., ; Wilson et al., ) studies of streaming have suggested either a uniform role for the whole of the auditory cortex in stream formation or that the majority of activity in response to stimuli similar to those used in the present study is localized on the superior temporal plane (either on Heschl's gyrus or just posterior to it). The results from the present study demonstrate that, in addition to there being responses in higher auditory areas (i.e., lateral STG), the activity within a given macroscopic brain area is not uniform, a result that has also been noted by other investigators using evoked responses from iEEG with other classic auditory paradigms (Howard et al., ; Crone et al., ; Brugge et al., , ; Edwards et al., , ). This can be seen in the single-subject data shown in Figure 4, where the responses in adjacent electrode sites (e.g., G14 and G15 on the pSTG) indicate intra-areal variability in the response to the ABA-triplets.

This discrepancy may be due to several factors. First, the lead fields of the electrodes used to measure brain activity in the present study are more likely to measure responses from gyral crowns than from sulcal sources such as those located on the superior temporal plane (the area to where non-invasive studies have localized dipoles during streaming), although others have reported iEEG potentials interpreted to arise from sulci (Edwards et al., ; Acar et al., ; Whitmer et al., ). We observed little evidence for sources on the STP in that (i) there were very rarely clear polarity reversals across the lateral fissure and (ii) the earliest peak in the average response to sequence onset was >50 ms, later than the earliest response in the medial portion of the transverse gyrus of Heschl, which occurs at <25 ms (Liegeois-Chauvel et al., ). This last point does not preclude the possibility that some of the responses we measured arose from lateral portions of the STP, particularly in the N1-latency range (Gutschalk et al., ; Snyder et al., ). However, to us, this seems unlikely given point (i). Second, the responses we observed from the lateral STG could have radial source orientations, which would not be identified with MEG but could be with EEG. Indeed, Snyder et al. () reported radially oriented sources which could have been localized to the STG. Third, although both aforementioned fMRI studies of streaming – as well as others (Deike et al., ) – reported activation maps with multiple foci of activation, the complex relationship between auditory-evoked responses and the fMRI BOLD signal (Mukamel et al., ; Gutschalk et al., ; Mayhew et al., ; Mulert et al., , ; Steinmann and Gutschalk, ) as well as BOLD-fMRI's low temporal resolution precludes a detailed characterization of areal sub-specialization. Fourth, and perhaps most likely, the activity recorded by EEG and, to a lesser extent, MEG, represents a spatially smoothed version of the true cortical source configuration (Halgren, ; Ahlfors et al., ), and does not tend to see brain activity having response variability with high spatial frequency, contrary to the locally generated signals measured by intracranial EEG.

The role of extra-auditory areas in streaming

The present study is the first to report brain activity from extra-auditory cortical areas with high temporal resolution during the streaming paradigm. As shown in Figures 5 and 6, evoked potentials from several widespread brain areas correlated with ΔF. Waveform morphology was spatially variable both across and within macroscopic brain areas (though consistent across trials), even within individual participants, suggesting that (1) areas outside the auditory cortex may play an as-yet undetermined role in streaming and (2) the role of a given macroscopic brain area may not be uniform, known issues of ERP variability notwithstanding (e.g., Edwards et al., ).

While several authors have posited a role for areas outside the classically defined auditory pathway in streaming (Micheyl et al., ; Snyder and Alain, ; Bidet-Caulet and Bertrand, ; Elhilali et al., ), nearly all neurophysiological studies of streaming have focused exclusively on the auditory cortex (but see, Cusack, ; Pressnitzer et al., ; Kondo and Kashino, ). Only two previous studies examined whole-brain activity during the streaming paradigm (Cusack, ; Kondo and Kashino, ).

Cusack (), using a perceptually bistable sequence of tones similar to those used in the present study, reported increased BOLD activity in the intraparietal sulcus during 2-stream vs. 1-stream percepts, but did not report percept or ΔF-based differences in the auditory cortex. The present study could not assess the intraparietal sulcus given that (i) the sub-dural electrodes used were confined to superficial gyri and (ii) the lead field of sub-dural electrodes is unlikely to measure activity from as deep in the sulcus as the foci reported by Cusack. Studies utilizing methods with high temporal resolution (e.g., MEG, iEEG, or microelectrodes in experimental animals) focusing on this region could elucidate it is precise role in streaming and auditory perceptual organization more generally (e.g., Rauschecker and Scott, ; Teki et al., ). Given the results of the present study as well as previous work (Fishman et al., , ; Bee and Klump, , ; Gutschalk et al., ; Micheyl et al., ; Snyder et al., ; Wilson et al., ; Bee et al., ), it is unclear why Cusack did not observe a neurophysiological correlate of ΔF in the auditory cortex, though an account based on subtle paradigmatic differences cannot be ruled out.

Kondo and Kashino () used an event-related fMRI paradigm in order to measure brain activity during perceptual switching. Their subjects listened to tone sequences nearly identical to those used in the present study and indicated when the percept switched from one to two streams and vice versa. In addition to the auditory cortex, significant switch-related activations were found in the posterior insula, medial geniculate body, and supra-marginal gyrus. No explicit contrasts were carried out to test for effects of perceptual organization or ΔF, but the results do highlight the need for further examination of the involvement of areas outside the auditory cortex in streaming.

Our results demonstrate that the cortical areas engaged during the streaming paradigm and much more complex and widespread than has been shown by previous work, and highlights the need for detailed neurophysiological examinations of the streaming paradigm in behavioral animal models.

Failure to observe correlates of bistability

Contrary to the study of the visual system in which there are many reports of brain activity covarying directly with perception (Logothetis, ; Leopold and Logothetis, ; Sterzer et al., ), such observations are scarce in the auditory system (Hillyard et al., ; Cusack, ; Gutschalk et al., , ). By recording brain activity with high spatiotemporal precision from widespread areas of the human cortex, the present study attempted to identify neural correlates of streaming, per se, in the absence of physical stimulus differences. As mentioned above, Cusack () reported increased BOLD activity in the anterior intraparietal sulcus during 2- vs. 1-stream percepts but did not find percept- or ΔF-based differences in the auditory cortex. The latter finding is contrary to what Gutschalk et al. () reported using magnetoencephalography, namely amplitudes of the P₁m and N₁m components evoked by the B-tone in a sequence of ABA- triplets which co-varied with both ΔF and perceptual organization, per se. No evidence for activity in the intraparietal sulcus was found in that study, though this could be due to activity in the Cusack study not being precisely time-locked to the stimuli, a condition necessary for the measurement of evoked responses with EEG or MEG. Neither finding – increased activity in the intraparietal sulcus or planum temporale during 2- vs. 1-stream percepts – was replicated by the present study, possibly due to lack of coverage in the areas of activity reported by both Cusack and Gutschalk et al. (intraparietal sulcus, transverse gyrus on the superior temporal plane) or, again, that the electrical activity responsible for the generation of the BOLD effects reported by Cusack was not time-locked to the stimuli.

Possible explanations for why we did not observe robust correlates of perceptual bistability despite widespread cortical sampling (see Figure 3) are many. First, although it seems unlikely to us given the large amount of data suggesting a role for frontal areas in conscious visual perception (Libedinsky and Livingstone, ), it could be that the areas reported by Cusack () and Gutschalk et al. () are unique in maintaining representations of auditory perceptual organization and that we simply were unable to examine activity from these areas. Second, although the possibility that the known issue of trial-to-trial variability in the evoked potentials caused the lack of a significant percept-based finding cannot be ruled out, we find this explanation unlikely given the robust effects of ΔF as well as the relatively flat waveforms in the pre-sequence baseline period we observed. Finally, the neural correlates of auditory streaming could be found (i) in another cortical area not sampled, (ii) in a distributed network of brain areas which could not be determined based on the uni-variate analyses used, (iii) on a finer spatial scale than was assessed by the present study, or (iv) in an aspect of neural activity not examined such as sustained potentials or sustained gamma-band activity, though our analysis of evoked gamma-band power showed neither ΔF- or percept-based effects. This is perhaps due to the relatively constant acoustic stimulation used in our paradigm vs. the less frequent stimuli used in previous reports demonstrating large gamma-band effects (Crone et al., , ; Edwards et al., ).