Abstract
Human perception requires the many-to-one mapping between continuous sensory elements and discrete categorical representations. This grouping operation underlies the phenomenon of categorical perception (CP)—the experience of perceiving discrete categories rather than gradual variations in signal input. Speech perception requires CP because acoustic cues do not share constant relations with perceptual-phonetic representations. Beyond facilitating perception of unmasked speech, we reasoned CP might also aid the extraction of target speech percepts from interfering sound sources (i.e., noise) by generating additional perceptual constancy and reducing listening effort. Specifically, we investigated how noise interference impacts cognitive load and perceptual identification of unambiguous (i.e., categorical) vs. ambiguous stimuli. Listeners classified a speech vowel continuum (/u/-/a/) at various signal-to-noise ratios (SNRs [unmasked, 0 and −5 dB]). Continuous recordings of pupil dilation measured processing effort, with larger, later dilations reflecting increased listening demand. Critical comparisons were between time-locked changes in eye data in response to unambiguous (i.e., continuum endpoints) tokens vs. ambiguous tokens (i.e., continuum midpoint). Unmasked speech elicited faster responses and sharper psychometric functions, which steadily declined in noise. Noise increased pupil dilation across stimulus conditions, but not straightforwardly. Noise-masked speech modulated peak pupil size (i.e., [0 and −5 dB] > unmasked). In contrast, peak dilation latency varied with both token and SNR. Interestingly, categorical tokens elicited earlier pupil dilation relative to ambiguous tokens. Our pupillary data suggest CP reconstructs auditory percepts under challenging listening conditions through interactions between stimulus salience and listeners’ internalized effort and/or arousal.
Introduction
Virtually all sensory signals vary along a physical continuum, yet, we tend to perceive them as discrete perceptual objects. Such categorical perception (CP) deciphers meaningful patterns in complex sensory input by organizing information into coherent groups (equivalence classes) (). Nowhere is this phenomenon more robustly demonstrated than in speech perception. When listeners hear tokens from a phonetic continuum, their discriminability is very good for sounds straddling the category boundary near the midpoint, but very poor for sounds on the same side (Liberman et al., 1967; Pisoni, 1973; ; Pisoni and Luce, 1987; ). CP streamlines speech processing by emphasizing acoustic contrasts between- rather than within- phoneme categories (Myers and Swan, 2012), presumably by weighting cues for comparison against internalized templates of a person’s native speech sounds (; ; ; ).
Neuroimaging work has revealed neural processes leading up to categorical decisions (Sharma and Dorman, 1999; ; ; Zhang et al., 2011; ; ). In the auditory sciences, research has associated measures of perceptual performance and “listening effort,” which is the deliberate allocation of (available) mental resources to overcome goals when carrying out a listening task (for review see, Zekveld et al., 2018). Under the Framework for Understanding Effortful Listening (FUEL), listening effort is determined by the combined effect of input-demands (e.g., signal quality) and internal factors (e.g., arousal, attention, and motivation) (Pichora-Fuller et al., 2016). Accounting for the latter is crucial interpreting apparent task-related differences.
Diverse experimental techniques have shown that noise degradation has robust consequences for perceptual performance (e.g., ), short-term memory performance (e.g., ), neural activity (e.g., Scott et al., 2000), and pupil reactivity (e.g., Zekveld et al., 2011). Acoustic noise burdens cognitive load, but speech intelligibility is not always straightforwardly predicted by signal-to-noise-ratio (SNR) (for review see, ). Under the Ease of Language Understanding (ELU) model (Rönnberg et al., 2013), acoustic input that deviates from a listener’s long-term phonological memory store requires additional cognitive resources for recognition, including working memory and executive functions. The degree to which listeners engage explicit cognitive processes is thought to reflect task-related listening effort, however, cognitive resources and intrinsic motivation may be insufficient for recognition when the mismatch between percept and expectation is too extreme (Ohlenforst et al., 2017).
Segregating a speech signal from acoustic noise is cognitively demanding, drawing on resources for encoding that are normally used for other processes (). Mechanisms for signal separation might be more readily engaged when category boundaries are particularly noisy (Livingston et al., 1998). Neuroimaging data indicates that the brain processes competing sound streams within the same neural pathways, but devotes more attention to the target stream (). Our recent electrophysiological study found that neural activity was not only stronger for category (unambiguous) relative to non-category (ambiguous) speech sounds but the former was more invariant to noise interference, suggesting CP promotes robust speech perception by “sharpening” category members in noisy feature space ().
Because underlying processes are difficult to measure behaviorally, researchers have assessed listening effort with indirect measurement techniques. For example, eyetracking offers an objective glimpse into real-time speech processing () not captured by behavioral measures and self-reports (Wendt et al., 2016). One non-volitional indicator of cognitive processes is pupil reactivity (pupillometry) (see Naylor et al., 2018). Studies have reported close relations between fluctuations in pupil diameter and underlying neural mechanisms (for review see, ). Pupil diameter increases with momentary cognitive demands () and correlates closely with neuronal activity from the locus coeruleus, which is the principal brain site for synthesizing norepinephrine (i.e., arousal) (). Thus, pupil diameter indirectly indicates processes below the threshold of consciousness, which can be modulated by task demands. On a practical note, pupillometry complements other online measures of speech processing, is relatively simple to administer, and can be simultaneously registered with neurophysiological measures (e.g., for review see, Winn et al., 2018).
From the perspective of listening effort, pupillometry is an ideal avenue for investigating the physiological nature and individual differences in speech categorization. Germane to our interests in speech processing, aspects of the pupil response systematically vary with processing load when interpreting languages (), speech intelligibility (Zekveld et al., 2010), divided attention during speech listening (), semantic ambiguity (Vogelzang et al., 2016), visual-auditory semantic incongruency (Renner and Wlodarczak, 2017), and pseudoword complexity (López-Ornat et al., 2018). Relevant to this study, researchers have used pupillometry and eyetracking methods to examine how acoustically degraded speech influences listening effort (e.g., ; Winn et al., 2015). Findings have been largely consistent: peak pupil dilation and latency systematically increase with decreasing speech intelligibility, but only to the extent that cognitive resources are not overloaded (see section “Discussion”) (Zekveld et al., 2010; Zekveld and Kramer, 2014; Wendt et al., 2016; Ohlenforst et al., 2018). Assessing how pupil responses vary with listening effort could reveal how CP reconstructs auditory percepts under challenging listening conditions. Presumably, speech categorization depends on interactions between stimulus salience (Liao et al., 2016) and listeners’ internalized effort and/or arousal (for attentional dependence of CP, see ).
Here, we investigated how noise interference impacts cognitive load during perceptual identification of speech. Members of speech sound continua were presented in varying levels of noise to parametrically manipulate listening effort above and beyond that needed to classify unambiguous and ambiguous speech. Using pupillometry, we acquired continuous recordings of pupil dilation as a proxy of listening effort. If the grouping mechanisms of CP aid figure-ground perception of speech, we hypothesized unambiguous phonemes (categories) should elicit less noise-related changes in pupil responses than ambiguous tokens lacking a clear categorical identity. Our data show that the categorical nature of speech not only reduces cognitive load (listening effort) but also assists speech perception in noise degraded environments.
Methods
Participants
Fifteen young adults (3 males, 12 females; age: M = 24.3, SD = 1.7 years) from The University of Memphis participated in the experiment. All exhibited normal hearing sensitivity (i.e., <20 dB HL thresholds, 250–8000 Hz). Each participant was strongly right-handed (87.0 ± 18.2 laterality index; Oldfield, 1971) and had obtained a collegiate level of education (17.8 ± 1.9 years). Musical training enhances categorical processing and speech-in-noise listening abilities (; Yoo and Bidelman, 2019). Consequently, all participants were required to have < 3 years of music training throughout their lifetime (mean years of training: 1.3 ± 1.8 years). All were paid for their time and gave written informed consent in compliance with a protocol approved by the Institutional Review Board at the University of Memphis.
Speech Stimuli and Behavioral Task
We used a synthetic five-step vowel continuum previously used to investigate the neural correlates of CP (see Figure 1 of ; ). Each token was separated by equidistant linear steps acoustically based on first formant frequency (F1) yet was designed to be perceived categorically from /u/ to /a/. Although vowel sounds are perceived less categorically than other speech sounds (e.g., stop-consonants; Pisoni, 1973, 1975; ), they do not carry intrinsic features upon which to make category judgments (formant transitions in consonants, for example, allow comparisons within the stimulus itself) (for discussion, see Xu et al., 2006). In contrast, steady-state features like the F1 contrast of our static vowels lack an intrinsic reference so categorical hearing of these stimuli necessarily requires acoustic features be matched to the best exemplar in long-term memory (Pisoni, 1975; Xu et al., 2006). Thus, we explicitly chose vowels because they more heavily tax perceptual-cognitive processing, and therefore listening effort, as might be revealed via pupillometry.
Tokens were 100 ms, including 10 ms of rise/fall time to reduce spectral splatter in the stimuli. Each contained identical voice fundamental (F0), second (F2), and third formant (F3) frequencies (F0: 150, F2: 1090, and F3: 2350 Hz). The F1 was parameterized over five equal steps between 430 and 730 Hz such that the resultant stimulus set spanned a perceptual phonetic continuum from /u/ to /a/ (). Speech stimuli were delivered binaurally at 75 dB SPL through shielded insert earphones (ER-2; Etymotic Research) coupled to a TDT RP2 processor (Tucker Davis Technologies). This same speech continuum was presented in one of three noise blocks to vary SNR: unmasked, 0 dB SNR, −5 dB SNR. The masker was a speech-shaped noise based on the long-term power spectrum (LTPS) of the vowel set. While we typically use speech babble in our ERP studies, pilot testing showed this type of noise was too difficult for concurrent vowel identification, necessitating the use of simpler LTPS noise. The noise was presented continuously so that it was not time-locked to the stimulus presentation. Block order was randomized within and between participants.
During eyetracking, participants heard 150 trials of each speech token (per noise block). On each trial, participants labeled the sound with a binary response (“u” or “a”) as quickly and accurately as possible. Following a behavioral response, the interstimulus interval (ISI) jittered randomly between 800 and 1000 ms (20 ms steps, uniform distribution) before the next trial commenced. EEG was also recorded during the categorization task. These data are reported elsewhere ().
Pupillometry Recording and Analysis
A Gazepoint GP3 eyetracker acquired listeners’ gaze fixations based on published procedures from our laboratory (). This device provides precise measurement of the location of ocular gaze and pupil diameter with an accuracy of ∼1° visual angle via an infrared, desktop mounted camera. In addition to cognitive effort, a number of factors affect pupillometry including the pupillary light reflex () produced by the sympathetic nervous system (). Consequently, the sound booth’s lights remained off during the task. Participants could wear corrective lenses in the form of contacts. Continuous eye data were collected from the left and right eyes every 16.6 ms (i.e., 60 Hz sampling rate). MATLAB logged data from the GP3 via an API interface. Continued alignment with the screen was ensured by re-calibrating the eyetracker before each stimulus block. The GP3’s internal routine calibrated the eyes at nine-points across the horizontal/vertical dimensions of the screen.
Continuous eye data were recorded online while participants performed the auditory CP task. A central fixation cross-hair (+) remained on the computer screen during the auditory task to center and maintain participants’ gaze. Time stamps triggered in the data file demarcated the onset of each stimulus presentation. This allowed us to analyze time-locked changes in eye data for each stimulus akin to an evoked potential in the EEG literature (; ). Continuous recordings were filtered using a passband of 0.001–15 Hz, epoched [−100 to 1000 ms] (where t = 0 marks speech onset), baseline corrected, and ensemble averaged in the time domain to obtain the evoked pupil dilation response for each speech token per SNR and participant. This resulted in 15 waveforms per participant (= 5 tokens ∗ 3 SNRs). Blinks were automatically logged by the eye tracker and epochs contaminated with these artifacts were discarded prior to analysis. Additionally, to correct for subtle changes in the distance between the eyetracker camera and the participant that could affect pupil measurements (e.g., during head movement), the Gazepoint records a continuous scale factor for each pupil; a scale value = 1 represents pupil depth (distance to the camera) at the time of calibration, scaling < 1 reflects when the user is closer to the eyetracker, and a scaling > 1 when the user is further away. This scale factor was then used to weight the running time course prior to averaging and correct for movement artifacts.
Data Analysis
Behavioral Data
Identification scores were fit with a sigmoid function P = 1/[1 + e–β1(x–β0)], where P is the proportion of trials identified as a given vowel, x is the step number along the stimulus continuum, and β0 and β1 the location and slope of the logistic fit estimated using non-linear least-squares regression. Larger β1 values reflect steeper psychometric functions and stronger categorical perception. Behavioral speech labeling speeds (i.e., reaction times; RTs) were computed as listeners’ median response latency across trials for a given condition. RTs outside 250–2500 ms were deemed outliers (e.g., fast guesses, lapses of attention) and were excluded from analysis (; ).
Pupillometry Data
To quantify the physiological data, we measured the peak (maximum) pupil diameter and latency within the search window between 300 and 700 ms. Visual inspection of the waveforms showed pupil responses were maximal in this timeframe (see Figure 2). Unless otherwise specified, dependent measures were analyzed using a two-way, mixed model ANOVA (subject = random factor) with fixed effects of SNR (three levels: unmasked, 0 and −5 dB SNR) and token [five levels: vw1-5] (PROC GLIMMIX, SAS® 9.4; SAS Institute, Inc.). Tukey–Kramer and Bonferroni adjustments were used to correct subsequent post hoc and planned multiple comparisons, respectively.
Results
Behavioral Data
fully describes the behavioral results. Figure 1A shows spectrograms of the individual speech tokens and Figure 1B shows behavioral identification functions across the SNRs. An analysis of slopes (β1) revealed a main effect of SNR [F2,28 = 35.25, p < 0.0001] (Figure 1C). Post hoc contrasts confirmed that while 0 dB SNR did not alter psychometric slopes relative to unmasked speech (p = 0.33), the psychometric function became shallower with −5 dB SNR relative to 0 dB SNR (p < 0.0001). Additionally, SNR marginally but significantly shifted the perceptual boundary [F2,28 = 5.62, p = 0.0089] (Figure 1D). Relative to unmasked speech, −5 dB SNR speech shifted the perceptual boundary rightward (p = 0.011), suggesting a small but measurable bias to report “u” (i.e., more frequent vw1-2 responses) when noise exceeds the signal. Collectively, these results suggest that categorical representations are largely resistant to acoustic interference until signal strength of noise exceeds that of speech.
FIGURE 1
Behavioral response times (RTs) show the speed of categorization (Figure 1E). RTs varied with SNR [F2,200 = 11.90, p < 0.0001] and token [F4,200 = 5.36, p = 0.0004]. RTs were similar for unmasked and 0 dB SNR speech (p = 1.0) but slower for −5 dB SNR (p < 0.0001). A priori contrasts revealed this slowing was most prominent for more categorical tokens (vw1-2 and vw4-5). Ambiguous tokens (vw3) elicited similar RTs across noise conditions (ps > 0.69), suggesting that noise effects on RT were largely restricted to accessing categorical representations, not general slowing of decision speed across the board. We examined whether conditions elicited customary slowing in RTs near the midpoint of the continuum (Pisoni and Tash, 1974; Poeppel et al., 2004; ). Planned contrasts revealed this CP hallmark for unmasked [mean(vw1,2,4,5) vs. vw3; p = 0.0003] and 0 dB SNR (p = 0.0061) conditions, but not at −5 dB SNR (p = 0.59).
Pupillometry Data
Figure 2 shows grand average pupil waveforms for each speech token and SNR as well as the responses specifically contrasting unambiguous [mean (vw1,vw5)] vs. ambiguous (vw3) tokens. Visually, the data indicated that both SNR and the categorical status of speech modulated pupil responses. To quantify these effects, we pooled the peak (maximum) pupil diameter and latency of unambiguous tokens (vw1 and vw5) (those with stronger category identities) and compared them with the ambiguous vw3 token (Liebenthal et al., 2010; ; ). Figure 3 shows the mean peak pupil diameters and latencies by SNR and behavioral RTs.
FIGURE 2
FIGURE 3
An ANOVA revealed a sole main effect of SNR on peak pupil size [F2,196 = 6.69, p = 0.0015] with no token [F4,196 = 0.53, p = 0.7157] nor token∗SNR interaction effect [F8,196 = 0.16, p = 0.9959] (Figure 3A). Planned contrasts of pupil size between pairwise SNRs showed that only unmasked speech differed from intermediate SNR speech. Specifically, pupil diameter increased when classifying speech in moderate interference (i.e., 0 dB > unmasked; p = 0.0007) but did not differ with further increases in noise level (i.e., 0 dB = −5 dB; p = 0.0794) (Figure 3B).
An ANOVA on pupil latency revealed that SNR strongly modulated pupil response timing [F2,196 = 4.60, p = 0.0112], as did whether the token was unambiguous [F4,196 = 3.25, p = 0.0130] (Figures 3C,D). There was not a token∗SNR interaction effect [F8,196 = 0.94, p = 0.4827]. Follow-up contrasts revealed similar latencies for unmasked and 0 dB speech (p = 0.5379), but longer latencies at −5 dB relative to 0 dB speech (p = 0.0061). Paralleling the RT data, a priori contrasts revealed an “inverted V-shaped” pattern analogous to the behavioral data—a slowing in response timing for ambiguous relative to unambiguous tokens in the 0 dB SNR [mean(vw1,2,4,5) vs. vw3; p = 0.0244]. Unmasked and −5 dB speech did not exhibit this pattern (ps > 0.27).
To further test whether behavior modulated eye behavior, we analyzed each listener’s single-trial vw3 pupil responses based on (i) a median split of their behavioral RTs into fast and slow responses (Figures 4A–E) and (ii) the vowel category they reported (e.g., “a” vs. “u”) (Figures 4F–J). This resulted in ∼75 trials for each subaverage. Despite having been elicited by an identical (though perceptually bistable) acoustic stimulus, vw3 pupil latencies were strongly dependent on the speed of listeners’ decision [F1,70 = 6.74, p = 0.0115]. Slow RTs were associated with slower pupil responses to the ambiguous token (Figure 4E). Pupil size was not dependent on RTs [SNR, speed, and SNR × speed effects: ps ≥ 0.0585] (Figure 4D). Split by listeners’ identification (i.e., vw3 reported as “u” vs. “a”), we found a sole main effect of SNR on pupil response magnitudes [F2,70 = 3.78, p = 0.0275]. Pupil responses were again largest for 0 dB SNR speech compared to the other noise conditions (Figure 4I). These data reveal that under similar states of speech ambiguity, pupil responses are modulated according to the speed of listeners’ behavioral categorization. Note, this contrasts EEG findings for the same stimuli, which show that electrical brain activity differentiates the ambiguous speech depending on listeners’ subjective report (i.e., vw3 heard as “u” vs. “a”) ().
FIGURE 4
Discussion
By recording continuous pupil responses during a rapid speech categorization task in noise, we assessed how acoustic interference impacts cognitive load and perceptual identification of phonemes. Our analyses revealed that speech perception was robust to moderate acoustic interference (i.e., ≥ 0 dB SNR). More category representative (less ambiguous) phonetic tokens reduced listening effort and were more resilient to moderate acoustic interference. While noise impacts perception of ambiguous phonemes, categorical coding appears to mitigate interference by enhancing representations of phonemes. We propose that categorical coding (i.e., speech with an unambiguous identity) helps partially counteract the negative effects of noise on perception, but only to the extent that speech signals are not too severely degraded. Our findings converge with notions that the process of categorization aids the extraction of speech from noise whereby abstract categories help fortify the speech code and make it more resistant to external noise interference (e.g., ; ).
Physiologically, our data suggest that difficulty of speech processing modulates pupil behavior, but not straightforwardly. It is a common finding that pupil size increases when tasks are difficult to perform (). Consistent with our predictions, pupil size increased for moderately corrupted relative to unmasked speech but plateaued for severely corrupted speech. Previous work has assessed the pupil response to speech (sentences) across a broad range of intelligibility levels [i.e., −36 to −4 dB in nine 4 dB steps] (Zekveld and Kramer, 2014). This work suggests that pupil dilation increases at intermediate SNRs, but minimally at low and high SNRs, which has been interpreted to reflect intelligibility and/or task difficulty (Ohlenforst et al., 2017). The fact that pupil diameter of our participants increased with moderate SNR suggests the task demands in this condition did not exceed available cognitive resources. A recent pupillometry study found that pupil behavior correlates with subjective ratings of salience defined in terms of how noticeable or remarkable sounds are considered, indicating greater listening demand or arousal (Liao et al., 2016). In this vein, our result might reflect a performance/arousal tradeoff known as Yerkes-Dodson law, a phenomenon where performance resembles an inverted-U function of arousal (Yerkes and Dodson, 1908). Pupil dilation correlates with arousal responses measured in the locus coeruleus (LC) (). A variety of cognitive tasks elicit a strong relationship between performance and LC activity, whereby activation in the middle of the Yerkes-Dodson curve is associated with increased performance and task engagement (for reviews, see ; ; Sara and Bouret, 2012). Under this framework, listeners are less attentive and disengaged (hypoarousal) and thus perform more poorly; when LC activity increases beyond intermediate range, listeners would be more distracted (hyperarousal), which would also reduce performance. Interestingly, a neuroimaging study reported a similar finding in neural responses over left temporal cortex and premotor cortex, with greater activity for slightly degraded speech relative to unmasked and severely degraded speech (), paralleling our pupillometry results.
The most interesting findings were for pupil latency. Previous work has shown that reduced speech intelligibility systematically delays pupil responses (Zekveld et al., 2010), implying increased listening effort. While we found responses were more delayed at severe than intermediate noise levels, latencies for unmasked and intermediate speech did not differ overall (i.e., unmasked = 0 dB). Listeners may have compensated by exerting more effort in the intermediate noise condition (McGarrigle et al., 2017). Importantly, pupil responses were more categorical at intermediate SNRs, as evidenced by a slowing in pupil responses for ambiguous tokens. This pattern was not observed at −5 dB SNR. These findings suggest categorical coding helps reconstruct degraded speech sounds with unambiguous identities, but only within a limited range of intelligibility.
Behaviorally, psychometric slopes were steeper for unmasked relative to noise-degraded speech, and only became flatter for severely degraded speech. Indeed, only highly degraded speech weakened CP, further suggesting that the natural binning process of categorical coding helps maintain robust perception of SIN. Presumably, CP enhances processing within the acoustic space to help phonetic representations stand out (e.g., Nothdurft, 1991; Perez-Gay et al., 2018). We argue that noise-related decrements in CP reflect weakening of internalized categories rather than less vigilant listening across the board because ambiguous tokens elicited similar RTs across noise levels. Moreover, both our behavioral and physiological data indicated more categorical responses to unambiguous relative to ambiguous tokens at intermediate noise levels. Thus, noise-related decrements in our data likely reflect fuzzier matches between speech signals and templates of speech sounds ().
Discrepancies between the behavioral and physiological data in SNR which showed categorical coding (i.e., inverted-V pattern) suggest perhaps that pupil responses are less sensitive than behavior and require the additional “load” of intermediate noise to show a categorical effect in response timing. Additionally, while the −5 dB condition produced significantly worse behavioral performance relative to quiet, it was the 0 dB condition instead that produced larger peak pupil dilation. This could reflect the fact that the 0 dB condition was more effortful than quiet, despite behavioral accuracy remaining high. Such findings align with notions of the FUEL model (Pichora-Fuller et al., 2016) suggesting performance is governed by a combination of signal quality (e.g., input SNR) and internal factors (e.g., arousal, attention, and motivation).
One interpretation of CP is that ambiguous or intermediate tokens are “drawn” toward prototypes or category centers, i.e., the veridical percept is warped by the existence of a category representation such that peripheral tokens are perceived as more central (e.g., “perceptual magnet” theory; ; ). Our physiological data loosely align with this notion, showing and influence of category prototypicality/centrality on degraded speech perception. Peripheral tokens (e.g., vw2 and vw4) elicited similar pupil responses to their central prototype (i.e., continuum endpoints), as evidenced by the inverted-V pattern in RT (Figure 1E) and pupil latency data (Figure 3C). Still, for speech sounds which split the perceptual boundary (i.e., vw3)—and are thus perceptually ambiguous—we find this perceptual draw is considerably weaker if made at all. This is supported by the fact pupil responses to vw3 were similar when split by listeners’ subjective report (“u” vs. “a”; Figures 4F–H). Collectively, these later findings align with more relaxed models of perception which consider gradiency, whereby the system must balance the efficiency of discarding potential rich and continuous acoustic details with discrete category representations (McMurray et al., 2008). Thus, one might equally discuss our findings as reflecting the gradience of phonetic categories (especially vowels), and more generally perceptual uncertainty, rather than CP per se. Under this interpretation, acoustic cues that allow the rapid assessment of category membership of unambiguous tokens (e.g., vw1, vw5) are acoustically/perceptually available until noise masking is too egregious. In cases in which speech cues are ambiguous (vw3), noise fails to alter the decision process much, because listeners are already dealing with ambiguous acoustic-phonetic information.
Collectively, our findings converge with notions that categorical representations of phonemes are more salient and resilient to noise degradation than acoustic-sensory ones (; , ). On the premise that phonetic representations (a high-level code) are more resilient to noise than surface level features (a low-level code) (; , ), the construction of perceptual objects and natural binning process of CP might mitigate noise by helping category members stand out among a noisy feature space. Despite being acoustically dissimilar, categorically equivalent sounds would elicit similar changes in local firing rate, whereas cross-category (perceptually distinct) sounds would not (e.g., Recanzone et al., 1993; ; ). Noise would create a noisier map for physical acoustic details, but phonetic categories would persist (e.g., Nothdurft, 1991; Perez-Gay et al., 2018).
We found that ambiguous speech increased listening effort (delayed pupil responses). Results from fMRI similarly suggest that activation of auditory cortical cells may be shorter for category prototypes than for other sounds (). Indeed, participants labeled unambiguous tokens more quickly than ambiguous tokens, suggesting more efficient processing of members from well-formed categories. This advantage was also observed in pupil latencies in the intermediate noise condition, but not in the unmasked condition. Delayed pupil responses might instead reflect processes of ambiguity resolution. In speech, there is no one-to-one correspondence between any single acoustic cue and phonetic representations (Lotto and Holt, 2016). Partial loss of acoustic cues would render phonemes highly confusable with one another. Connectionist models of speech perception such as TRACE (McClelland and Elman, 1986) posit bi-directional, interactive activation of phonemic traces that help recover meaning when signal features are missed. Under TRACE, speech processing transpires through a neuronal network representing speech features at increasingly higher levels. Incoming acoustic input activates nodes for features (and inhibits others), which in turn activate phonemes at the next level. During this process, traces of inhibited representations remain activated for a period, helping the listener recover information if errors are perceived (e.g., missing an acoustic segment). If noise leads to partial loss of cues, delayed pupil responses observed in our data might reflect ongoing activation (through a TRACE-like network) of multiple phonetic representations in attempt to disambiguate what is being heard.
In sum, the present findings demonstrate that pupillometry can be used as an effective technique for assessing underlying processes of speech perception and categorical processing. Here, the benefits of tracking CP with pupillometry were twofold: (a) providing complementary physiological data for comparison with existing data, and (b) lending temporally sensitive insight into mental processes not available from behavioral measures alone.
Statements
Data availability statement
The datasets generated for this study are available on request to the corresponding author.
Ethics statement
The studies involving human participants were reviewed and approved by The University of Memphis IRB. The participants provided their written informed consent to participate in this study.
Author contributions
Both authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.
Funding
This work was supported by the National Institute on Deafness and Other Communication Disorders of the National Institutes of Health under award number R01DC016267 (GB).
Acknowledgments
We thank Lauren Bush and Alex Boudreaux for assistance in data collection.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
1
AltmannC. F.UesakiM.OnoK.MatsuhashiM.MimaT.FukuyamaH. (2014). Categorical speech perception during active discrimination of consonants and vowels.Neuropsychologia64C13–23. 10.1016/j.neuropsychologia.2014.09.006
2
AndreassiJ. L. (ed.). (2000). “Pupillary Response and Behavior,” in Psychophysiology: Human Behavior & Physiological Response.New Jersey: Lawrence Erbaulm Associates, Inc, 289–307.
3
Aston-JonesG.CohenJ. D. (2005). An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance.Annu. Rev. Neurosci.28403–450. 10.1146/annurev.neuro.28.061604.135709
4
BeattyJ. (1982). Task-evoked pupillary responses, processing load, and the structure of processing resources.Psychol. Bull.91276–292. 10.1037//0033-2909.91.2.276
5
Ben-DavidB. M.ChambersC. G.DanemanM.Pichora-FullerM. K.ReingoldE. M.SchneiderB. A. (2011). Effects of aging and noise on real-time spoken word recognition: Evidence from eye movements.J. Speech, Lang. Hea. Res.54243–262. 10.1044/1092-4388(2010/09-0233)
6
BerridgeC. W.WaterhouseB. D. (2003). The locus coeruleus–noradrenergic system: modulation of behavioral state and state-dependent cognitive processes.Brain Res. Rev.4233–84. 10.1016/s0165-0173(03)00143-7
7
BidelmanG. M. (2015). Induced neural beta oscillations predict categorical speech perception abilities.Brain Lang.14162–69. 10.1016/j.bandl.2014.11.003
8
BidelmanG. M.BrownB.MankelK.PriceC. N. (2019a). Psychobiological responses reveal audiovisual noise differentially challenges speech recognition.Ear. Hear.10.1097/AUD.0000000000000755[Epub ahead of print].
9
BidelmanG. M.BushL. C.BoudreauxA. M. (2019b). The categorical neural organization of speech aids its perception in noise.bioRxiv [preprint]. 10.1101/652842
10
BidelmanG. M.SigleyL.LewisG. (2019c). Acoustic noise and vision differentially warp speech categorization.J. Acoust. Soc. Am.14660–70. 10.1121/1.5114822
11
BidelmanG. M.LeeC.-C. (2015). Effects of language experience and stimulus context on the neural organization and categorical perception of speech.Neuroimage120191–200. 10.1016/j.neuroimage.2015.06.087
12
BidelmanG. M.MorenoS.AlainC. (2013). Tracing the emergence of categorical speech perception in the human auditory system.Neuroimage79201–212. 10.1016/j.neuroimage.2013.04.093
13
BidelmanG. M.WalkerB. (2017). Attentional modulation and domain specificity underlying the neural organization of auditory categorical perception.Eur. J. Neurosci.45690–699. 10.1111/ejn.13526
14
BidelmanG. M.WeissM. W.MorenoS.AlainC. (2014). Coordinated plasticity in brainstem and auditory cortex contributes to enhanced categorical speech perception in musicians.Eur. J. Neurosci.402662–2673. 10.1111/ejn.12627
15
BidelmanG. M. (2017). “Communicating in challenging environments: noise and reverberation,” in Springer Handbook of Auditory Research: The frequency-following response: A window into human communication. vol Springer Handbook of Auditory Research, Vol. 61edsKrausN.AndersonS.White-SchwochT.FayR. R.PopperA. N. (New York, N.Y: Springer Nature).
16
BinderJ. R.LiebenthalE.PossingE. T.MedlerD. A.WardB. D. (2004). Neural correlates of sensory and decision processes in auditory object identification.Nat. Neurosci.7295–301. 10.1038/nn1198
17
ChangE. F.RiegerJ. W.JohnsonK.BergerM. S.BarbaroN. M.KnightR. T. (2010). Categorical speech representation in human superior temporal gyrus.Nat. Neurosci.131428–1432. 10.1038/nn.2641
18
CousinsK. A.DarH.WingfieldA.MillerP. (2014). Acoustic masking disrupts time-dependent mechanisms of memory encoding in word-list recall.Mem. Cogn.42622–638. 10.3758/s13421-013-0377-7
19
DavisM. H.JohnsrudeI. S. (2003). Hierarchical processing in spoken language comprehension.J. Neurosci.233423–3431. 10.1523/jneurosci.23-08-03423.2003
20
EcksteinM. K.Guerra-CarrilloB.Miller SingleyA. T.BungeS. A. (2017). Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development?Dev. Cogn. Neurosci.2569–91. 10.1016/j.dcn.2016.11.001
21
EvansS.McGettiganC.AgnewZ. K.RosenS.ScottS. K. (2016). Getting the cocktail party started: masking effects in speech perception.J. Cogn. Neurosci.28483–500. 10.1162/jocn_a_00913
22
FanX.YaoG. (2011). Modeling transient pupillary light reflex induced by a short light flash.IEEE Trans. Biomed. Eng.5836–42. 10.1109/TBME.2010.2080678
23
GatehouseS.GordonJ. (1990). Response times to speech stimuli as measures of benefit from amplification.Br. J. Audiol.2463–68. 10.3109/03005369009077843
24
GoldstoneR. L.HendricksonA. T. (2010). Categorical perception.Wiley Interdiscip. Rev.Cogn. Sci.169–78. 10.1002/wcs.26
25
GuentherF. H.GjajaM. N. (1996). The perceptual magnet effect as an emergent property of neural map formation.J. Acoust. Soc. Am.100(2 Pt 1), 1111–1121. 10.1121/1.416296
26
GuentherF. H.Nieto-CastanonA.GhoshS. S.TourvilleJ. A. (2004). Representation of sound categories in auditory cortical maps.J. Speech Lang. Hear Res.4746–57. 10.1044/1092-4388(2004/005)
27
HarnadS. R. (1987). Categorical Perception: The Groundwork of Cognition.New York, NY: Cambridge University Press.
28
HeinrichA.SchneiderB. A.CraikF. I. (2008). Investigating the influence of continuous babble on auditory short-term memory performance.Q. J. Exp. Psychol.61735–751. 10.1080/17470210701402372
29
HelieS. (2017). The effect of integration masking on visual processing in perceptual categorization.Brain Cogn.11663–70. 10.1016/j.bandc.2017.06.001
30
HyönäJ.TommolaJ.AlajaA.-M. (1995). Pupil dilation as a measure of processing load in simultaneous interpretation and other language tasks.Q. J. Exp. Psychol.48598–612. 10.1080/14640749508401407
31
IversonP.KuhlP. K.Akahane-YamadaR.DieschE.TohkuraY.KettermannA.et al (2003). A perceptual interference account of acquisition difficulties for non-native phonemes.Cognition87B47–B57.
32
KahnemanD.BeattyJ. (1966). Pupil diameter and load on memory.Science1541583–1585. 10.1126/science.154.3756.1583
33
KillionM. C.NiquetteP. A.GudmundsenG. I.RevitL. J.BanerjeeS. (2004). Development of a quick speech-in-noise test for measuring signal-to-noise ratio loss in normal-hearing and hearing-impaired listeners.J. Acoust. Soc. Am.116(4 Pt 1), 2395–2405. 10.1121/1.1784440
34
KoelewijnT.Shinn-CunninghamB. G.ZekveldA. A.KramerS. E. (2014). The pupil response is sensitive to divided attention during speech processing.Hear. Res.312114–120. 10.1016/j.heares.2014.03.010
35
KuhlP. K. (1991). Human adults and human infants show a “perceptual magnet effect” for the prototypes of speech categories, monkeys do not.Percept. Psychophys.5093–107. 10.3758/bf03212211
36
LiaoH.-I.KidaniS.YoneyaM.KashinoM.FurukawaS. (2016). Correspondences among pupillary dilation response, subjective salience of sounds, and loudness.Psychon. B Rev.23412–425. 10.3758/s13423-015-0898-0
37
LibermanA. M.CooperF. S.ShankweilerD. P.Studdert-KennedyM. (1967). Perception of the speech code.Psychol. Rev.74431–461.
38
LiebenthalE.DesaiR.EllingsonM. M.RamachandranB.DesaiA.BinderJ. R. (2010). Specialization along the left superior temporal sulcus for auditory categorization.Cereb. Cortex202958–2970. 10.1093/cercor/bhq045
39
LivingstonK. R.AndrewsJ. K.HarnadS. (1998). Categorical perception effects induced by category learning.J. Exp. Psychol.Learn. Mem. Cogn.24732–753. 10.1037//0278-7393.24.3.732
40
López-OrnatS.KarousouA.GallegoC.MartínL.CameroR. (2018). Pupillary measures of the cognitive effort in auditory novel word processing and short-term retention.Front. Psychol.9:2248. 10.3389/fpsyg.2018.02248
41
LottoA. J.HoltL. L. (2016). Speech perception: the view from the auditory system. in Neurobiology of LanguageedsHickokG.SmallS. L.. (Amsterdam:: Elsevier), 185–194.
42
McClellandJ. L.ElmanJ. L. (1986). The TRACE model of speech perception.Cognit. Psychol.181–86. 10.1016/0010-0285(86)90015-0
43
McGarrigleR.DawesP.StewartA. J.KuchinskyS. E.MunroK. J. (2017). Measuring listening-related effort and fatigue in school-aged children using pupillometry.J. Exp. Child Psychol.16195–112. 10.1016/j.jecp.2017.04.006
44
McMurrayB.AslinR. N.TanenhausM. K.SpiveyM. J.SubikD. (2008). Gradient sensitivity to within-category variation in words and syllables.J. Exp. Psychol. Hum. Percept. Perform.341609–1631. 10.1037/a0011747
45
MyersE. B.SwanK. (2012). Effects of category learning on neural sensitivity to non-native phonetic categories.J. Cogn. Neurosci.241695–1708. 10.1162/jocn_a_00243
46
NaylorG.KoelewijnT.ZekveldA. A.KramerS. E. (2018). The Application of Pupillometry in Hearing Science to Assess Listening Effort.Los Angeles, CA: SAGE Publications Sage CA.
47
NothdurftH. C. (1991). Texture segmentation and pop-out from orientation contrast.Vision Res.311073–1078. 10.1016/0042-6989(91)90211-m
48
OhlenforstB.WendtD.KramerS. E.NaylorG.ZekveldA. A.LunnerT. (2018). Impact of SNR, masker type and noise reduction processing on sentence recognition performance and listening effort as indicated by the pupil dilation response.Hear. Res.36590–99. 10.1016/j.heares.2018.05.003
49
OhlenforstB.ZekveldA. A.LunnerT.WendtD.NaylorG.WangY.et al (2017). Impact of stimulus-related factors and hearing impairment on listening effort as indicated by pupil dilation.Hear. Res.35168–79. 10.1016/j.heares.2017.05.012
50
OldfieldR. C. (1971). The assessment and analysis of handedness: the Edinburgh inventory.Neuropsychologia997–113. 10.1016/0028-3932(71)90067-4
51
Perez-GayF.SicotteT.TheriaultC.HarnadS. (2018). Category learning can alter perception and its neural correlate.arXiv [Preprint]. Available at: https://arxiv.org/abs/1805.04619(accessed November 4, 2019).
52
Pichora-FullerM. K.KramerS. E.EckertM. A.EdwardsB.HornsbyB. W.HumesL. E.et al (2016). Hearing impairment and cognitive energy: the framework for understanding effortful listening (FUEL).Ear. Hear.375S–27S. 10.1097/AUD.0000000000000312
53
PisoniD. B. (1973). Auditory and phonetic memory codes in the discrimination of consonants and vowels.Percept. Psychophys.13253–260. 10.3758/bf03214136
54
PisoniD. B. (1975). Auditory short-term memory and vowel perception.Mem. Cognit.37–18. 10.3758/BF03198202
55
PisoniD. B.LuceP. A. (1987). Acoustic-phonetic representations in word recognition.Cognition2521–52. 10.1016/0010-0277(87)90003-5
56
PisoniD. B.TashJ. (1974). Reaction times to comparisons within and across phonetic categories.Percept. Psychophys.15285–290. 10.3758/bf03213946
57
PoeppelD.GuilleminA.ThompsonJ.FritzJ.BavelierD.BraunA. R. (2004). Auditory lexical decision, categorical perception, and FM direction discrimination differentially engage left and right auditory cortex.Neuropsychologia42183–200. 10.1016/j.neuropsychologia.2003.07.010
58
RecanzoneG. H.SchreinerC. E.MerzenichM. M. (1993). Plasticity in the frequency representation of primary auditory cortex following discrimination training in adult owl monkeys.J. Neurosci.1387–103. 10.1523/jneurosci.13-01-00087.1993
59
RennerL. F.WlodarczakM. (2017). When a Dog is a Cat and How it Changes Your Pupil Size: Pupil Dilation in Response to Information Mismatch.Stockholm: INTERSPEECH, 2017, 674–678.
60
RönnbergJ.LunnerT.ZekveldA.SörqvistP.DanielssonH.LyxellB.et al (2013). The Ease of Language Understanding (ELU) model: theory, data, and clinical implications.Front. Syst. Neurosci.7:31. 10.3389/fnsys.2013.00031
61
SaraS. J.BouretS. (2012). Orienting and reorienting: the locus coeruleus mediates cognition through arousal.Neuron76130–141. 10.1016/j.neuron.2012.09.011
62
ScottS. K.BlankC. C.RosenS.WiseR. J. (2000). Identification of a pathway for intelligible speech in the left temporal lobe.Brain1232400–2406. 10.1093/brain/123.12.2400
63
SharmaA.DormanM. F. (1999). Cortical auditory evoked potential correlates of categorical perception of voice-onset time.J. Acoust. Soc. Am.1061078–1083. 10.1121/1.428048
64
VogelzangM.HendriksP.van RijnH. (2016). Pupillary responses reflect ambiguity resolution in pronoun processing.Lang. Cogn. Neurosci.31876–885. 10.1080/23273798.2016.1155718
65
WendtD.DauT.HjortkjaerJ. (2016). Impact of background noise and sentence complexity on processing demands during sentence comprehension.Front. Psychol.7:345. 10.3389/fpsyg.2016.00345
66
WinnM. B.EdwardsJ. R.LitovskyR. Y. (2015). The impact of auditory spectral resolution on listening effort revealed by pupil dilation.Ear. Hear.36e153–e165. 10.1097/AUD.0000000000000145
67
WinnM. B.WendtD.KoelewijnT.KuchinskyS. E. (2018). Best practices and advice for using pupillometry to measure listening effort: an introduction for those who want to get started.Trends Hea.22:2331216518800869. 10.1177/2331216518800869
68
XuY.GandourJ. T.FrancisA. (2006). Effects of language experience and stimulus complexity on the categorical perception of pitch direction.J. Acoust. Soc. Am.1201063–1074. 10.1121/1.2213572
69
YerkesR. M.DodsonJ. D. (1908). The relation of strength of stimulus to rapidity of habit−formation.J. Comp. Neurol. Psychol.18459–482. 10.1002/cne.920180503
70
YooJ.BidelmanG. M. (2019). Linguistic, perceptual, and cognitive factors underlying musicians’ benefits in noise-degraded speech perception.Hear. Res.377189–195. 10.1016/j.heares.2019.03.021
71
ZekveldA. A.KoelewijnT.KramerS. E. (2018). The pupil dilation response to auditory stimuli: current state of knowledge.Trends Hear.22:2331216518777174. 10.1177/2331216518777174
72
ZekveldA. A.KramerS. E. (2014). Cognitive processing load across a wide range of listening conditions: insights from pupillometry.Psychophysiology51277–284. 10.1111/psyp.12151
73
ZekveldA. A.KramerS. E.FestenJ. M. (2010). Pupil response as an indication of effortful listening: the influence of sentence intelligibility.Ear. Hear.31480–490. 10.1097/AUD.0b013e3181d4f251
74
ZekveldA. A.KramerS. E.FestenJ. M. (2011). Cognitive load during speech perception in noise: the influence of age, hearing loss, and cognition on the pupil response.Ear. Hear.32498–510. 10.1097/AUD.0b013e31820512bb
75
ZhangL.XiJ.XuG.ShuH.WangX.LiP. (2011). Cortical dynamics of acoustic and phonological processing in speech perception.PLoS One6:e20963. 10.1371/journal.pone.0020963
Summary
Keywords
pupillometry, categorical perception, speech-in-noise (SIN) perception, listening effort, eye behavior
Citation
Lewis GA and Bidelman GM (2020) Autonomic Nervous System Correlates of Speech Categorization Revealed Through Pupillometry. Front. Neurosci. 13:1418. doi: 10.3389/fnins.2019.01418
Received
04 June 2019
Accepted
16 December 2019
Published
10 January 2020
Volume
13 - 2019
Edited by
Yi Du, Institute of Psychology (CAS), China
Reviewed by
Alexander Francis, Purdue University, United States; Jackson Everett Graves, University of Minnesota Twin Cities, United States
Updates
Copyright
© 2020 Lewis and Bidelman.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Gavin M. Bidelman, gmbdlman@memphis.edu
This article was submitted to Auditory Cognitive Neuroscience, a section of the journal Frontiers in Neuroscience
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.