# TRANSITIONS BETWEEN CONSCIOUSNESS AND UNCONSCIOUSNESS

EDITED BY: Marcus Rothkirch, Morten Overgaard and Guido Hesselmann PUBLISHED IN: Frontiers in Psychology

#### *Frontiers Copyright Statement*

*© Copyright 2007-2018 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

> *All copyright, and all rights therein, are protected by national and international copyright laws.*

*The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88945-431-0 DOI 10.3389/978-2-88945-431-0

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# **TRANSITIONS BETWEEN CONSCIOUSNESS AND UNCONSCIOUSNESS**

Topic Editors:

**Marcus Rothkirch**, Charité-Universitätsmedizin Berlin, Germany **Morten Overgaard**, Aarhus University, Denmark **Guido Hesselmann**, Charité-Universitätsmedizin Berlin, Germany

Image: Yvonne Daschowski.

Over the last years, a large body of experimental data have been generated in the attempt to understand consciousness and its neural underpinnings. In this respect, particular interest has been paid to the attempt to distinguish between conscious experience and unconscious states which however may still be considered as mental states (e.g., in virtue of their representational nature). This is of course not without reason. A deep understanding of that which specifically characterizes conscious states, including neural correlates and cognitive functions, may crucially inform the ambition of understanding the relation between experience and the physical world. Nevertheless, the question has historically been challenged by the fact that consciousness is available in the first person only – not to other people, including scientists. Different methodological traditions and choices have led to quite different understandings of how conscious and unconscious states relate, and diverse empirical work has been inspired and guided by various cognitive and neurobiological theories of consciousness. The very diverse viewpoints include such different positions as the idea that unconscious states are associated with the very same functional characteristics as conscious states, and the idea that no informational state that is available for action can be completely unconscious. The Research Topic "Transitions between consciousness and unconsciousness" is therefore devoted to this particular question, how to understand the relation and transition between consciousness and unconsciousness. We hope that the reader will find the collected articles both informative and thought-provoking, and that this Research Topic will stimulate the scientific debate.

**Citation:** Rothkirch, M., Overgaard, M., Hesselmann, G., eds. (2018). Transitions Between Consciousness and Unconsciousness. Lausanne: Frontiers Media. doi: 10.3389/978-2-88945-431-0

# Table of Contents


*91 Neural Correlates of Subjective Awareness for Natural Scene Categorization of Color Photographs and Line-Drawings*

Qiufang Fu, Yong-Jin Liu, Zoltan Dienes, Jianhui Wu, Wenfeng Chen and Xiaolan Fu

*104 Relative Spatial Frequency Processing Drives Hemispheric Asymmetry in Conscious Awareness*

Elise A. Piazza and Michael A. Silver

### **Chapter 3: How Does Consciousness Emerge?**

*111 The Emergence of Visual Awareness: Temporal Dynamics in Relation to Task and Mask Type*

Markus Kiefer and Thomas Kammer

*125 In and Out of Consciousness: How Does Conscious Processing (D)evolve Over Time?*

Jaan Aru and Talis Bachmann

*129 A Sensorimotor Signature of the Transition to Conscious Social Perception: Co-regulation of Active and Passive Touch*

Hiroki Kojima, Tom Froese, Mizuki Oka, Hiroyuki Iizuka and Takashi Ikegami

*143 Oscillatory Correlates of Visual Consciousness* Stefano Gallotto, Alexander T. Sack, Teresa Schuhmann and Tom A. de Graaf

# **Chapter 4: To What Extent Can Rules and Regularities Be Learned Unconsciously?**

*159 Impact of Response Stimulus Interval on Transfer of Non-local Dependent Rules in Implicit Learning: An ERP Investigation*

Jianping Huang, Hui Dai, Jing Ye, Chuanlin Zhu, Yingli Li and Dianzhi Liu

*169 The Emergence of Explicit Knowledge in a Serial Reaction Time Task: The Role of Experienced Fluency and Strength of Representation* Sarah Esser and Hilde Haider

# **Chapter 5: How Can the Conscious Representation of Visual Input Be Dissociated from its Consequences?**

*187 Subliminal Impending Collision Increases Perceived Object Size and Enhances Pupillary Light Reflex*

Lihong Chen, Xiangyong Yuan, Qian Xu, Ying Wang and Yi Jiang

# **Chapter 6: Which High-Level Functions Can Proceed Unconsciously?**


# **Chapter 7: Strategies for Future Research**

*212 What We Talk about When We Talk about Unconscious Processing – A Plea for Best Practices*

Marcus Rothkirch and Guido Hesselmann

# Editorial: Transitions between Consciousness and Unconsciousness

#### Marcus Rothkirch<sup>1</sup> , Morten Overgaard<sup>2</sup> and Guido Hesselmann<sup>1</sup> \*

<sup>1</sup> Visual Perception Laboratory, Department of Psychiatry and Psychotherapy, Charité—Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany, <sup>2</sup> Center for Functionally Integrative Neuroscience, Aarhus University, Aarhus, Denmark

Keywords: consciousness, neural correlates of consciousness, unconscious processing, awareness, diversity

#### **Editorial on the Research Topic**

#### **Transitions between Consciousness and Unconsciousness**

Over the last years, a large body of experimental data have been generated in the attempt to understand consciousness and its neural underpinnings. In this respect, particular interest has been paid to the attempt to distinguish between conscious experience and unconscious states which however may still be considered as mental states (e.g., in virtue of their representational nature). This is of course not without reason. A deep understanding of that which specifically characterizes conscious states, including neural correlates and cognitive functions, may crucially inform the ambition of understanding the relation between experience and the physical world. Nevertheless, the question has historically been challenged by the fact that consciousness is available in the first person only—not to other people, including scientists. Different methodological traditions and choices have led to quite different understandings of how conscious and unconscious states relate (e.g., Rothkirch and Hesselmann, this research topic), and diverse empirical work has been inspired and guided by various cognitive and neurobiological theories of consciousness. The very diverse viewpoints include such different positions as the idea that unconscious states are associated with the very same functional characteristics as conscious states (e.g., Hassin, 2013), and the idea that no informational state that is available for action can be completely unconscious (Overgaard and Mogensen, 2014, 2015).

The Research Topic "Transitions between consciousness and unconsciousness" is therefore devoted to this particular question, how to understand the relation and transition between consciousness and unconsciousness. It comprises 18 articles from different backgrounds, including original studies, as well as reviews and commentaries, which speaks to the multifaceted research in this field. In the following, we will provide a brief summary for each contribution.

One of the most appealing questions in the field of consciousness research is whether stimuli that cannot be consciously perceived by the observer can nevertheless influence the observer's behavior, and if, to what extent they do so. Four submissions to the research topic approached this fascinating question while focusing on different processes and behavioral outcomes. One fruitful approach to study the aforementioned unconscious influences is masked priming. Priming refers to the observation that the response to a target stimulus can be influenced by the presentation of an irrelevant prime stimulus prior to the target. In masked priming, in particular, the prime stimulus is not consciously perceived. On the basis of this paradigm, Goller et al. investigated affective priming, indicating that an incongruence between prime and target entails a negative evaluation of a neutral symbol following the target. The authors observed that such affective priming effects were stronger for unconscious than for conscious primes, which they interpreted as a misattribution of the prime-target

#### Edited and reviewed by:

Samuel Evans, University of Westminster, United Kingdom

\*Correspondence: Guido Hesselmann guido.hesselmann@charite.de

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 30 December 2017 Accepted: 09 January 2018 Published: 24 January 2018

#### Citation:

Rothkirch M, Overgaard M and Hesselmann G (2018) Editorial: Transitions between Consciousness and Unconsciousness. Front. Psychol. 9:20. doi: 10.3389/fpsyg.2018.00020 incongruence to the unrelated neutral symbol. In a similar vein, Khalid and Ansorge used masked priming to study the subliminal processing of faces displaying disgust. Using low- and high-pass filtered faces as prime stimuli, the authors intended to identify a potential subcortical origin of the priming effect. Surprisingly, however, they found a reversed priming effect, such that a prime-target congruence led to slower reaction times. This effect was further restricted to conditions in which attention was diverted away from the prime. This points toward a unique unconscious effect of disgusting faces, which does not seem to rely on subcortical pathways. While also studying the effect of subliminal facial expressions, Winkielman and Gogolushko focused on another, "higher-level" behavioral outcome, namely the consumption of a beverage. When primed with a positive facial expression participants tended to consume more than after being primed with a negative expression. This effect was observed for supra- and subliminal primes and restricted to pictorial primes (in comparison to words). Finally, Ruch et al. could show that subliminally presented information can also impact decision-making. In a first phase, faces were presented subliminally along with written high- or low-wage occupations. These faces were presented again in a second phase, this time supraliminally and either with congruous or incongruous occupations in comparison to the first presentation. A later recall phase showed that both the previous sub- and supraliminally presented information biased participants' decision on the incomes of the depicted person.

A further relevant question in the realm of consciousness research is how stimuli get access to awareness and, in particular, how specific stimulus attributes facilitate this process. Based on previous findings indicating a hemisphere-specific processing of spatial frequencies, Piazza and Silver set out to test whether the awareness of spatial frequency information also differs between the two hemispheres. Using binocular rivalry to approach this question, the authors demonstrate that the visual system's classification of high and low spatial frequencies, and therefore the hemisphere preferentially processing the given frequency, is dependent on other, concurrently presented spatial frequencies. This indicates that a relative rather than an absolute processing of spatial frequencies contributes to hemispheric differences in perceptual selection. Another popular technique to study the access to awareness is breaking continuous flash suppression. Noel et al. applied this technique to investigate the preferential processing of self-relevant stimuli. While they did not observe such a preference for self- vs. non-self-related words, they found that participants' response criterion for the categorization of these words (i.e., self vs. non-self) was dependent on an acoustic signal administered either within or outside the peripersonal space. However, a common observation in studies using breaking continuous flash suppression is the high variability between participants. What is more, as Gayet and Stein demonstrate, the magnitude of reaction time differences between conditions is highly correlated with each individual's overall suppression times. As a remedy, the authors advocate the usage of a simple latency normalization method, which also yields reaction time distributions that are better suited for parametric testing.

Besides focusing on specific stimulus attributes that facilitate awareness, one can also ask in a broader sense how consciousness develops over time. Is the transition between unconscious and conscious states a gradual or dichotomous phenomenon? This is one of the long-standing questions that has steered heated debates in consciousness research. Using backward masking with word targets, Kiefer and Kammer varied the context by the modulation of the task and the mask type. From their results, the authors concluded that the emergence of awareness is neither purely gradual nor dichotomous but rather depends on the specific parameters of the task and the type of mask. In the context of social interactions, Kojima et al. were specifically interested in how people become aware of the presence of others. Measures of turn-taking and movement synchrony that were assessed during a social interaction paradigm indicated that the awareness of the other's presence was reciprocally coregulated by both agents. At the neural level, the identification of brain processes that are related to or even causally determine conscious experiences has received attention under the term "the neural correlates of consciousness." Measuring event-related potentials (ERPs) during a backward masking paradigm, Fu et al. addressed the question whether visual awareness is related to a visual awareness negativity (VAN). The authors found that ERP components were related to visual awareness for color photographs but not for line drawings. Furthermore, the VAN varied with visual awareness in a linear fashion while positive late potentials changed in a non-linear way, indicating that different ERP components are related to different types of visual awareness. In their review paper, Gallotto et al. provide a basic overview of neural oscillations and ways to measure them. They also emphasize that the distinction between the neural prerequisites, substrates, and consequences of conscious experience remains a major challenge for future research. Also with respect to the general time course of consciousness, many aspects have been left unclear to date, as pointed out by Aru and Bachmann. These open questions are particularly related to the form of the functions that describe how preconscious content accesses consciousness but also how a conscious representation can decay again. The authors particularly point toward the importance of the context, which is in line with the aforementioned findings by Kiefer and Kammer.

We received two studies addressing the question to what extent rules and regularities can be learned unconsciously. Huang et al. used ERPs to explore the impact of the responsestimulus-interval (RSI) on the transfer of knowledge of abstract implicit rules. Only at the longer of two RSIs, participants were able to acquire abstract implicit knowledge. Furthermore, the results suggest that amplitude variations of the N200 and P300 components of the ERP may be useful for detecting transferrelated effects. Esser and Haider investigated how unconscious knowledge becomes conscious knowledge in the serial reaction time task (SRTT). Regular (i.e., in line with the rule) and deviant trials (i.e., violating the rule) were either presented in mini-blocks or randomly mixed. While the degree of implicit knowledge, as assessed on the basis of a wagering task, was not affected by the presentation order, the subjectively experienced fluency was higher for the presentation in mini-blocks. More explicit knowledge was gathered for longer mini-blocks. The authors interpret their findings in light of the unexpected event hypothesis, according to which explicit knowledge arises from the observation of one's own changes in behavior that, in turn, are based on implicit learning.

In their pupillometry study, Chen at al. asked how the conscious representation of visual input can be dissociated from its consequences. They presented looming spheres on the screen such that they would either collide with the observer or miss the observer's head by a small margin. Participants had the task to either judge the size of the stimulus or to decide whether the stimulus would have collided with them. In all experiments (with the exception of the first one), participants could not distinguish between collisions and nearmisses. The results showed that participants judged the size of the colliding stimuli as being bigger than the near-miss stimuli, and pupil constrictions turned out to be greater for the colliding stimuli. The authors conclude that threatening stimuli may influence visual perception without necessarily evoking a conscious representation of the threat.

In addition, one general commentary and one original research article were concerned with the scope of unconscious high-level cognitive functions. In their commentary, Goldstein and Hassin follow up on the debate about the "Yes It Can" (YIC) principle (Hassin, 2013; Hesselmann and Moors, 2015). According to YIC, unconscious processes can carry out every fundamental high-level function that conscious processes can perform. As the authors point out, one of the implications of YIC is that the search for a "holy grail" —i.e., the function that only consciousness can do—is the wrong way to go. Instead, understanding what it means to be human would be better achieved by understanding how unconscious processes pursue functions that only they can pursue. In their study, Garrison and Handley tested the hypothesis that "unconscious thought"

REFERENCES


(Dijksterhuis and Nordgren, 2006) is distinct from intuitive processes, and thus might be rational. The authors manipulated participants' reliance on the experiential versus rational system (Epstein, 1994) and found that a period of distraction facilitated outcomes independently from these two processing modes. They also manipulated unconscious thought (during a distraction phase) toward solving a logical reasoning problem, and observed that unconscious thought was superior at this analytical task, suggesting that unconscious thought may be rational. As the authors point out, however, the concept of "unconscious thought" itself remains controversial (Nieuwenstein et al., 2015).

The main goal of this Research Topic "Transitions between consciousness and unconsciousness" was to provide a snapshot of the current state of affairs in this research area. The final collection of 18 articles does exactly that and provides an overview of current trends and opinions, as well as perspectives on theoretical and methodological questions. As pointed out by two of us in a perspective article, research on conscious and unconscious processes is characterized by a large diversity of methods, measures, statistical analyses, and concepts. The same holds true for this collection. We hope that the reader will find the collected articles both informative and thought-provoking, and that this Research Topic will stimulate the scientific debate.

# AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

# FUNDING

MR and GH were supported by the German Research Foundation (grants: RO 4836/2-1 and HE 6244/1-2).

Overgaard, M., and Mogensen, J. (2014). Conscious perception: A representational, non-reductionistic, level-dependent approach. Philos.

Trans. R. Soc. Lond. Ser B Biol. Sci. 369:20130209. doi: 10.1098/rstb.2013.0209 Overgaard, M., and Mogensen, J. (2015). Reconciling current approaches to blindsight. Conscious. Cogn. 32, 33–40. doi: 10.1016/j.concog.2014.08.003

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Rothkirch, Overgaard and Hesselmann. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# A Double Dissociation between Conscious and Non-conscious Priming of Responses and Affect: Evidence for a Contribution of Misattributions to the Priming of Affect

#### Florian Goller<sup>1</sup> \*, Shah Khalid<sup>2</sup> and Ulrich Ansorge<sup>1</sup>

<sup>1</sup> Faculty of Psychology, University of Vienna, Vienna, Austria, <sup>2</sup> Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany

#### Edited by:

Guido Hesselmann, Charité – Universitätsmedizin Berlin, Germany

#### Reviewed by:

Roy Salomon, Bar-Ilan University, Israel Marine Vernet, National Institute of Mental Health (NIH), USA

> \*Correspondence: Florian Goller florian.goller@univie.ac.at

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 31 October 2016 Accepted: 10 March 2017 Published: 27 March 2017

#### Citation:

Goller F, Khalid S and Ansorge U (2017) A Double Dissociation between Conscious and Non-conscious Priming of Responses and Affect: Evidence for a Contribution of Misattributions to the Priming of Affect. Front. Psychol. 8:453. doi: 10.3389/fpsyg.2017.00453 Studies have demonstrated conscious and non-conscious priming of responses and of affect. Concerning response priming, presenting a target-related (congruent) distractor prior to a target typically facilitates target responses. This facilitation – the responsepriming effect – is observed in comparison to a less related (incongruent) distractor. An incongruent distractor would interfere with the required response to the target. This response-priming effect is found with both conscious distractors, of which participants are aware, and non-conscious distractors, of which participants are not aware. In partly related research, distractors have also yielded affective priming effects on the evaluations of task-unrelated neutral symbols that followed the target: In comparison to the congruent condition, participants evaluated a neutral symbol presented after an incongruent distractor-target sequence as more negative. This affective priming effect was sometimes ascribed to the participants' misattributions of distractor-target conflict to the unrelated neutral symbols. Here, we set out to test this possibility. If the misattribution explanation of affective priming holds true, affective priming would be stronger with non-conscious than with conscious distractors: Mostly the non-conscious distractors would mask distractor-target conflict as the true affect-origin and, therefore, invite participants' misattribution of the primed affect to the neutral symbol in temporal vicinity. In contrast, only with conscious distractors, participants would be aware of distractor-target conflict as the true affect-origin and should, therefore, be better able to attribute their affective responses to the distractor-target relationship itself. In three experiments, we confirmed this prediction of a stronger affective priming effect in non-conscious than conscious distractor conditions, while at the same time showing conscious response-priming effects to even exceed non-conscious response-priming effects. Together, these results amount to a double dissociation between affective priming, being stronger with unconscious distractors, and response priming, being stronger with conscious distractors. This double dissociation supports the misattribution explanation and makes clear that the amount of distractor-elicited response conflict alone does not account for the amount of affective priming. Moreover, the participants' unawareness of the distractors is critical for the amount of affective priming of neutral symbols in temporal vicinity.

Keywords: flanker task, masking, non-conscious processing, misattributions of affect, conflict

# INTRODUCTION

fpsyg-08-00453 March 23, 2017 Time: 15:30 # 2

Have you ever snapped at a friend for no apparent reason? Have you ever felt sad without knowing why? These examples illustrate that humans can experience negative (or positive) affective responses without an awareness of the objects that elicited the affective responses. In the present article, affective responses are defined as the actual positive or negative impressions that are elicited by an object, and the true affect-origin is defined as the object that triggered these impressions. As our examples imply, humans are sometimes uncertain about the true origins of their affective responses, an observation in line with emotion theories. For example, Arnold (1960) explained that emotions can be triggered without an awareness of their true origins. In general, Arnold believed that emotions are based on appraisals of objects or events as positive or negative. These appraisals correspond to what we called positive and negative affective responses. In the case of an automatic emotion generation, Arnold argued that the appraisal process is hidden from introspection, meaning that participants do not have to become aware of the true affectorigins. In more recent emotion theories, this claim was repeated. A number of researchers claimed that feelings that are initially elicited are only subsequently attributed to an object as its origin (Schachter and Singer, 1962; Weiner, 1986). For instance, in the feeling-as-information theory (Schwarz, 1990; see also Ortony et al., 1990), the feeling is the first state of awareness that informs a person about her own appraisal of the object that triggered the feeling–that is, before her feeling, this person would not know whether or not s/he liked an object (i.e., evaluated an object as positive or negative).

Considered from the perspective of such emotion theories, a person's uncertainty about the true affect-origin in general, and the unawareness of the true affect-origin in particular, could be favorable side conditions of a misattribution of affective responses to alternative objects than the true affect-origin (cf. Zajonc, 1968; Murphy and Zajonc, 1993; Payne et al., 2010). Over the course of three experiments, we tested this hypothesis. We primed affective responses by distractors of which our participants were unaware (i.e., with visually masked distractors) or aware (i.e., with clearly visible distractors) and tested whether participants' unawareness of the masked distractors as the true affect-origins is a favorable precondition for misattributions of affective responses away from the masked distractors and toward neutral objects in close temporal vicinity.

In our experiments, we used response priming to elicit affective responses (cf. Fritz and Dreisbach, 2013; see also van Steenbergen et al., 2010). In each trial of our experiments, participants had to discriminate between the orientations of a centrally presented target. If the current trial's target was a square, participants had to press one key, and if the target was a diamond, they had to press another key. Prior to the central target, we presented two distractors at target-adjacent positions. In congruent trials, the distractor was of the same shape as the target and, thus, indicated the same response as the target. In incongruent trials, the distractor was of the alternative shape as compared to the target and, thus, led to response conflict between distractor and target. In this experimental situation, past research has demonstrated that, in comparison to congruent distractors, incongruent distractors delay the response to the target. This response-priming effect has been found with conscious distractors, of which participants were aware (Eriksen and Eriksen, 1974; Gratton et al., 1988), and with non-conscious distractors, of which participants were not aware (Schwarz and Mecklinger, 1995). However, with non-conscious distractors, this response-priming effect is typically weaker than with conscious distractors (Tapia et al., 2010).

Critically, in incongruent conditions a response conflict also elicits more negative affective responses than in congruent conditions where a response conflict is absent. This affective priming effect is measured in evaluations of neutral symbols that follow the distractor and the target in close temporal vicinity. Affective priming of neutral symbols through response conflict has hitherto been demonstrated with conscious distractors in a Stroop paradigm only (Fritz and Dreisbach, 2013). In their experiment, participants showed more negative ratings of a neutral Chinese symbol if presented after an incongruent Stroop (a mismatch between a color word and the print color of the word, e.g., GREEN printed in red) than after a congruent (GREEN printed in green) Stroop. Studies showing that affective priming of neutral objects have sometimes been ascribed to misattributions of affective responses away from the eliciting stimulus and toward the neutral symbol or object (Payne et al., 2005).

Here, we tested this misattribution explanation of affective priming. If misattributions account for affective priming, affective priming could be stronger with non-conscious than with conscious affect-origins. The reason is that with conscious affect-origins, participants are well aware of degree of the distractor-target incongruence as the true affect-origin. This makes it less likely that the participants misattribute the same affective response elicited by the distractor-target pair also to a different neutral symbol following the distractor and target (cf. Oikawa et al., 2011). In contrast, with non-conscious affect-origins, participants are not aware of the distractor-target incongruence as the true affect-origin. This could foster the misattribution of the distractor-elicited affective response to the neutral symbol that follows the distractor and target. In other words, affective priming effects for a neutral symbol

could be stronger with non-conscious distractors than with conscious distractors, despite the fact that response priming could be stronger with conscious distractors than with nonconscious distractors. Together, these expectations thus amount to the prediction of a double dissociation between response priming and affective priming by non-conscious and conscious distractors.

#### EXPERIMENT 1

The aim of our study was to assess the influence of conscious and non-conscious conflict on the evaluation of a neutral symbol. Conflict was elicited by target-distractor congruence in a flanker task (Eriksen and Eriksen, 1974). We chose a flanker task because non-conscious conflict is established very well in this task (Schwarz and Mecklinger, 1995; Tapia et al., 2010) and is less controversial than with other congruence tasks (e.g., Stroop tasks, see Kouider and Dupoux, 2004).

To measure affective priming, participants additionally rated a neutral Chinese symbol as positive or negative at the end of some trials. If target-distractor incongruence elicits more negative affective responses than target-distractor congruence and if these affective responses are (sometimes) misattributed, we expected more negative evaluations of the symbols in incongruent than congruent trials. Furthermore, a failure to perceive the true affectorigin could increase the likelihood of misattributions, resulting in more frequent affect misattributions following non-conscious than conscious distractors (cf. Murphy and Zajonc, 1993).

At the end of each block, we also measured the participants' awareness of the distractors. This was done to ensure that our participants were indeed not aware of the masked, non-conscious distractors, as well as to ensure that our participants were aware of the clearly visible, conscious distractors. In this awareness test, we applied the chance-level performance criterion of unawareness (Klotz and Neumann, 1999; Schmidt and Vorberg, 2006). In each trial of the awareness test, participants had to decide if they though the current prime was a square or a diamond. In this awareness test, chance performance means that the participants were unable to successfully discriminate between the distractor shapes and, thus, that the participants were unaware of the distractors. In contrast, above chance performance could mean that (1) masking was not perfect, such that on some trials the distractors were visible and participants were aware of them, or that (2) there distractors are processed in a consciousnessindependent way, allowing above chance performance without conscious access to them. Despite these limitations, we expected chance-level performance in the non-conscious condition but better than chance performance in the conscious condition.

## Methods

#### Participants

Twenty-eight participants (15 female, 13 male, MAge = 24.05, SDAge = 8.64) were tested. We calculated this sample size and all following sample sizes by using G∗Power (Faul et al., 2009) under the assumption of a small to medium effect size and a statistical power of 90%. Two participants were excluded due to more than 20% errors. Here and in all following experiments, participants received course credit, were right-handed, had normal or corrected-to-normal vision, and reported no prior experience with Chinese symbols. Prior to all experiments, all participants gave written consent and were informed that participation and data collection were fully anonymous. Participants could withdraw at any time during the experiment without any further consequences. All studies were conducted in accordance with the Declaration of Helsinki (revised, 1983) and the guidelines of the Faculty of Psychology, University of Vienna. We further followed the Austrian Universities Act, 2002 (UG2002) – which was active at the time of the experiments – which required only medical universities to appoint ethics committees for clinical testing, application of medical methods and applied medical research. Therefore, no additional ethical approval was sought.

#### Apparatus and Stimuli

The experiment was programmed and controlled using Matlab 7.7.0 (The MathWorks inc., Natick, MA, USA) and the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997). Viewing distance was stable at 57 cm, supported by a chin and forehead rest. Responses were manual key presses with the left versus right index finger on a keyboard. Target responses were given through the keys 'f' and 'j'. Targets and distractors (1.5◦ × 1.5◦ ) were squares and diamonds (Klotz and Neumann, 1999). To decrease participants' awareness of the distractors, in the non-conscious condition, we used circular metacontrast masks that were neutral with respect to the response-relevant angular target shapes (2◦ × 2 ◦ ). A fixation-cross (0.5◦ × 0.5◦ ) was displayed at screen center. Targets appeared always on the screen's vertical meridian, 6.2◦ equally likely above or below the fixation-cross. Distractors and masks were placed equally distant (2.2◦ ) left and right of the target. In half of the trials, target and distractors were of the same shape (congruent trials). In the other half of the trials, target and distractors were of different shapes (incongruent trials). All stimuli were colored black (CIE Lab: 0.9/−0.3; 0.8 cd/m<sup>2</sup> ) and were presented against a gray background (CIE Lab: 0.9/−8.4; 31.1 cd/m<sup>2</sup> ).

Randomly, one third of the trials were followed by a rating task with a black Chinese symbol (4◦ × 4 ◦ ) at screen center. The symbols were randomly selected from an online English-Chinese dictionary<sup>1</sup> . In a pre-study, fresh participants rated a total set of 700 symbols on valence, arousal, and complexity. To rule out a priori evaluation differences between the symbols used, we selected symbols with moderate ratings on these dimensions. Also, by randomization of the symbols across trials, each specific symbol was equally likely used in congruent and incongruent trials. We verified this assumption by conducting chi-square tests of the occurrence of each individual symbol on each level of the variables consciousness (conscious vs. non-conscious) and congruence (congruent vs. incongruent). None of these tests were significant, all χ <sup>2</sup> < 2.59, all p < 0.108.

#### Procedure

After the fixation display (800 ms), distractors were added (40 ms), and then, with an inter-stimulus interval (ISI) of 50 ms, a target was shown for 100 ms (see also **Figure 1**). Participants

were first tested in the non-conscious block, and afterward in the conscious block. The order of blocks was kept constant to ensure participants' minimal awareness of the non-conscious distractors prior to the distractor-awareness test at the end of the experiment (see below). Participants pressed the left or right key, with each key mapped to a specific target shape (counterbalanced across participants). An on-screen feedback informed about too slow (>1 s) or erroneous responses. In one third of the trials, following target discrimination, participants rated the valence of a Chinese symbol as either negative (left key) or positive (right key). This mapping was the same for all participants since the dominant hand seems to be associated with positive concepts (Casasanto, 2009). In total, the experiment consisted of 768 trials.

In a separate part at the end of the experiment, participants' awareness of the distractors was assessed. Participants were informed about the presence and identities of the distractors, and their task was to discriminate and report distractor identities. Across trials, stimulus-to-response (S-R) mapping varied randomly (announced via presentation of a post-target onscreen mapping rule). S-R mapping rules varied from trial to trial and were only specified after the targets to ensure that the awareness test was exclusive – that is, that the awareness test was only sensitive for the conscious perception of the primes (cf. Reingold and Merikle, 1988). If we would have used a fixed S-R mapping rule for different distractor shapes, which the participants knew in advance of the distractors, awarenessindependent response specification by the masked distractors could have contributed to the number of correct responses in the awareness test (see Experiment 4 of Neumann and Klotz, 1994). This would have yielded a non-exclusive measure of distractor awareness – that is, an "awareness score" that does not deserve this label as it would have reflected both awareness-dependent and awareness-independent contributions of distractor processing. As only awareness was to be measured by this test, awareness-independent contributions had to be ruled out. In total, the distractor visibility assessment block consisted of 320 trials. In general, independence of awareness was to be ensured by participants' chance-level performance in this awareness-test block. However, one problem with this kind of criterion is that a non-significant p-value of a t-test against chance performance (50%) or zero does not inform us about whether truly no effect was found or our test was insensitive to the effect (see Dienes, 2014). To resolve this problem, we additionally conducted Bayesian Factor (BF) analysis of the corresponding t-tests (Rouder et al., 2009), with an R scale of 1. Here we reported the scaled JZS Bayes factor values. They indicate the relation between the probabilities of the data being in favor of the null relative to being in favor of the alternative hypothesis (or vice versa). On the basis of Jeffreys (1961) convention, a Bayes factor greater than 3.00 was considered as a substantial evidence in favor of the null or the alternative hypothesis accordingly (Dienes, 2011).

#### Results

#### Reaction Times (RTs)

Trials with erroneous responses (8.80%) were analyzed separately (see below). Mean correct reaction times (RTs) were subjected to a repeated-measurements analysis of variance (ANOVA), with the within-participant variables consciousness (conscious; nonconscious) and congruence (congruent; incongruent). Where appropriate, degrees of freedom were Greenhouse-Geisser corrected. For better transparency, the uncorrected degrees of freedom but the corrected p-values are reported. **Figure 2** (left panel) illustrates the results.

standard error of the mean (SEM).

Besides main effects of consciousness, F(1,25) = 73.55, p < 0.001, η 2 <sup>p</sup> = 0.75, and congruence, F(1,25) = 102.12, p < 0.001, η 2 <sup>p</sup> = 0.80, we also found an interaction, F(1,25) = 40.86, p < 0.001, η 2 <sup>p</sup> = 0.62. In the conscious block, mean RT was lower in congruent (492 ms) than incongruent trials (548 ms), t(25) = −12.73, p < 0.001, d = 3.53. The same was true of the non-conscious block (congruent: 562 ms; incongruent: 582 ms), t(25) = −4.08, p < 0.001, d = 1.13, but the congruence effect (incongruent RT minus congruent RT) was significantly larger in the conscious (56 ms) than in the non-conscious (20 ms) block, t(25) = 6.39, p < 0.001, d = 1.77.

Since the to-be-rated symbol was only shown in a third of all trials, we conducted a separate, complementary analysis of the RTs on only those trials in which a Chinese symbol had to be rated. The results were essentially the same as above. We had two significant main effects and an interaction (all F > 20.13, all p < 0.001). Again, RTs were faster in congruent than incongruent trials in both the conscious (489 ms vs. 550 ms), t(25) = −10.07, p < 0.001, d = 2.79), and the non-conscious block (563 ms vs. 582 ms), t(25) = −2.55, p < 0.001, d = 0.71). Additionally, the congruence effect was larger in the conscious (61 ms) than the non-conscious (19 ms) block, t(25) = 4.49, p < 0.001, d = 1.24.

#### Error Rates (ERs)

The arcsine transformed error rates (ERs) were subjected to a similar ANOVA as was used for RTs. A main effect of congruence, F(1,25) = 40.30, p < 0.001, η 2 <sup>p</sup> = 0.62, interacted with consciousness, F(1,25) = 39.74, p < 0.001, η 2 <sup>p</sup> = 0.61. No main effect of consciousness was found, F = 0.27. In the conscious block, we found higher ERs in incongruent (11.54%) than congruent (4.70%) trials, t(25) = −7.39, p < 0.001, d = 2.05. No such difference was found in the non-conscious block (7.05% vs. 7.51%), t(25) = −1.23, p = 0.230, d = 0.34.

#### Ratings of the Chinese Symbols

From all correct flanker-task trials, we computed the probabilities of negative ratings of the Chinese symbols for each participant and combination of consciousness and congruence. Across all conditions, the probability of a negative rating was not significantly different from 50%, t-test against 50%: t(25) = 1.27, p = 0.215, d = 0.35, BF = 3.10 in favor of the null hypothesis. The same was true for separate analyses of the non-conscious, t(25) = 1.16, p = 0.124, d = 0.45, BF = 3.50 in favor of the null hypothesis, and the conscious, t(25) = 1.02, p = 0.316, d = 0.28, BF = 4.03 in favor of the null hypothesis, blocks. This indicates that on average symbols were rated relatively neutral, meaning that no general a priori trend for negative or positive ratings of the Chinese symbols was present.

A repeated-measurements ANOVA of the arcsine transformed probabilities of negative ratings revealed an interaction between congruence and consciousness, F(1,25) = 9.75 p = 0.004, η 2 <sup>p</sup> = 0.28. No significant main effects were found, both F < 0.61, both p < 0.249. **Figure 2** (right panel) illustrates the results. In the non-conscious block, participants rated the symbols more often as negative in incongruent trials (50.00%) than congruent trials (43.65%), t(25) = 3.46, p = 0.002, d = 0.96, BF = 17.33 in favor of the alternative hypothesis. Most notably, no such difference was found in the conscious block (congruent: 48.27%; incongruent: 45.09%), t(25) = 1.27, p = 0.216, d = 0.35, BF = 3.10 in favor of the null hypothesis.

#### Distractor Awareness

Awareness of the distractors was assessed by d' (Green and Swets, 1966/1974). Scores of d' were obtained from direct calculation of hit rates and false alarm rates. For the calculation of d', diamonds counted as signals and squares as noise. Here, d' is the z-transformed false alarm rate subtracted from the z-transformed hit rate. This index becomes zero in the case of chance

performance (i.e., unawareness), and it can infinitely increase with ever increasing discrimination performance. Performance was above chance in conscious trials (d' = 1.05), t-test against zero: t(25) = 5.32, p < 0.001, d = 1.47, BF = 1435.32 in favor of the alternative hypothesis, and not different from chance in non-conscious trials (d' = 0.08), t-test against zero: t(25) = 0.78, p > 0.249, d = 0.22, BF = 4.94 in favor of the null hypothesis.

To exclude the possibility of a response bias, or more importantly, different response biases in the conscious and nonconscious trials, we also analyzed the criterion β, which is calculated as the ratio between the likelihood of choosing one response in signal trials and the likelihood of choosing the same response in a noise trial (Stanislaw and Todorov, 1999). This ratio becomes 1 if participants favor neither response over the other. The β values were not significantly different in conscious (β = 1.07) and non-conscious (β = 1.18) trials, t(25) = 0.41, p > 0.249, d = 0.11, BF = 6.10 in favor of the null hypothesis.

#### Discussion

We see from the chance performance in the distractor-awareness tests of the non-conscious block that participants were not aware of the masked distractors. Although non-conscious distractors elicited a response conflict, as shown in the RTs, the true origin of this conflict in the target-distractor relation, therefore, remained unknown to the participants. This should have led to an active search for an alternative origin of the conflictelicited affect–that is, an alternative origin to the distractor-target congruence versus incongruence. In line with this expectation, neutral Chinese symbols following the targets were judged more often as negative following a non-conscious incongruent than following a non-conscious congruent distractor. Participants evidently misattributed the origin of their negative affect to a neutral stimulus in close temporal vicinity: the Chinese symbol.

Crucially, in line with a supportive role of unawareness of the true affect-origins for affect misattribution, no affect misattributions were found following conscious distractors: With conscious congruent and incongruent distractors, equal proportions of negative and positive judgments were observed, although the RTs were clearly influenced by conscious targetdistractor congruence/incongruence, too: Conscious distractors produced an even stronger effect on RTs than non-conscious distractors (for a similar finding see Tapia et al., 2010). Together with the results of the non-conscious block, the data amount to a double dissociation between response priming and affective priming. While response priming was stronger in conscious than non-conscious blocks, affective priming was stronger in non-conscious than conscious blocks. This means that the difference in terms of affective priming was not simply due to a stronger response-congruence effect in the non-conscious block.

A stronger misattribution effect with non-conscious than conscious distractors is in line with our predictions (see also Murphy and Zajonc, 1993, for an exposure-elicited affect misattribution). However, an entire lack of affect misattributions following conscious distractors is at variance with affect misattributions following conscious Stroop trials in the study of Fritz and Dreisbach (2013). One decisive difference between the present study and the one by Fritz and Dreisbach (2013) was that participants did not respond to the Stroop target itself but only had to judge the Chinese symbol following the Stroop target. In contrast, in the rating trials of the present experiment, participant first had to respond to the target of the flanker task and then rated the Chinese symbol following the target. Therefore, the use of no-go trials prior to the symbol ratings in the study of Fritz and Dreisbach (2013) was maybe critical for affect misattributions in conscious target-distractor trials. For instance, responding correctly to the flanker targets in the incongruent trials of the present study could have changed our participants' evaluation of conflict from negative to more positive (Schouppe et al., 2015). Therefore, our participants could have felt more positive about their successful response to a target following a conscious incongruent distractor than the participants of Fritz and Dreisbach (2013). Additionally, our participants might have felt less positive about their correct responses to non-conscious incongruent distractors simply because our participants would have failed to register the non-conscious incongruent distractor as a challenge in the first place. Consequently, only the present conscious incongruent distractors, but not the non-conscious incongruent distractors might have prompted positive feelings of success that counteracted the negative affective responses (and their misattributions to the symbols).

One last point needs also mentioning: Our participants might have associated the response key with the valence conveyed by the distractor-target pair (see Gawronski and Ye, 2014), meaning that a repetition of the response key (e.g., right key for square and the same right key for a positive evaluation) might have influenced the ratings in some cases. To clarify these issues, we conducted Experiment 2, in which our participants rated the Chinese symbols in go and no-go trials, the former replicating our Experiment 1, the latter being more similar to the experimental setup of Fritz and Dreisbach (2013).

# EXPERIMENT 2

We used a go/no-go flanker task, where participants had to react to a target shape (e.g., a square, the go target) and had to withhold their response if the alternative target shape was presented (e.g., a diamond, the no-go target). Shortly before the target, the distractors were presented in the same manner as in Experiment 1. A no-go distractor is incongruent to the go target and accordingly delays the go response (for congruence effects with non-conscious distractors in the go/no-go task see Experiment 5 of Ansorge, 2004). We expected an RT congruence effect in the go trials, and affect misattributions based on congruence versus incongruence in go and no-go trials. If the participants' unawareness of the distractors facilitated affect misattributions, we expected more misattributions following non-conscious than conscious distractors. However, if in Experiment 1 overt correct responses to conscious incongruent distractors changed the evaluations and prevented a misattribution effect, we expected more misattributions following conscious distractors in the present no-go than go trials.

# Methods

#### Participants

Twenty-eight participants were tested. Two participants were post hoc excluded due to more than 20% errors in the flanker task. The final sample consisted of 17 females and 9 males (MAge = 23.46, SDAge = 3.59).

#### Apparatus and Stimuli

fpsyg-08-00453 March 23, 2017 Time: 15:30 # 7

These were the same as in Experiment 1, with two exceptions. To reduce visual crowding and task difficulty, we reduced distractor and target eccentricities to 4.5◦ (6.2◦ in Experiment 1), and slightly increased target-distractor distances to 2.5◦ (2.0◦ in Experiment 1).

#### Procedure

The sequence of events in a trial was as in Experiment 1 (see **Figure 1**), but the task was changed. If the target was a diamond (or a square, counterbalanced across participants), participants pressed the spacebar. Otherwise, they had to withhold their response and to wait until the next trial started. There was a fixed time window of 1 s for participants to give their answer (in go trials) or to wait (in no-go trials). Otherwise the task and the procedure were the same as in Experiment 1.

# Results

#### Reaction Times (RTs)

All error trials (7.55%) were excluded and analyzed separately (see below). Main correct go-trial RTs were subjected to a repeated-measurements ANOVA, with the within-participant variables consciousness (conscious; non-conscious) and congruence (congruent; incongruent), analogous to Experiment 1. The results are shown in **Figure 3** (left panel). Besides significant main effects of consciousness, F(1,25) = 11.14, p = 0.003, η 2 <sup>p</sup> = 0.31, and congruence, F(1,25) = 54.58, p < 0.001, η 2 <sup>p</sup> = 0.69, we also found an interaction between these variables, F(1,25) = 8.19, p = 0.008, η 2 <sup>p</sup> = 0.25. Similar to Experiment 1, RTs were faster in congruent than incongruent trials for both the non-conscious (congruent: 547 ms; incongruent: 572 ms), t(25) = 5.08, p < 0.001, d = 1.41, and the conscious (congruent: 517 ms; incongruent: 558 ms), t(25) = 7.25, p < 0.001, d = 2.01) block. Again, the RT congruence effect (incongruent RT minus congruent RT) was significantly larger in conscious (41 ms) than non-conscious (25 ms) blocks, t(25) = 2.86, p = 0.008, d = 0.79. An analysis restricted to only the trials where a symbol was rated, yielded essentially the same results.

#### Error Rates (ERs)

The arcsine transformed mean ERs were subjected to a similar ANOVA, with the additional variable of task (go; no-go). The main effects of task, F(1,25) = 11.84, p = 0.002, η 2 <sup>p</sup> = 0.32, and congruence, F(1,25) = 41.36, p < 0.001, η 2 <sup>p</sup> = 0.62, as well as the interactions between consciousness and congruence, F(1,25) = 4.83, p = 0.037, η 2 <sup>p</sup> = 0.16, and task and congruence, F(1,25) = 11.26, p = 0.003, η 2 <sup>p</sup> = 0.31, are best explained by looking at the interaction between all three factors, F(1,25) = 8.62, p = 0.007, η 2 <sup>p</sup> = 0.26. No other main effects or interactions were found, all non-significant F < 0.07.

To explore the significant three-way interaction, we conducted separate ANOVAs for the go and no-go trials. In the go trials, we obtained a significant main effect of congruence, F(1,25) = 38.36, p < 0.001, η 2 <sup>p</sup> = 0.61, and an interaction with consciousness, F(1,25) = 9.53, p = 0.005, η 2 <sup>p</sup> = 0.28. No main effect for consciousness was found, F = 0.10. In the conscious block, ERs were higher in incongruent (11.18%) than congruent (2.69%) trials, t(25) = 5.73, p < 0.001, d = 1.59. In the nonconscious block, a similar pattern was found (incongruent: 8.71%; congruent: 4.66%), t(25) = 3.58, p = 0.001, d = 0.99. Analogous to the RTs, the ER congruence effect was larger in the conscious (8.48%) than in the non-conscious (4.04%) block, t(25) = 3.09, p = 0.005, d = 0.86.

In the no-go trials, only a main effect of congruence was found, F(1,25) = 11.37, p = 0.002, η 2 <sup>p</sup> = 0.31, indicating a higher ER in incongruent (4.43%) than congruent (2.84%) trials. No other effects were found for the no-go trials, all non-significant F < 0.02.

#### Ratings of the Chinese Symbols

The analysis was based on correct flanker-task trials only. No overall bias toward negative (or positive) evaluations was found, t-test against 50%: t(25) = −0.49, p > 0.249, d = 0.13, BF = 5.89 in favor of the null hypothesis. The same holds almost true if tests were conducted separately for the non-conscious block, t(25) = −1.76, p = 0.090, d = 0.49, BF = 1.60 in favor of the null hypothesis–this value is less than 3, and therefore not substantial evidence–, and the conscious block, t(25) = −0.52, p > 0.249, d = 0.14, BF = 5.81 in favor of the null hypothesis. A repeated-measurements ANOVA, with the same variables as in the ERs, revealed a main effect of consciousness, F(1,25) = 4.84, p = 0.037, η 2 <sup>p</sup> = 0.16, and an interaction between consciousness and congruence, F(1,25) = 9.16, p = 0.006, η 2 <sup>p</sup> = 0.27. In the non-conscious block, participants rated the symbols less often as negative in congruent (43.13%) than incongruent (49.00%) trials, t(25) = 2.88, p = 0.008, d = 0.80, BF = 4.88 in favor of the alternative hypothesis. There was no such difference in the conscious block (congruent: 52.51%; incongruent: 50.71%), t(25) = 1.46, p = 0.157, d = 0.41, BF = 2.45 in favor of the null hypothesis. Furthermore, we found a strong trend toward a main effect of task, F(1,25) = 4.06, p = 0.055, η 2 <sup>p</sup> = 0.14, with a lower proportion of negative ratings in go trials (47.16%) than no-go trials (50.51%). A follow-up Bayesian t-test with a Bayes factor of 1.84 indicates, however, that this trend is not entirely conclusive. No other effects were found, all non-significant F < 3.13, all p > 0.089. **Figure 3** (right panel) illustrates the results.

#### Distractor Awareness

Distractor discrimination was above chance in the conscious trials (d' = 1.32), t-test against zero: t(25) = 6.96, p < 0.001, d = 1.93, BF = 68177.77 in favor of the alternative hypothesis, and not different from chance in the non-conscious trials (d' = 0.07), t(25) = 0.64, p > 0.249, d = 0.18, BF = 5.43 in favor of the null hypothesis. No difference in the response bias was found between the conscious trials (β = 1.06) and non-conscious trials

are indicated on the x-axis. The separate lines illustrate conscious (solid lines) and non-conscious (dashed lines) trials. Error bars represent SEM.

(β = 0.99), t(25) = 1.20, p = 0.240, d = 0.33, BF = 3.35 in favor of the null hypothesis.

#### Discussion

Experiment 2 confirmed that affect misattributions were restricted to the non-conscious distractors. With conscious distractors, congruence-elicited affect misattributions were missing in go and no-go trials. The results are in line with a critical role of unawareness for affect misattributions to the neutral symbols (as in Experiment 1). In contrast, successful conscious discrimination of the incongruent distractors just prior to a symbol was not responsible for lacking affect misattributions following conscious distractors. Otherwise, we would have found a difference between the ratings of the symbols in go and no-go trials.

We also observed some evidence for another congruenceindependent type of affect misattribution to neutral symbols that was based on events of which the participants were aware: a trend in the ANOVA toward more negative ratings of the symbols in no-go than go trials. To note, the Bayesian test revealed that this trend is not entirely convincing. However, if this effect were real, these affective responses probably reflected "distractor devaluation," which denotes that (no-go) stimuli requiring to be ignored or associated with (response) inhibition are evaluated more negatively than (go) targets (Raymond et al., 2003; Fenske et al., 2005). Because this effect, if it existed, was due to the no-go targets but measured in the evaluations of the symbols, devaluation would have been due to affect misattribution, too. Yet, participants were aware of the no-go status of the targets as indicated by their low error rates in no-go trials, and, thus, participants must have been aware of the true affect-origins in these cases. Therefore, even in the present study not all affect misattributions would have depended on the participants' unawareness of the true affect-origins (see also Payne et al., 2005; Fritz and Dreisbach, 2013).

### EXPERIMENT 3

Experiment 2 essentially replicated the results of Experiment 1. However, both experiments shared one major caveat: Participants were first tested in the non-conscious block and afterward in the conscious block. We used a fixed block order to minimize participants' distractor awareness or distractor suspicion during the non-conscious block. Therefore, we cannot be sure whether our effects were truly caused by the manipulation of consciousness or whether block order was (partly) responsible for our results. Maybe participants were just more used to the method in the later occurring conscious blocks than in the earlier occurring non-conscious blocks. For instance, if the participants had learned to actively ignore the unwanted affective priming influence of the target-distractor congruence on the symbol ratings, this influence would have had a greater effect on the affective priming effect in the conscious than in the non-conscious block.

To address this issue, we conducted a control experiment in which we replicated Experiment 2, but manipulated distractor consciousness between participants, not within participants. We decided to use a between-participants design to rule out all possible transfer effects between the conscious and nonconscious blocks. To state the results of Experiment 3 right from the outset: We essentially mirrored the results of Experiment 2. Hence, our results truly stem from the manipulation of distractor awareness.

# Methods

#### Participants

Thirty-six participants were tested, 18 in the conscious condition (12 females and 6 males, MAge = 21.00, SDAge = 1.88) and 18 in the non-conscious condition (16 females, 2 males, MAge = 22.00, SDAge = 6.65).

#### Apparatus, Stimuli, and Procedure

fpsyg-08-00453 March 23, 2017 Time: 15:30 # 9

These were the same as in Experiment 2, with the notable difference that participants were tested only in the conscious or only in the non-conscious block.

# Results

#### Reaction Times (RTs)

All errors (8.59%) were excluded and analyzed separately (see below). Mean correct go-trial RTs were subjected to a mixedmodel ANOVA, with the within-participant variable congruence (congruent, incongruent) and the between-participants variable consciousness (conscious, non-conscious). A significant main effect of congruence, F(1,34) = 82.71, p < 0.001, η 2 <sup>p</sup> = 0.71, interacted with consciousness, F(1,34) = 6.21, p = 0.018, η 2 <sup>p</sup> = 0.15. No main effect of consciousness was found, F(1,34) = 0.12. As in Experiments 1 and 2, RTs were shorter in congruent than incongruent trials for both the non-conscious (congruent: 558 ms; incongruent: 581 ms), t(17) = 5.43, p < 0.001, d = 1.81, and the conscious (congruent: 542 ms; incongruent: 582 ms) conditions, t(25) = 7.29, p < 0.001, d = 2.43. Yet, the RT congruence effect (incongruent RT minus congruent RT) was significantly larger in the conscious (40 ms) than in the non-conscious (23 ms) condition, t(34) = 2.49, p = 0.018, d = 0.83. As in Experiments 1 and 2, an analysis restricted to only the rating trials yielded essentially the same results. **Figure 4** (left panel) illustrates the results.

#### Error Rates (ERs)

The analysis was the same as for the RTs, with the additional within-participant variable task (go, no-go). The main effects of task, F(1,34) = 40.25, p < 0.001, η 2 <sup>p</sup> = 0.54, and congruence, F(1,34) = 17.17, p < 0.001, η 2 <sup>p</sup> = 0.34, also interacted with each other, F(1,34) = 5.12, p = 0.030, η 2 <sup>p</sup> = 0.13. Only in the go trials was the ER higher in incongruent (10.62%) compared to congruent trials (5.09%), t(35) = 4.56, p < 0.001, d = 1.07. No such difference was found in the no-go trials, t(35) = 1.73, p = 0.093, d = 0.41. No other effects were found, all nonsignificant F < 2.88, all p > 0.099.

#### Ratings of the Chinese Symbols

The analysis was based on correct flanker-task trials. No overall bias toward negative (or positive) evaluations was found, neither in the non-conscious, t-test against 50%: t(17) = 0.90, p > 0.249, d = 0.30, BF = 3.81 in favor of the null hypothesis, nor in the conscious condition, t(17) = −0.69, p > 0.249, d = 0.23, BF = 4.45 in favor of the null hypothesis. A mixed-model ANOVA, with the same variables as in the ERs, revealed a main effect of congruence, F(1,34) = 10.95, p = 0.002, η 2 <sup>p</sup> = 0.24, and an interaction between congruence and consciousness, F(1,34) = 10.81, p = 0.002, n<sup>p</sup> <sup>2</sup> = 0.24. In the non-conscious block, participants rated the symbols less often as negative in congruent (47.75%) than in incongruent (55.63%) trials, t(17) = 4.17, p = 0.001, d = 1.39, BF = 55.39 in favor of the alternative hypothesis. There was no such difference in the conscious block (congruent: 48.63%; incongruent: 48.61%), t(17) = −0.02, p > 0.249, d = 0.01, BF = 5.59 in favor of the null hypothesis. No other effects were found, all non-significant F < 2.41, all p > 0.130. The results are illustrated in **Figure 4** (right panel).

#### Distractor Awareness

Participants in the conscious condition could discriminate the distractor with above chance-level accuracy (d' = 1.36), t-test against zero: t(17) = 6.13, p < 0.001, d = 2.04, BF = 2299.58 in favor of the alternative hypothesis. Discrimination performance for participants in the non-conscious condition was not different from chance (d' = −0.14), t(17) = 1.36, p = 0.191, d = 0.45, BF = 2.40 in favor of the null hypothesis (corresponding to no

are indicated on the x-axis. The separate lines illustrate conscious (solid lines) and non-conscious (dashed lines) trials. Error bars represent SEM.

substantial evidence). No difference in the response bias was found between the conscious condition (β = 1.22) and nonconscious condition (β = 0.99), t(34) = 1.40, p = 0.172, d = 0.47, BF = 3.04 in favor of the null hypothesis.

# Discussion

Experiment 3 replicated Experiment 2 and confirmed that the block order was not responsible for the differences between conscious and non-conscious blocks. Again, we found a larger proportion of negative ratings after incongruent than congruent target-distractor pairs. However, this result was only apparent in the non-conscious block, not in the conscious block. Furthermore, we found slower RTs in incongruent than congruent trials, and this congruence effect was larger in the conscious than in the non-conscious condition.

One notable difference to Experiment 2 was the absence of an influence of task (go versus no-go trials) on ratings. This could be explained by the smaller statistical power of the mixedmodel design compared with the within-participant design of Experiment 2. Furthermore, the Bayesian t-test of this effect in Experiment 2 already indicated that evidence of this effect was not entirely conclusive, so that our failure to replicate this effect might reflect data insensitivity.

# GENERAL DISCUSSION

We found more affect misattributions following non-conscious distractors than conscious distractors. Only following nonconscious incongruent distractors but not after conscious incongruent distractors, the proportion of negative ratings of an otherwise neutral symbol increased. This affective priming effect was likely due to the participants' lacking awareness of the non-conscious distractors as the true affect-origins, as indicated by the participants' unawareness of the nonconscious distractors. The unawareness of the true affectorigins probably facilitated a misattribution of the conflictelicited affective responses to the neutral symbols. As such, the results support a misattribution explanation of the affective priming effect on neutral symbols (Payne et al., 2010). In fact, because we found more response priming with the conscious distractors than with the non-conscious distractors, the full data pattern amounted to a double dissociation between response priming (stronger by conscious distractors) and affective priming (stronger by non-conscious distractors). This means that simply more response priming by the nonconscious distractors was not responsible for their stronger affective priming effect. Instead, the unawareness of the nonconscious distractors somehow facilitated the affective priming effect and the awareness of the distractors in the conscious conditions somehow undermined the affective priming effect. For example, participants' awareness of the distractor-target conflict might have allowed them to correctly attribute their feelings to this conflict. This in turn could have undermined any misattributions of the same feelings to the Chinese symbols. Alternatively, awareness of the distractor-target conflict could have also allowed the participants to actively suppress the spread of their feelings to the evaluation of the Chinese symbols because the Chinese symbols were now clearly recognized as not being responsible for the currently pertaining feeling.

To our surprise, however, the results in the conscious condition were relatively extreme: We found no evidence for affective priming of the symbol ratings following conscious distractors whatsoever. This is surprising because a prior Stroop study found affective priming with conscious Stroop stimuli (e.g., Fritz and Dreisbach, 2013). Several reasons for this difference are conceivable. First of all, conflict in Stroop tasks is not the same as in flanker tasks. Some sources of conflict in the Stroop task are not effective in the flanker task. For example, only in the Stroop but not in the flanker task, incongruent trials created higher attentional demands during long-term memory retrieval (Keele, 1972; MacLeod, 1991). The reason is that only in the Stroop task, colors (as targets) and color names (as distractors) were used, so that long-term memory associations between irrelevant color names and responses could have contributed to interference in incongruent conditions. No such long-term memory associations would have been at work in the incongruent flanker conditions of the present study. Another difference between Stroop task and our flanker task concerns the degrees of task conflict in incongruent conditions. In an incongruent Stroop trial, the distractors had the potential to elicit task conflict between the distractor words that tend to elicit their (not instructed) reading and the target colors that, per instruction, had to be named (see MacLeod and MacDonald, 2000; Goldfarb and Henik, 2007). This kind of task interference was absent in the incongruent conditions of the flanker task that we used here. If either of the two aforementioned Stroop-specific conflict sources depends on an awareness of the distractors, the presence of long-term memory based conflict or of the presence of task conflict in the incongruent Stroop trials of Fritz and Dreisbach (2013) could have fostered an affective priming effect of the conscious distractors on symbol ratings in that study. By the same token, the absence of the same conflict sources in the flanker task could have prevented some conflict-dependent affect misattributions based on the present study's conscious distractors.

Moreover, in the present experiments, conflict was elicited by a target-distractor relation that could follow a different time course than the Stroop interference. In the present study, we used distractor-target sequences that were presented 1 s before and, thus, well ahead of the neutral symbols. In contrast, the interval between Stroop stimuli and neutral symbols in the study of Fritz and Dreisbach (2013) was 400 ms which is, relatively short. Moreover, Fritz and Dreisbach (2015) showed that with increasing interval between conflicting Stroop stimulus and tobe-rated neutral stimulus, the effect of incongruent Stroop stimuli on the ratings of neutral stimuli decreased. Since our procedure used an interval between the target and the neutral symbol of 1 s, we might have missed out on time-dependent and more fleeting affective priming effects in the conscious condition. However, unless one assumes that the affective priming effect is more robust and less fleeting in the non-conscious condition, it is not

clear why the time interval would have selectively influenced the affective priming effect in the conscious and not in the nonconscious condition. In other words, our interpretation of a lack of awareness of the true affect-origin as a supportive factor for affect misattributions in non-conscious conditions would still be plausible and not be challenged by this possibility.

### CONCLUSION

For many applications, such as for rational decisions based on evaluations and for attitudes toward objects and persons, it is important to understand the favorable side conditions of correct and incorrect affect attributions. This is so important because the affective responses toward objects and persons shape and trigger human attitudes toward these objects and persons, and ultimately determine actions, such as how much another person is pitied and helped. Here, we have identified non-conscious processing of affect-origins as a favorable side condition for affect misattributions away from their true affect-origins and toward other objects. This finding implies that sufficient awareness of the true affect-origins is helpful for correct affect attributions and, thus, could be a precondition for rational human action (see also Topolinski and Strack, 2009).

## REFERENCES


# AUTHOR CONTRIBUTIONS

FG and UA developed the study concept and the study design. FG performed the data collection, data analysis and interpretation under the supervision UA. FG drafted the manuscript, both SK and UA substantially contributed to the interpretation of the data, and provided many important critical revisions. All authors approved the final version of the manuscript for submission. All authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

# FUNDING

Open access funding provided by University of Vienna.

# ACKNOWLEDGMENT

We would like to thank Adina Pohling and Tamara Moser for their help with the data collection.



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Goller, Khalid, Ansorge. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Subliminal Face Emotion Processing: A Comparison of Fearful and Disgusted Faces

#### Shah Khalid<sup>1</sup> \* and Ulrich Ansorge<sup>2</sup>

1 Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany, <sup>2</sup> Faculty of Psychology, University of Vienna, Vienna, Austria

Prior research has provided evidence for (1) subcortical processing of subliminal facial expressions of emotion and (2) for the emotion-specificity of these processes. Here, we investigated if this is also true for the processing of the subliminal facial display of disgust. In Experiment 1, we used differently filtered masked prime faces portraying emotionally neutral or disgusted expressions presented prior to clearly visible target faces to test if the masked primes exerted an influence on target processing nonetheless. Whereas we found evidence for subliminal face congruence or priming effects, in particular, reverse priming by low spatial frequencies disgusted face primes, we did not find any support for a subcortical origin of the effect. In Experiment 2, we compared the influence of subliminal disgusted faces with that of subliminal fearful faces and demonstrated a behavioral performance difference between the two, pointing to an emotion-specific processing of the disgusted facial expressions. In both experiments, we also tested for the dependence of the subliminal emotional face processing on spatial attention – with mixed results, suggesting an attention-independence in Experiment 1 but not in Experiment 2 –, and we found perfect masking of the face primes – that is, proof of the subliminality of the prime faces. Based on our findings, we speculate that subliminal facial expressions of disgust could afford easy avoidance of these faces. This could be a unique effect of disgusted faces as compared to other emotional facial displays, at least under the conditions studied here.

#### Keywords: face emotion processing, priming, subcortical, disgust, fear

# INTRODUCTION

From a phylogenetic perspective, the quick and effortless recognition of human emotional facial expressions provided an evolutionary benefit: A high ability to recognize fearful or disgusting faces, for example, would have increased human sensitivity to potentially harmful threats in the environment that, if avoided, would have increased inclusive fitness (Whalen, 1998).

In line with this reasoning, several studies provided evidence of even subliminal processing (i.e., processing of stimuli presented below the threshold of awareness) of human emotional expressions (Whalen et al., 1998; de Gelder et al., 1999; Morris et al., 1999; Dimberg et al., 2000; Kiss and Eimer, 2008; Smith, 2012). For example, using electroencephalography (EEG), Smith (2012) studied the time-course of processing of fearful, disgusted, happy, and neutral facial expressions. The participants were asked to categorize the masked faces for their expressions. The author

#### Edited by:

Guido Hesselmann, Charité Universitätsmedizin Berlin, Germany

#### Reviewed by:

Lucie Charles, University College London, United Kingdom Marcus Rothkirch, Charité Universitätsmedizin Berlin, Germany

\*Correspondence: Shah Khalid shah.khalid@uni-osnabrueck.de

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 20 October 2016 Accepted: 02 June 2017 Published: 21 June 2017

#### Citation:

Khalid S and Ansorge U (2017) Subliminal Face Emotion Processing: A Comparison of Fearful and Disgusted Faces. Front. Psychol. 8:1028. doi: 10.3389/fpsyg.2017.01028

found differences in activity patterns in the frontal and occipito-temporal brain regions, where visual backward masking by a stimulus following a face displaying an emotion prevented the face's visibility. What is particularly noteworthy about Smith's study is that, although two types of masked faces contained negative expressions, there were some processing differences between these faces: The masked fearful faces produced greater activation in the frontal region when compared to the masked disgusted faces. Such EEG differences are impressive as they indicate that, even if not registered consciously, two facial expressions of different emotions but of the same negative affective valence recruit different brain areas, ruling out that negative (vs. positive) valence of the affect accounted for the EEG differences and supporting instead an interpretation in terms of emotion-specific processing (see also, e.g., Willenbockel et al., 2012).

In the current study, we picked up on this emotion-specificity to look closer into the processing of disgustedly looking faces (Experiments 1 and 2) and to compare their processing to that of fearful faces (Experiment 2). Based on the known differential sensitivity of human subcortical and cortical visual processing pathways to high-spatial frequency (HSF) content, in Experiment 1, we tested if subliminally presented disgusted faces could probably be processed by the subcortical route, as suggested by more or less related prior research showing evidence of subcortical processing of emotional facial expressions (Pegna et al., 2005; Leh et al., 2006; Willenbockel et al., 2012; for a more general argument, see also LeDoux, 1996, 1998). To that end, we conducted a masked priming experiment, with masked emotional faces as primes and visible faces as targets (Morris et al., 1999; Finkbeiner and Palermo, 2009). In each trial of the experiment, participants judged if the clearly visible target face that they saw was a neutral or a disgusted face. In addition, unknown to the participants, a masked face of the same or of a different emotional expression was presented before each target face. In congruent conditions, displayed emotions of prime and target were the same (e.g., both were disgusted faces), while in incongruent conditions, they were different (e.g., a disgusted face prime preceded a neutral face target). Typically, when comparing between the two conditions, one can expect a priming or congruence effect, with better performance in congruent than incongruent conditions (cf. Neumann and Lozo, 2012). Furthermore, one can also expect to see little or no evidence for the ability to discriminate between different primes in the same participants that show the priming effect. In the current study, this latter prediction was tested in a separate prime visibility test at the end of each experiment.

Importantly, to test a potential subcortical origin of the expected priming effect, in Experiment 1, we used high-pass filtered (HSF) faces as primes for half of our participants and low-pass filtered (LSF) faces as primes for the other half, where LSF face primes would be processed by both, cortical and subcortical, processing routes, and where HSF face primes would only be processed by the cortical processing route (Khalid et al., 2013). We hypothesized that, if subcortical processing accounted for the expected priming effects, priming effects should be stronger or selectively present with unfiltered primes (i.e., having a full band of frequencies) and low-spatial frequency (LSF) primes (i.e., with LSF primes) but not with HSF primes (i.e., with HSF-pass filtered primes).

Experiment 2 tested if the priming effect of Experiment 1 was indeed due to emotion-specific processing. To that end, we included fearful and neutral faces in one block of trials and disgusted and neutral faces in the other block of trials, and we looked for differences between the priming effects of the different blocks. These were to be expected if the priming effects reflected emotion-specific processing (here: of disgust vs. fear). (Of course, in case that the priming effects of disgusted and fearful face primes were the same, they would not necessarily have been also the same in terms of the underlying brain processes. However, at least, if a difference in the priming effects was found, that would have supported their different origins.) In contrast to Experiment 1, only non-filtered face primes were used in Experiment 2, allowing us to investigate if the unexpected sign of the priming effect of the disgusted face primes (that we found in Experiment 1) owes to the fact that the face primes were filtered.

Finally, in both experiments, we also tested the influence of attention (here, of spatial cueing) on priming effects. To that end, face primes were presented slightly above and targets slightly below screen center, so that a visual cue temporally preceding the prime could direct the participants' attention either toward the prime location or toward the target location (Finkbeiner and Palermo, 2009). In this way, we were able to investigate if the priming effect depended upon spatial attention. If the priming effect depended upon spatial attention, we expected to see stronger or selective priming effects in prime-cued conditions as compared to target-cued conditions, but in past research it was found that, under relatively similar conditions (though with a different task and different faces), face priming effects were independent of attention (i.e., of cue position), supporting the classical notion that at least some subliminal processes operate independently of attention (Posner and Snyder, 1975; Finkbeiner and Palermo, 2009).

# EXPERIMENT 1

In this experiment, we tested whether the emotional facial expression of disgust can be processed subliminally and whether such processing follows the subcortical route. Participants categorized visible target faces as either displaying a neutral or a disgusted facial expression. The target face was preceded by a masked prime face in a prime-target congruent (both faces displaying the same emotional expression) or incongruent sequence (both faces displaying different emotional expressions). The face primes were unfiltered, HSF (for half of the participants) or LSF (for the other half of the participants). The target faces were unfiltered. In the visual display, the primes were presented slightly above and the targets below screen center. In order to shift participants' attention, a cue was presented before the prime, either at the prime's location (in the prime-cued condition) or at the target's location (in the target-cued condition; Finkbeiner and Palermo, 2009; Khalid et al., 2013, 2015).

If processing of the subliminal disgusted face's occurs only along the subcortical route, we expected a priming effect for

the LSF primes and the unfiltered primes because this route has been shown to be sensitive to the LSF content of visual signals in general (Schiller et al., 1979) and to emotional expressions in particular (Vuilleumier et al., 2003). The HSF primes would then simply fail to produce a priming effect because of their lack of influence on the subcortical route (cf. Khalid et al., 2015, 2013). However, if subliminal face processing reflects processing along the primary retino-geniculate projection, no such dependence of the priming effect on LSF content should be observed and the priming effect should be found with HSF primes, too, because the retino-geniculate projection is sensitive to this type of visual input.

# Method

#### Participants

Forty students (27 female) with a mean age of 23.1 years participated. Measurement by the German version of the Questionnaire for the Assessment of Disgust Proneness (QADP; Schienle et al., 2002) showed that all participants had normal sensitivity (M = 2.0, SD = 0.4) level. Half of the participants were tested in the HSF group and the other half in LSF group. All participants in the current study had normal or corrected to normal vision, gave their informed consent, were treated in accordance with APA standards and the rules of the declaration of Helsinki, and received course credit in exchange for their participation. For the current study, we obtained ethical approval from the ethics committee of the University of Osnabrueck, Germany, where the study was conducted.

Using G∗Power (Faul et al., 2007), the a-priori power analysis for the main effect of congruence, from the effect size f = 0.61 achieved in an earlier study of Khalid et al. (2013), showed that a required sample size of n = 32 would have been needed to obtain statistical power (1 – β) at the 0.95 level.

#### Apparatus

Visual stimuli were presented on a 17-inch, color flat screen display, with a refresh rate of 59.1 Hz, steered by an NVIDIA GeForce GT 220 (with 1024 MB) graphics adapter. Accurate timing of the display was verified by repeatedly presenting the stimuli for their intended duration one by one and recording their duration through a cathode ray oscilloscope. We found that in all cases the precision of presentation duration was better than 1 ms.

Participants sat at a distance of 57 cm from the screen in a quiet, dimly lit room, with their head resting in a chin rest to ensure a constant viewing distance and a straight-ahead gaze direction. Reaction times (RTs) were registered via a standard serial computer keyboard, placed directly in front of the participants. Target responses were given through the keys 'C' and 'M' (covered and labeled as 'left' and 'right,' respectively). At the beginning, participants placed their left and right index fingers on the appropriate keys. After reading the instructions, the participants pressed the spacebar with one of their thumbs to start the experiment. Pressing the space bar was required prior to each trial, so that the participants could also take breaks at their convenience by simply not pressing the bar.

#### Stimuli

We used the forward mask and cues of previous studies (Finkbeiner and Palermo, 2009; Khalid et al., 2013, 2015). The forward mask was a checkerboard pattern. The backward mask was a scrambled composite of all face images used. The face targets and primes wore neutral or disgusted emotional expressions. All face images in the current study were originally taken from the Karolinska Directed Emotional Faces (KDEF) database (Lundqvist et al., 1998). All of the face stimuli were cropped in an oval layer so that only the face features were presented. Each image subtended a visual angle of 3.0◦ vertically and 2.5◦ horizontally. All face stimuli, here as well as in the following experiment, were equated for luminance and contrast (mean root mean square contrast = 8.39, SD = 0.05), as well as spectral power (mean amplitude = 6.63, SD = 0.48).

The face targets and primes were selected on the basis of participants' performance in a pilot experiment. In the pilot experiment, six male and six female face images in each fearful and disgusted category were presented to 26 participants (18 female, mean age = 21.8 years) using the same procedure as for the current study, except that the masks, primes, and cues were replaced with blank screens, thus, showing only the clearly visible face targets. Participants were asked to categorize the face targets as disgusted or being of a different emotion. On the basis of the participants' performance (RTs and error rates [ERs]), four disgusted and four neutral faces, two male and two female individuals (mean RTs = 604 ms, mean ERs = 11.8%), were selected as primes and targets for the current study (see **Figure 1** for the faces). Targets and primes were similar, except that half of the primes were either HSF or LSF as shown in **Figure 1**.

#### Procedure

See **Figure 2** for examples of sequences of events in a trial. Stimuli were presented on a black screen (luminance < 0.1 cd/m<sup>2</sup> ). In each trial, two streams of stimuli were presented, one stream directly above the screen center, the other directly below it, with the target always at the lower location and the prime always at the upper location (Finkbeiner and Palermo, 2009; Khalid et al., 2013, 2015).

Each trial began with a checkerboard mask for 500 ms. Next, a non-predictive cue was shown for 50 ms at either prime or target location, together with the checkerboard mask at the non-cued location. The cue was used to capture attention toward the prime (in the prime-cued condition) or toward the target (in the targetcued condition). After this, the checkerboard mask was shown for 50 ms, again at both locations, followed by the prime face at the upper location together with the checkerboard mask at the target's location for 50 ms.

All face stimuli were used both as primes and targets, but half of the primes were LSF for one group and HSF for another group of participants. To avoid repetition priming, in each trial, the same face was never shown as both prime and target (Forster and Davis, 1984; Norris and Kinoshita, 2008). In the final frames of a trial, the target face was presented at the lower location for 300 ms together with a scrambled-face backward mask at the upper (prime) location.

Across trials, emotional expressions of prime and target varied orthogonally to create congruent and incongruent conditions. In the congruent condition, the prime face's emotional expression was the same as that of the target face: A disgusted face prime was presented prior to a disgusted face target, or a neutral face prime was presented prior to a neutral face target. In the incongruent condition, the prime face's emotional expression was different from that of the target face: A disgusted face prime was presented prior to a neutral face target, or a neutral face prime was shown prior to a disgusted face target.

In the target-discrimination task, participants had to discriminate the emotional expression of the target face as either neutral or disgusted by pressing one of the assigned buttons (counter balanced across participants) as quickly and as accurately as possible. Slow responses (RTs > 850 ms) and errors were discouraged by slightly delaying feedback displays

targets in half of the trials.) Arrows depict the flow of time.

presented for 750 ms, reading 'respond faster!' and 'wrong key!,' respectively. The target-discrimination task consisted of two within-participant blocks, one with unfiltered primes and the other with filtered primes, the order of which was counter balanced across participants. Each block consisted of 32 repetitions of each combination of cueing (prime-cued, target-cued), target type (disgusted, neutral), and prime-target congruence level (congruent, incongruent), for a total of 256 trials, thus, a total of 512 trials for the target-discrimination task. In the filtered primes block, we also included 10% filler trials (not included in the analysis) in which the HSF and the LSF primes were presented as targets. These filler trials were randomly intermixed with the unfiltered face target trials to ensure that participants did not search only for the unfiltered face targets but also for the HSF and LSF content (cf. Ansorge et al., 2010a; Khalid et al., 2013). An additional 32 training trials were included at the start of each block.

At the end of the experiment, participants performed a block of prime-discrimination trials, in which they had to categorize the masked primes. For this prime-discrimination task, participants first categorized the visible target (just as they had in the targetdiscrimination task), and then they categorized the masked prime. The same emotion-to-response-button mapping was used for the prime faces as for the target faces (cf. Finkbeiner and Palermo, 2009; Khalid et al., 2013, 2015). The primediscrimination task also consisted of two blocks, one with unfiltered primes and one with filtered primes, with 128 trials per each block, thus, a total of 256 trials. Analogously to the

target-discrimination block, the filtered primes block additionally contained 10% trials with the filtered primes as targets (excluded from analysis).

Within each block, the different conditions were presented in a pseudo-random sequence, with the two constraints that no particular face target was repeated in immediately succeeding trials and that no more than four trials in a row required the same target response. The experiment started with the target-discrimination block, followed by the masked primediscrimination block.

At the very end of the experiment, we included one further block, with unmasked face primes of 50 ms duration at their upper position only. This final block was included to verify that the face prime alone contained sufficient information to allow successful emotional expression discrimination, at least under supraliminal (unmasked) conditions. In this final block, all of the masks (checkerboards and composites of scrambled faces) and the targets were left out (i.e., all these stimuli were replaced by blank screens), and the participants were asked to discriminate between the emotional expressions of the unmasked prime faces. This block consisted of a total of 80 trials, 40 trials with unfiltered primes and further 40 trials with filtered primes.

The experiment was run in a single session, with eight short breaks, one in the middle and one at the end of each of the four within-participant target-discrimination and primediscrimination blocks. The whole experiment took approximately 40 min.

Besides the major questions of Experiment 1, we also wanted to test the temporal development of the priming effect. This was done by sorting RTs from fast to slow responses and dividing the ordered distribution into three roughly equally sized bins per each condition of interest. Due to this binning procedure, we were able to look at the differences between congruent and incongruent conditions as a function of the RT bin. For example, priming effects can sometimes be short-lived and are only evident among the fastest responses, so that with a more conservative response criterion, if more time passes since the prime has been presented, less priming effects can be observed (Kinoshita and Hunt, 2008; Ansorge et al., 2010b). However, one should note that this binning procedure has two potential drawbacks. Firstly, although according to Ratcliff (1979), 10 trials are sufficient per each mean in vincentizing, in the current study, this procedure indeed left only a small number of trials per bin and condition (i.e., 10.67). Secondly, binning did not allow for perfect counterbalancing with the other experimental factors because of the fractional number of trials per bin and because of not replacing outliers and errors. (The latter, however, is often the case in vincentizing.) However, slightly different numbers of trials for the different levels of the same independent variables are standard in psychological research because different error rates for different levels of independent variables typically jeopardize equal numbers of trials per each level of each variable anyway. Also, similar approaches – inclusion of an additional variable "bins" after an initial analysis – have been taken in other research areas with the aim of scrutinizing the conclusions of an initial analysis once more. For instance, with the help of a median split of the RTs, McDonald et al. (2013) showed, contrary to initial conclusions of Hickey et al. (2006), that interference effects of irrelevant distractors were restricted to the slowest responses. As an aside, we noted that McDonald et al. also used a larger sample size than Hickey et al. to overcome power issues of the former study. This, however, was not recommended in the case of the present study as our analysis indicated sufficient power.

# Results

#### Target-Discrimination Task

For our first analysis, only correct target responses were considered. Mean correct RTs for each participant and condition were calculated. In order to correct for potential speed-accuracy tradeoffs, we applied the so-called "killing-the-twin" procedure (Grice et al., 1977; Eriksen, 1988) here and throughout the study: Separately for each condition, twins (nearest in magnitude) of erroneous RTs were searched in the correct RTs and discarded (5.6%; Inclusion versus exclusion of twins of the erroneous responses had no qualitative or quantitative impact on the results.) Furthermore, trials that were faster or slower than 2.5 standard deviations of the corrected mean were also discarded (3.8%). Maybe a less biased approach to the removal of outliers could have been based on the interquartile range (IQR), but we preferred to apply the same outlier criterion as was used in prior studies of face priming in this paradigm to rule out that potential differences were due to marginal methodological details. The priming effects in mean correct RTs (incongruent RT minus congruent RT) for all our variables are depicted in **Figure 3**, upper panel, for the HSF block, and in **Figure 4**, upper panel for the LSF block.

An omnibus mixed-measures analysis of variance (ANOVA) was performed, with the within-participant variables target valence (neutral vs. disgusted), cue type (target-cued vs. prime-cued), prime filtering (unfiltered vs. filtered primes), and prime-target congruence (congruent vs. incongruent), as well as the between-participants variable prime frequencies (HSF vs. LSF). For any post hoc tests, Bonferroni adjustments for multiple comparisons and the alpha level of 0.05 for all statistics were applied here and throughout the study. See **Table 1** for the results of the ANOVA.

Bins analysis: We ran the omnibus ANOVA with the additional variable of bin (1, 2, and 3). Besides the above reported results, this ANOVA additionally showed a tendency toward a significant four-way interaction of bin with prime frequencies, target valence, and congruence, F(2,76) = 3.03, p = 0.054, partial η <sup>2</sup> = 0.07. However, follow-up analyses did not show any significant interaction of bin with congruence.

#### Error Rates

See **Figure 3**, lower panel for mean ERs in the HSF block, and **Figure 4**, lower panel for the LSF block. The same omnibus ANOVA as for RTs was also run for the error rates (ERs, total = 5.6%). See **Table 2** for the results of this ANOVA.

#### Prime Visibility

The participants were not able to successfully discriminate the emotion of the masked primes as either neutral or disgusted with better than chance accuracy. One-sample t-tests against

a chance-level performance criterion of 50% correct responses were conducted. As these frequentist tests are not optimized for the support of the null hypothesis, we also computed their Bayes Factors (BFs) with their recommended bench mark scale factor R of 1.0 (Rouder et al., 2009). In the current study, we report the scaled JZS (Jeffreys, Zellner, and Siow) BFs. These BFs indicate the ratio of marginal likelihoods for the null versus the alternative hypothesis (or vice versa) (Jeffreys, 1961; Kass and Raftery, 1995). Conventionally, a Bayes factor greater than 3 is considered as substantial evidence in favor of the null or the alternative hypothesis (Jeffreys, 1961; Dienes, 2011). The t-tests indicated that the participants performed not significantly different from chance level in the unfiltered primes' target-cued condition (M = 50.1%), t(39) = 0.07, p = 0.94, BF = 8.10 in favor of the null hypothesis, and prime-cued condition (M = 48.9%), t(39) = 0.98, p = 0.33, BF = 5.10 in favor of the null hypothesis, as well as in the filtered primes' target-cued condition (M = 51.5%), t(39) = 1.21, p = 0.23, BF = 4.01 in favor of the null hypothesis, and prime-cued condition (M = 51.5%), t(39) = 1.06, p = 0.30, BF = 4.72 in favor of the null hypothesis.

#### Unmasked Prime Discrimination

The participants were able to successfully discriminate the emotion of the unmasked primes as either neutral or disgusted with better than chance accuracy. In the HSF block, unfiltered primes (M = 95.1%), and filtered primes (M = 92.8%) were

discriminated with high rates of accuracy, and the same was true of the LSF block: unfiltered primes (M = 93.8%), filtered primes (M = 92.9%), in all conditions ts > 32.00, ps < 0.001, with all BFs > 1300000.00 in favor of the alternative for tests against chance performance (50%).

#### Discussion

The results showed unanimous evidence for reversed priming effects with the disgusted face targets. With the disgusted faces as targets, error rates were lower in incongruent than in congruent conditions with HSF, LSF, and unfiltered primes. A similar effect was found in the RTs of the LSF primes, where the reversed priming effect seemed to extend to the neutral targets. This generalization of the reversed priming effect to the RTs of the LSF primes is the only weak indication of a subcortical origin of the, in this case reversed, priming effect. These effects are surprising in terms of motor activation theory because prior reports would have suggested to expect an advantage in congruent as compared to incongruent conditions (e.g., Neumann and Lozo, 2012). In addition, usually, reversed priming effects in masked priming experiments are more typical of longer prime-target intervals (e.g., Eimer and Schlaghecken, 1998; for a review, see Sumner, 2007). Maybe a strong signaling function of disgusted facial expressions of adverse situations is more compatible

with the participants' quick avoidance or suppression of this situation. If that was the case, target emotion perception in the congruent disgusted condition could have fallen prey to the quick suppression of the disgusted face prime, whereas this suppression could have in turn facilitated responding to the alternative emotionally neutral face category. Another theoretical interpretation in terms of more adaptation to disgusted than to neutral faces does not make much sense. Although adaptation to a facial emotional expression can in principle decrease processing of the same expression in a temporally following face stimulus (Winston et al., 2004), adaptation effects typically occur on a longer time scale (around 5 s) and are not observed with such short adaptor durations and adaptor-target intervals as were realized here, where the prime would have been a potential adaptor. It is also unclear, why an effect of adaptation should have been restricted to the disgusted face targets and would not have extended to the neutral face targets.

In addition to the perplexing reversed priming effect with the disgusted face targets, we found a more straightforward advantage in the error rates of the congruent as compared to the incongruent conditions with the neutral face targets. The only exception to this pattern was the reversed priming effect in the RTs to neutral target faces of the LSF group that implies that the ER priming effect with neutral face targets in the LSF group could have likewise reflected a speed-accuracy trade-off.

Of further interest, we also found evidence of a cueing effect. Responses to the target faces were faster in targetcued than prime-cued conditions. This cueing effect was likely due to attention, facilitating the processing of the target if the cue happened to direct attention to the target position. Also, as the priming effect of the masked primes was not affected by the cueing effect and as participants showed no indication of an awareness of the masked primes, as in prior studies (Finkbeiner and Palermo, 2009; Khalid et al., 2013),

TABLE 1 | Results of an analysis of variance (ANOVA) of the mean correct reaction times (RTs), with the within-participant variables target valence (neutral vs. disgusted), cue type (target-cued vs. prime-cued), prime filtering (unfiltered vs. filtered primes), and prime-target congruence (congruent vs. incongruent), as well as the between-participants variable prime frequencies (high spatial frequencies [HSF] vs. low spatial frequencies [LSF]) in the target-discrimination task of Experiment 1.


TABLE 2 | Results of an analogous ANOVA of the error rates (ERs), with the within-participant variables target valence (neutral vs. disgusted), cue type (target-cued vs. prime-cued), prime filtering (unfiltered vs. filtered primes), and prime-target congruence (congruent vs. incongruent), as well as the between-participants variable prime frequencies (high-spatial frequencies [HSF] vs. [LSF]) in the target-discrimination task of Experiment 1.


the pattern of results is in line with a coincidence of an independence of awareness and of an independence of attention of the processing of the masked primes and, thus, with a classic view of automatic processing of emotional expressions (cf. Posner and Snyder, 1975). One should note, however, that our manipulation of spatial attention was soft and that our cueing procedure could have, thus, spared enough attentional resources for prime processing even under target-cued conditions. Authors have argued for the usage of highly demanding secondary tasks for potentially more

conservative tests of attention-independent processing (Lavie, 1995; Pessoa, 2005).

Finally, the control conditions with the unasked primes made clear that each of the primes that we used in the present study carried sufficient information to be correctly discriminated under aware conditions. This finding shows that the differences between the priming effects of neutral and disgusted faces cannot simply be explained by the difficulty with which disgusted versus neutral face primes could be discriminated per se.

# EXPERIMENT 2

In Experiment 2, we intended to replicate the reversed priming effect of Experiment 1. As filtering had no strong impact on priming effects in Experiment 1, in Experiment 2, we only used unfiltered primes. In addition, we also used a within-participant comparison between the priming effects of disgusted faces and a second class of negative emotional faces, fearful faces. This was done to investigate if the reversed priming effect that we found with masked disgusted faces in Experiment 1 was emotion-specific, or whether it was maybe typical of any negative valence stimulus. To note, both disgusted and fearful faces express emotions of a negative valence, and a valence-based effect should therefore be the same with fearful and with disgusted faces. However, disgusted and fearful faces depend on different underlying processes (Breiter et al., 1996; Anderson et al., 2003; Phillips et al., 2004; for a meta-analysis see Vytal and Hamann, 2010). Thus, it is also well conceivable that the priming effects of disgusted and fearful faces differ from one another. For instance,

fearful faces seem to attract attention more than disgusted faces (Khalid et al., 2017), meaning that if the reversed priming effect with the disgusted faces was really due to more avoidance and suppression of these faces, it might not generalize to fearful faces.

# Method

#### Participants

Thirty-five students (26 female) with a mean age of 21.9 years participated. Measurement by the German version of the State-Trait Anxiety Inventory (STAI; Laux et al., 1981) showed that all participants had normal state (M = 34.7, SD = 6.5) and trait (M = 41.6, SD = 8.2) anxiety levels. Further, assessment by the German version of the QADP (Schienle et al., 2002) showed that all participants had normal disgust sensitivity (M = 2.0, SD = 0.4).

#### Apparatus, Stimuli, and Procedure

These were the same as before with the following exceptions. First, all primes were unfiltered and there was, thus, no betweenparticipants variable. Second, for the second block, four fearful faces, two male and two female (mean RTs = 605 ms, mean ERs = 11.7%), were selected from the pilot experiment. As mentioned in Experiment 1, the participants' performance in the pilot experiment for the disgusted faces was (mean RTs = 604 ms, mean ERs = 11.8%). Thus, we balanced the performance difficulty for the two classes of negative expressions based upon participants' responses to them (see **Figure 1**). Third, the block order, fearful faces before disgusted faces or the other way round, was balanced across participants. Fourth, all prime-visibility tests were administered at the end of the experiment, again blocked for different emotional expressions, again with tests of masked faces before tests of unmasked faces. Fifth, as in Experiment 1, each block consisted of 32 repetitions of each combination of cueing (prime-cued, target-cued), target emotion (neutral, emotional), and prime-target congruence level (congruent, incongruent), for a total of 256 trials, and, thus, a total of 512 trials for the target-discrimination task. Also the prime discrimination consisted of two blocks, one for the fearful block and one for the disgusted block, with 128 trials per each block, thus, a total of 256 trials. Again there were 80 trials for the unmasked blocks and 32 training trials in the beginning of each of the target-discrimination blocks. Sixth, the task of the participants was to categorize the neutral versus disgusted, or the neutral versus fearful faces. Seventh, the experiment was run in a single session, with eight short breaks, two in the middle and one after each target-discrimination block, plus one after each of the prime-discrimination blocks, taking approximately 1 h in total.

## Results

#### Target-Discrimination Task

Mean correct RTs, devoid of killed twins (4.2%) for erroneous RTs, were filtered by the same RT criterion as in Experiment 1 (leading to the elimination of 6.0% of all trials).

An omnibus repeated-measures ANOVA was run, with the within-participant variables target emotion (disgusted vs. fearful; between blocks), target valence (neutral vs. emotional; within blocks), cue type (target-cued vs. prime-cued; within blocks),

Bins analysis: For an exhaustive analysis, again, RTs were divided into three bins from fast to slow responses. We ran the above omnibus ANOVA with the additional variable of bin (1, 2, and 3; within blocks). In contrast to Experiment 1, RT bin interacted significantly with congruence (see below). The priming effects in mean correct RTs as a function of all our variables are depicted in **Figure 5**, upper panel.

See **Table 3** for the results of the ANOVA.

#### Error Rates

The same omnibus ANOVA was conducted on ERs (4.2% in total). See **Table 4** for the results of the ANOVA.

#### Prime Visibility

The participants were once again not able to successfully discriminate the emotion of the masked primes as either neutral or disgusted with better than chance accuracy. One-sample t-tests against chance level accuracy (50%) indicated that the participants performed not significantly different from chance level performance in the disgusted block, target-cued condition (M = 51.4%), t(34) = 0.79, p = 0.44, BF = 5.63 in favor of the null hypothesis, and prime-cued condition (M = 52.9%), t(34) = 1.69, p = 0.10, BF = 1.99 in favor of the null hypothesis (this latter value is less than 3, and therefore not substantial evidence), as well as in the fearful block, target-cued condition (M = 48.2%), t(34) = 1.19, p = 0.24, BF = 3.87 in favor of the null hypothesis, and prime-cued condition (M = 52.2%), t(34) = 1.77, p = 0.09, BF = 1.76 in favor of the null hypothesis (again, this latter value is less than 3, and therefore not substantial evidence).

#### Unmasked Prime Discrimination

The participants were able to successfully discriminate the emotional expressions of the unmasked prime faces in both unmasked disgusted faces (M = 92.3%), t(34) = 37.76, p < 0.001, BF > 1200000.00 in favor of the alternative hypothesis, and unmasked fearful faces (M = 89.1%,), t(34) = 26.24, p < 0.001, BF > 1100000.00 in favor of the alternative hypothesis. Moreover, the discrimination of the unmasked disgusted primes was significantly better than that of the fearful primes, t(34) = 2.82, p < 0.01, BF = 4.30 in favor of the alternative hypothesis.

#### Discussion

The results of Experiment 2 showed a reversed priming effect in the ERs for both emotional facial expressions that we used. This was in contrast to the ER effects with neutral faces that showed a more typical priming effect, with advantages in congruent as compared to incongruent conditions. Moreover, only with the target-cued disgusted faces, the reversed priming effect was also found in the RTs. In all other conditions, RTs showed some evidence of an RT priming effect that was in the opposite direction of the respective ER priming effect: With primecued disgusted target faces and with fearful target faces, RTs were faster in congruent than incongruent conditions, while

TABLE 3 | Results of an ANOVA of the RTs, with the within-participant variables bin (1, 2, and 3; within blocks), target emotion (disgusted vs. fearful; between blocks), target valence (neutral vs. emotional; within blocks), cue type (target-cued vs. prime-cued; within blocks), and prime-target congruence (congruent vs. incongruent; within blocks) in the target-discrimination task of Experiment 2.


(Continued)

#### TABLE 3 | Continued


with neutral target faces, RTs were faster in incongruent than congruent conditions. In conclusion, we only found clear-cut evidence of a reversed priming effect with target-cued disgusted face targets, whereas all other priming effects showed signs of speed-accuracy trade-offs. Please also note that the priming effects were non-significant in the initial RTs analysis, but were revealed only in the subsequent bins analysis and in the ERs.

What might have caused the dependence of the reversed priming effect on attention is unclear. This finding, at variance with what was observed in Experiment 1, shows that the classical criteria of automatic processing, such as awareness-independence and attention-independence, do not necessarily always converge (Kahneman and Treisman, 1984; Bargh, 1994; Moors and De Houwer, 2006). Maybe the reverse priming effect indeed reflected some type of suppression of the prime content that was here further exacerbated by the target-directed attention elicited by the cues. The fact that this was not found in Experiment 1, however, suggests that simple group differences between the participants of the two experiments might have played a role, too. In any case, one should bear in mind that the current manipulation of spatial attention might not have been the strongest measure of depleting attentional resources at the expense of prime processing, so that

TABLE 4 | Results of ANOVA of the mean error rates (ERs), with the within-participant variables target emotion (disgusted vs. fearful; between blocks), target valence (neutral vs. emotional; within blocks), cue type (target-cued vs. prime-cued; within blocks), and prime-target congruence (congruent vs. incongruent; within blocks) in the target-discrimination task of Experiment 2.


future studies should try to investigate the role of attention for (reverse) subliminal priming effects also with alternative means (e.g., highly demanding secondary tasks; cf. Lavie, 1995; Pessoa, 2005). In fact, in the current experiment, there was not much evidence for the intended attentional manipulation in the first place, as the cueing effects were in an unexpected direction.

With respect to the priming effects of all other conditions, primes used with fearful face targets, primes used with neutral face targets, and prime-cued primes used with disgusted face targets, the results were disappointing as no clear priming effects could be found. This finding is disappointing because at least motor priming should have fostered the priming effect: The primes were used as targets and carried a meaning that fitted to the participants' task of a categorical decision between neutral and emotional faces. These would have been ideal conditions for an awareness-independent motor priming effect (Klotz and Neumann, 1999; Kunde et al., 2003). The fact that the masked and, thus, liminal primes were without effect is, however, in line with the claim that little processing of emotional facial expressions occurs outside awareness, once physical stimulus differences are controlled for Hedger et al. (2015). This is exactly what we have done in the current study: We have used faces of different emotional expressions as primes that were devoid of spectral power and luminance differences between different emotional expressions, thus, probably curtailing differential effects that any of the masked faces might have had on target recognition. This finding is at least in line with the efficiency of our masking procedure, reflected in chance performance in the mask discrimination task, with the exception of the prime-cued conditions in both disgusted and fearful faces blocks, where the Bayes factors did not show substantial evidence in favor of the null hypothesis.

In addition to these effects, we also again found that the unmasked primes contained sufficient information to also successfully discriminate between fearful and neutral face primes.

# GENERAL DISCUSSION

In light of past research, the current study asked two questions: Is there any evidence for subcortical visual processing of subliminal facial expressions of disgust, and can we find evidence for the emotion-specificity of the processing of subliminal disgusted faces? Question 1 was addressed in Experiment 1. Subliminal emotional face processing seems to reflect humans' sensitivity to highly useful visual input that could have been of help in repeatedly overcoming the same threats to human survival in the course of evolution (Öhman and Mineka, 2001; Öhman, 2002). Such sensitivity has the chance to prevent harm and, thus, increase inclusive fitness among those showing this phenotypic trace, for instance, by an increased sensitivity to sources of danger and the possibility to avoid them among those who inherited a proneness to the detection of fearful human facial displays by genetic predisposition (Öhman, 2002). As researchers have repeatedly linked this hypothesis to the evidence for the subcortical visual processing of a variety of subliminal emotional expressions that are different from disgust (Dolan et al., 1996; Whalen et al., 1998; Jolij and Lamme, 2005; Tamietto and de Gelder, 2010), we tested the possibility that subliminal processing of disgusted faces, equally pointing to potential sources of threat in the environment, could also be carried out along the subcortical visual pathway. For this test, we took advantage of the known exclusive sensitivity of the subcortical visual pathway to LSF content (Schiller et al., 1979; Merigan and Maunsell, 1993; Deruelle and Fagot, 2005; de Gardelle and Kouider, 2010; Khalid et al., 2013). We used differently filtered images of disgusting faces as primes and reduced their visibility to subliminal processing levels by backward masking. We tested if priming effects of subliminal faces are restricted to LSF primes and unfiltered primes, but cannot be found with subliminal HSF primes that are known to not be processed along the subcortical pathway. In Experiment 1, we did not find any evidence for subcortical processing of disgusted faces, as it did not matter if an LSF or HSF prime was used for the face primes.

Also, to our surprise, the only evidence of subliminal processing of disgusted faces took the form of a reversed priming effect, with slower RTs and more errors in conditions in which a subliminal disgusted face prime preceded a supraliminal disgusted face target as compared to a condition with a subliminal emotionally neutral face prime preceding the same disgusted face target. This was surprising because, based on the existing literature, we would have expected the opposite: better performance with congruently than incongruently primed emotional targets (e.g., Neumann and Lozo, 2012). Maybe the reversed priming effect reflected efficient avoidance and suppression of the disgusted faces–a response that seems to be reasonable in terms of the most fruitful and reinforcing behavioral consequences that should be triggered by a disgustsignaling stimulus (Oaten et al., 2009).

The second question was studied in Experiment 2. In Experiment 2, we intended to replicate the surprising inverse priming effect with disgusted face targets and at the same time wanted to test if this effect is emotion-specific. To that end, we compared subliminal priming effects with disgusted face targets to that with fearful face targets within the same participants. In this experiment, little evidence of subliminal emotional processing was found, but the few indications that the subliminal prime faces were processed were again showing a reversed priming effect with the disgusted face targets. Different from Experiment 1, however, in Experiment 2 the reverse priming effect was restricted to conditions in which a cue would have distracted attention away from the primes. Maybe this finding reflected that indeed the avoidance and suppression of the processing of the disgusted faces was responsible for the reverse priming effect and that this suppression was supported by a cue away from the prime.

To note, in both experiments, we had used pre-cues to direct the participants' attention to either the targets or the primes. This was done in an attempt to test if subliminal processing of emotional displays of faces was also independent

of attention (cf. Finkbeiner and Palermo, 2009). Yet, the evidence for an attention-independence of the subliminal emotional face processing was restricted to Experiment 1, as implied by the interaction of cueing and priming with the disgusted faces in Experiment 2, and we do not know why the cues should have exerted their effects on subliminal processing of the disgusted faces in Experiment 2 only. Be that as it may, jointly the results seemed to support the notion that different criteria of automatic processing do not necessarily converge (Bargh, 1994). However, on a more cautionary note, the current manipulation of spatial attention might have been simply too weak to really deplete attentional resources. In line with this assumption, in Experiment 2, the manipulation of attention did not work as intended. Thus, future studies should use more drastic measures, such as a more demanding secondary task, to investigate if subliminal facial priming effects are indeed independent of attention (cf. Pessoa, 2005).

What was also striking was the absence of any clear-cut evidence for subliminal priming effects of the fearful face primes in Experiment 2. This was in contrast with previous studies (e.g., Dolan et al., 1996) and might have reflected the particular care that we took in creating physically equal face prime energies for the different emotional displays. Prior research had already suggested that evidence for the (differential) processing of subliminal emotions could owe to the confounded physical energy differences between different emotional displays (Hedger et al., 2015), so that it may not be such a wonder that our emotional faces, which were equated for their frequency spectra, were not creating much of a subliminal priming effect. In fact, this consideration should also be taken to put the reverse priming effect of the subliminal disgusted faces into perspective: Maybe the reverse priming effect reflected that bit of a net priming effect with subliminal emotional expressions that survived the elimination of the physical differences between different face expressions.

What can hardly be doubted following our experiments is that the face primes contained sufficient information for successful discrimination of different emotional expressions. This conclusion was supported by the participants' performance in trials in which the same faces that were used as subliminal primes were presented supraliminally. Under supraliminal conditions, participants had little difficulty to tell different emotional face primes from one another.

#### REFERENCES


# CONCLUSION

Subliminal priming by disgusted faces was found but took the unusual form of reversed priming. This effect seemed to be emotion-specific rather than reflecting the valence of the disgusted faces, as no such processing was found with subliminal faces of a different emotion (i.e., fearful faces). Yet, no evidence for subcortical visual processing of subliminal disgusted faces could be found. From what we know, it could well be that subliminal processing of disgusted faces occurs along the retinogeniculate pathway, maybe even at cortical face processing sites (Kouider et al., 2009; Smith, 2012).

### ETHICS STATEMENT

This study was carried out in accordance with the recommendations of APA standards and the rules of the declaration of Helsinki, ethics committee of the University of Osnabrück with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the ethics committee of the University of Osnabrück.

### AUTHOR CONTRIBUTIONS

Both authors, SK and UA, substantially contributed to the conception and design of the work, acquisition, analysis, and interpretation of data for the research, as well as drafting the manuscript. Both authors approved the final version for publication, and both agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

# FUNDING

Supported by, Deutsche Forschungsgemeinschaft Grant KH 341/1-1 to SK, and the open access funding provided by University of Osnabrück. We are thankful to Nikola Komlenac, Lisa Cadzlek, and Adriana Zban¸t for help with the data collection.

2nd Edn, eds R. S. Wyer and T. K. Srull (Hillsdale, NJ: Erlbaum), 1–40.




emotional facial expressions–an intracranial ERP study. Front. Psychol. 3:237. doi: 10.3389/fpsyg.2012.00237

Winston, J. S., Henson, R. N. A., Fine-Goulden, M. R., and Dolan, R. J. (2004). fMRI-adaptation reveals dissociable neural representations of identity and expression in face perception. J. Neurophysiol. 92, 1830–1839. doi: 10.1152/jn. 00155.2004

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer MR and handling Editor declared their shared affiliation, and the handling Editor states that the process nevertheless met the standards of a fair and objective review.

Copyright © 2017 Khalid and Ansorge. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Subliminally and Supraliminally Acquired Long-Term Memories Jointly Bias Delayed Decisions

Simon Ruch1, 2 \*, Elizabeth Herbert <sup>3</sup> and Katharina Henke1, 2

<sup>1</sup> Department of Psychology, University of Bern, Bern, Switzerland, <sup>2</sup> Center for Cognition, Learning and Memory, University of Bern, Bern, Switzerland, <sup>3</sup> School of Physiology, Pharmacology and Neuroscience, University of Bristol, Bristol, United Kingdom

#### Edited by:

Marcus Rothkirch, Charité Universitätsmedizin Berlin, Germany

#### Reviewed by:

Surya Gayet, Utrecht University, Netherlands Guido Hesselmann, Charité Universitätsmedizin Berlin, Germany Christian Valuch, University of Göttingen, Germany

> \*Correspondence: Simon Ruch simon.ruch@psy.unibe.ch

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 30 October 2016 Accepted: 24 August 2017 Published: 12 September 2017

#### Citation:

Ruch S, Herbert E and Henke K (2017) Subliminally and Supraliminally Acquired Long-Term Memories Jointly Bias Delayed Decisions. Front. Psychol. 8:1542. doi: 10.3389/fpsyg.2017.01542 Common wisdom and scientific evidence suggest that good decisions require conscious deliberation. But growing evidence demonstrates that not only conscious but also unconscious thoughts influence decision-making. Here, we hypothesize that both consciously and unconsciously acquired memories guide decisions. Our experiment measured the influence of subliminally and supraliminally presented information on delayed (30–40 min) decision-making. Participants were presented with subliminal pairs of faces and written occupations for unconscious encoding. Following a delay of 20 min, participants consciously (re-)encoded the same faces now presented supraliminally along with either the same written occupations, occupations congruous to the subliminally presented occupations (same wage-category), or incongruous occupations (opposite wage-category). To measure decision-making, participants viewed the same faces again (with occupations absent) and decided on the putative income of each person: low, low-average, high-average, or high. Participants were encouraged to decide spontaneously and intuitively. Hence, the decision task was an implicit or indirect test of relational memory. If conscious thought alone guided decisions (= H0), supraliminal information should determine decision outcomes independently of the encoded subliminal information. This was, however, not the case. Instead, both unconsciously and consciously encoded memories influenced decisions: identical unconscious and conscious memories exerted the strongest bias on income decisions, while both incongruous and congruous (i.e., non-identical) subliminally and supraliminally formed memories canceled each other out leaving no bias on decisions. Importantly, the increased decision bias following the formation of identical unconscious and conscious memories and the reduced decision bias following to the formation of non-identical memories were determined relative to a control condition, where conscious memory formation alone could influence decisions. In view of the much weaker representational strength of subliminally vs. supraliminally formed memories, their long-lasting impact on decision-making is noteworthy.

Keywords: subliminal stimulation, decision-making, unconscious processing, long-term memory, relational learning, hippocampus

# INTRODUCTION

Decision-making is the cognitive process of collectively integrating relevant knowledge to determine the optimal option among several possibilities. Decision-making is considered to depend on conscious deliberation (Simon, 1955; Bettman et al., 1998; Kahneman, 2003; Newell and Shanks, 2014) because good decisions require long-term episodic memory, a proper analysis and integration of information, strategic reasoning and formal logic. Increasing evidence suggests that decision-making is also influenced by unconscious mental processes (Betsch and Gloeckner, 2010; Custers and Aarts, 2010; Baumeister et al., 2011; van Gaal et al., 2012; Creswell et al., 2013; Hassin, 2013). Unconscious cognition might be most relevant in situations where decisions have to be made intuitively, for example if a deliberate analysis of decision alternatives and their possible consequences is not possible (Betsch and Gloeckner, 2010; Zander et al., 2016). Here, we ask if intuitive decisions benefit from unconscious knowledge, i.e., from memories that were formed outside of awareness. We further ask how such unconscious knowledge interacts with consciously formed memories during decisionmaking. Are decisions and choices exclusively determined by conscious information, or do humans integrate all available conscious and unconscious knowledge while making decisions? To address these questions, we tested whether subliminal information, i.e., information presented below the threshold of awareness, is stored in memory to guide later decisions unconsciously. We further explored whether such subliminally acquired unconscious knowledge would interact with later encoded supraliminal information in guiding decision-making.

Prominent evidence for unconscious influences on decision processes is provided by priming studies, in which subliminal stimuli are presented as primes to alter the conscious processing of subsequently presented target stimuli (for reviews see e.g., Van den Bussche et al., 2009; Ansorge et al., 2014; Henson et al., 2014). Subliminal primes were found to facilitate responses to subsequently presented suprathreshold targets—either by reducing response latency or by increasing response accuracy if primes and targets are associated with the same response. Importantly, priming has been observed even if primes were perceptually different from targets but shared the same semantic category, and even if prime stimuli were never used as targets and thus were not directly associated with a specific response (Ortells et al., 2016). These findings suggest that subliminally presented information can influence decisions not only by triggering pre-established motor responses, but by activating conceptual knowledge about the upcoming target (Van den Bussche et al., 2009). In addition to facilitating responses to target stimuli, subliminal primes were found to bias "free choices" in targetabsent trials where participants could freely choose between response alternatives (Schlaghecken and Eimer, 2004; Klapp and Haas, 2005; Kiesel et al., 2006; Milyavsky et al., 2012; Parkinson and Haggard, 2014; Ocampo, 2015, 2016). Hence, the processing of subliminal stimuli may not only influence the speed and accuracy of responses concerning a supraliminal target but may bring forth new behaviors in situations where choices depend entirely on subjective preferences.

Subliminal priming is notoriously short-lived suggesting that unconsciously processed information might not be stored in long-term memory and thus might fail to inform delayed decisions. The influence of subliminal primes usually dissipates within a few hundred milliseconds and is restricted to the immediately following target (Forster et al., 1990; Ferrand, 1996; Greenwald et al., 1996; Mattler, 2005; Dupoux et al., 2008). Although, the precise mechanisms of subliminal priming are still debated (Henson, 2003; Kiesel et al., 2007; Kouider et al., 2007; Gomez et al., 2013; Henson et al., 2014), there is consensus that subliminal primes briefly enhance the accessibility of pre-established perceptual or semantic knowledge or stimulusresponse associations. This enhanced accessibility facilitates the conscious processing of perceptually or semantically primerelated targets but does not leave an enduring memory trace (but see Bodner and Masson, 2001, 2014 for an alternative view).

Besides the evidence for a rapid decay of subliminal priming there is a growing literature reporting long-term effects of subliminal stimuli, suggesting mechanisms of unconscious encoding that yield long-term memory traces. The repeated exposure to the same subliminal stimulus was found to enhance perceptual sensitivity to the same stimulus' features when presented delayed and supraliminally, thereby facilitating conscious stimulus identification and classification (Seitz and Watanabe, 2009; Seitz et al., 2009; Rosenthal and Humphreys, 2010; Atas et al., 2013; Pascucci et al., 2015). Hence, subliminal perceptual learning can durably enhance the saliency of a stimulus and might thus boost its impact on delayed decisions. Several studies revealed that humans can even learn entirely novel associations between subliminal stimuli and supraliminal events or between subliminal cues and specific behavioral outcomes. In classical conditioning experiments, initially neutral subliminal stimuli began to elicit arousal-related physiological responses if these stimuli were repeatedly paired with an arousing event such as an electric shock (Wong et al., 2004; Both et al., 2008; Raio et al., 2012; Lipp et al., 2014; Jensen et al., 2015). Operant conditioning experiments revealed that subliminally presented stimuli helped to improve task performance if these stimuli reliably predicted which behavior would lead to success in the next trial (Pessiglione et al., 2008; Palminteri et al., 2009; Pine et al., 2014; Mastropasqua and Turatto, 2015). Similar results were obtained if initially irrelevant subliminal stimuli were repeatedly followed by a switch of tasks in task-switching experiments or were paired with specific target categories in priming experiments: with enough practice, participants improved on switch trials when subliminal cues had predicted the task switches (Zhou and Davis, 2012; Manly et al., 2014; Farooqui and Manly, 2015) and were faster at classifying targets that were predicted by a preceding subliminal prime (Custers and Aarts, 2011; Marcos Malmierca, 2015). Finally, dichoptic stimulation experiments showed that the repeated presentation of a sequence of subliminal events allowed participants to later identify this same sequence despite their inability to detect the underlying events (Rosenthal et al., 2010, 2016). These findings demonstrate that humans can learn about the predictive

value of subliminal stimuli and can utilize this knowledge to improve their decisions. However, these findings also suggest that subliminal learning might be slow and might only yield rigid knowledge that cannot be transferred flexibly to other tasks or contexts (Rosenthal and Soto, 2016). Indeed, subliminal learning in these studies required extensive training and manifested only as improvement within the very task that participants had been trained on. Unconsciously acquired knowledge might thus be too weak to interact with consciously formed memories during decision-making.

Studies investigating the role of consciousness in memory yielded evidence that humans are able to rapidly form novel relational memories between subliminal stimuli and may flexibly retrieve these unconscious relational memories in new contexts when facing various decision tasks (e.g., Reber and Henke, 2011; Chong et al., 2014; Reber et al., 2014; Kawakami and Yoshida, 2015). For example, the subliminal presentation of faces paired with written occupations later biased participants' conscious decisions about the income of these individuals if the faces were re-presented supraliminally for conscious inspection. Faces that had been paired with a high-wage occupation during subliminal stimulation later received higher income ratings than faces paired with a low-wage occupation (Ruch et al., 2016). This effect was observed despite the fact that participants had not noticed the subliminal stimuli, did not consciously remember the faces and occupations, and rated the income based on their mere intuition. Similar effects of subliminally presented face-occupation pairs on decision-making were obtained if participants were later asked about the education, regularity of income, or creativity of the work performed by the depicted individuals (Duss et al., 2011). The subliminal impact on decisions was observed after delays of up to 25 min and was achieved with only one subliminal encoding trial (Duss et al., 2011; Ruch et al., 2016). These findings evince rapid binding of subliminal stimulus pairs into relational long-term memories that can be used flexibly in novel tasks and contexts. One-trial encoding, relation formation, and long-term representation of relations in a flexible format are the defining properties of episodic memory (Cohen and Eichenbaum, 1993; O'Reilly and Rudy, 2001; Henke, 2010). Because memories formed from subliminal stimuli lived up to this computational definition (Reber et al., 2012; Duss et al., 2014; Ruch et al., 2016) it must be assumed that episodic memory formation may proceed with and without conscious awareness (Henke, 2010; Dew and Cabeza, 2011; Hannula and Greene, 2012; Olsen et al., 2012; Ortu and Vaidya, 2013). Such a notion is all the more surprising as the formation and retrieval of episodic memory, mediated by the hippocampus, was long thought to depend on conscious awareness (e.g., Moscovitch, 1995; Squire and Zola, 1996; Tulving, 2002).This view was inspired by the finding that hippocampal lesions impaired declarative (conscious) memory for facts and episodes but spared nondeclarative (non-conscious) forms of memory such as motor skill learning, priming, and conditioning (Squire and Zola, 1996). Studies using implicit or indirect tests to examine nonconscious forms of episodic memory (e.g., Rose et al., 2002; Henke et al., 2003; Greene et al., 2007; Hannula and Ranganath, 2009; Duss et al., 2014; Ryals et al., 2015; Rosenthal et al., 2016) revealed, however, that the function of the hippocampus does not depend on consciousness but on the type of processing involved at learning: the hippocampus mediates the rapid relational encoding and storage of both consciously and unconsciously apprehended information (for reviews see Henke, 2010; Dew and Cabeza, 2011; Hannula and Greene, 2012; Olsen et al., 2012; Ortu and Vaidya, 2013). Both conscious and unconscious relational memories were stored for long-term (Ruch et al., 2016) within the hippocampal memory space (Züst et al., 2015) with overlapping conscious and unconscious relational memories interacting with each other (Henke et al., 2013; Züst et al., 2015).

Here we ask whether unconsciously and consciously acquired relational memories formed in one encoding trial would interact to guide delayed decision-making. What happens if unconscious and conscious memories contain conflicting contents that exert an opposing influence on decisions? How are the two conflicting strands of information weighted? In a day-to-day scenario, this type of conflict may occur if, for example, one reads a news article, which portrays a particular politician in a negative light after having unconsciously integrated surrounding information presenting the same politician in a positive light. How much influence would the unconsciously acquired information have on one's voting behavior? To study potential interactions between unconsciously and consciously formed memories during decision-making, we presented participants with subliminal pairs of faces and written occupations or faces and consonant strings (control condition) for unconscious encoding. Following a resting period of 20 min duration, participants consciously (re-) encoded the same faces presented supraliminally along with either the same written occupations, occupations congruous to the subliminally presented occupations (same wage-category), or incongruous occupations (opposite wage-category). Faces unconsciously encoded with consonant strings (control condition) were presented along with new occupations. Five minutes following the conscious acquisition of face-occupation associations, we assessed the influence of unconsciously and consciously formed memories on decision-making. Participants were presented with the same faces (no occupations) and were instructed to rate each person's putative income on a four-point scale: low, low-average, high-average, or high income. Importantly, participants were encouraged to respond spontaneously and intuitively when deciding about the income of a person. The decision-making task thus constituted an indirect (implicit) test of memory for subliminally (unconsciously) and supraliminally (consciously) presented face-occupation pairs. If conscious thought alone guided income decisions, supraliminal information should determine decision outcomes independently of its congruency with subliminal information. We hypothesized, however, that subliminally formed memories would interact with consciously formed memories to exert an influence on decision-making. To test this hypothesis, we measured the bias that consciously encoded face-occupation pairs exerted on income decisions and examined how this bias was modulated by subliminally presented occupations that were identical, congruous, or incongruous to the supraliminally presented occupations. We expected the decision bias provoked by supraliminal occupations to be increased, reduced, or unchanged depending on whether the same/congruous occupations, incongruous occupations, or no occupations (control condition) had been presented subliminally. Faces that were supraliminally shown as highearning "manager" or low-earning "postman" should thus yield higher or lower income ratings, respectively, if they were subliminally encoded with the same or a congruous occupation instead of an occupation of the opposite wage-category (see **Figure 1**).

# MATERIALS AND METHODS

## General Procedure

We presented subliminal faces paired with written highand low-wage occupations (experimental condition) and faces paired with consonant strings (control condition) for unconscious relational encoding. Participants were ignorant of subliminal presentations. They performed an attention task that directed their focal attention to the screen center while subliminal faces and occupations were presented around the screen center (the method of Degonda et al., 2005). This task ensured that subliminal stimuli were attended, which is important to engage unconscious processing of subliminal information (Marzouki et al., 2007; Finkbeiner and Palermo, 2009). Unconscious encoding was followed by a delay period of 20 min during which participants were asked to relax. Quiet resting is known to improve consolidation and retention of newly acquired memories (Brokaw et al., 2016) and thus was expected to enhance unconscious influence on subsequent decisions. Following resting, participants encoded the same faces presented supraliminally. Faces that had been unconsciously encoded with high- and low-wage occupations were supraliminally presented along with either the same high- or low-wage occupations, with occupations congruous to the subliminally presented occupations (same wage-category), or incongruous occupations (opposite wagecategory). Control faces that had been flashed subliminally with consonant strings were supraliminally presented along with new high- or low-wage occupations. These faces served as baseline condition in which conscious information alone could bias subsequent decision-making. While all participants encoded control faces, the three congruency conditions were implemented in different subgroups (between-subjects factor). This was necessary to reduce memory load and because the German language does not provide enough distinct occupation words to implement every condition in each participant. Participants encoded the supraliminally presented material with an incidental learning task. They were not informed of a later recall test.

Conscious encoding was followed by a brief delay during which participants again were asked to relax (2 min) and then were instructed about the decision task (∼3 min). To measure the common influence of unconsciously and consciously encoded information on decision-making, participants were again presented with all faces to decide on the income of each person (low, low average, high average, high). They were encouraged to respond spontaneously and intuitively and to listen to their gut feeling when deciding on the income. Following each income decision, participants were also asked to remember the actual occupation of the person.

Once the decision task and recall test were completed, participants were questioned about the strategies they had used to rate the income of the depicted persons (open-ended question). This interview should reveal the actually employed strategies in the income rating task. Next, participants were queried on their expectancy and subjective awareness of masked stimuli during the experiment. This inquiry was succeeded by objective awareness tests in which masked stimulus presentations were

immediately followed by forced-choice questions to test explicit stimulus visibility.

The sequence of tasks and delay periods was designed to provide optimal conditions for studying the joint impact of unconsciously and consciously formed long-term memories on decision-making. The delay periods that followed the two encoding runs allowed us to demonstrate that both unconscious and conscious influences on decision-making depended on longterm memory. The duration of delays was chosen such that ∼30 min passed between unconscious encoding and the decision task. This experiment replicates and extends a previous experiment (Ruch et al., 2016), which suggested that unconsciously formed relational memories can bias decision processes across delays of up to half an hour. Importantly, unconscious encoding preceded conscious encoding by a delay period of 20 min. This guaranteed that subliminal stimuli had to be stored for long-term in order to shape subsequent behavior.

#### Participants

A total of 96 right-handed volunteers participated in this study. Sample size was determined based on previous studies (Degonda et al., 2005; Duss et al., 2011; Ruch et al., 2016) and by the number of participants needed to fully counterbalance stimulus lists across conditions. Three participants failed to adhere to instructions, one was under the influence of psychiatric medication, and technical problems occurred during the testing of three further participants. Data of these seven participants were thus excluded from analysis. The remaining sample consisted of 69 women and 20 men between 19 and 59 years of age (mean = 28.15 years, SD = 11.10). Participants were assigned to 3 groups that corresponded to the three experimental conditions "identical," "congruous," "incongruous": identical N = 32 (same face-occupation pairs given for subliminal and supraliminal encoding), congruous N = 29 (same faces plus semantically congruous occupations), and incongruous N = 28 (same faces plus semantically incongruous occupations). Experimenters were aware of the group assignment. We used a cover story ("we study attention and person perception") to keep participants naïve regarding subliminal presentations and the study rationale. This was necessary to render participants susceptible to subliminal information (Verwijmeren et al., 2013). We obtained written semi-informed consent before experimentation. At the end of the experiment, participants were fully debriefed about the study purpose and the presence of subliminal messages and were reminded of the possibility to withdraw their consent to the use of their data. The study was carried out in accordance with the Helsinki declaration and the recommendations of the local ethics committee "Kantonale Ethik Kommission (KEK) Bern, Switzerland." The study protocol and the consent procedure were approved by the local ethics committee.

## Tasks

#### Unconscious Encoding

We used an established forward and backward masking paradigm (Duss et al., 2011, 2014; Ruch et al., 2016) to render face-occupation pairs subliminal for unconscious encoding. Subliminal stimuli were embedded in an attention task. The attention task required participants to hit a key each time a regularly occurring fixation screen (F) was replaced by a screen displaying a horizontal (FH) or vertical (FV) line segment instead of a complete fixation cross (F+). Each trial of the attention task took 1 s and started with a fixation screen (F+, FH, or FV) that was followed by four visual noise masks (M) and two subliminal presentations (S) constituting the sequence F+/H/V–M–S–M–M–S–M. Fixation screens were presented for 233 ms, subliminal stimuli for 17 ms, and masks for 183 ms. Each face-occupation pair was presented 12 times in sequence during six adjacent attention trials spanning 6 s. During this time, one in six fixation screens featured a target (F<sup>H</sup> or FV) that participants had to respond to (see **Figure 2A** and **Supplementary Videos 1**, **2** for an illustration of one subliminal encoding episode). The added total exposure time of a subliminal face-occupation pair was 204 ms (12<sup>∗</sup> 17 ms). Each participant encoded 40 subliminal faces, whereof 10 faces were combined with a low-wage occupation, another 10 faces with a high-wage occupation and the remaining 20 faces with a non-sense consonant string. Unconscious encoding took 4 min.

#### Resting Phase

Following unconscious encoding, participants relaxed on a comfortable reclining seat for 20 min to allow for a period of undisturbed memory consolidation. Half of participants in each experimental condition (identical, congruous, incongruous) relaxed fully and half were required to perform an undemanding acoustic vigilance task, namely to hit a button when hearing a tone played at random intervals of 15–25 s. This manipulation served the aim of finding out whether a deeper relaxation (no task vs. a vigilance task) would benefit memory consolidation. Selfreports in the no-task condition and the number of missed target tones in the vigilance task condition revealed that participants of both groups were equally inclined to fall asleep, suggesting that both conditions were very relaxing. The factor resting condition (no task/vigilance task) was therefore excluded from analyses but the pattern of findings remains the same if it is included.

#### Conscious Encoding

Following rest, participants were presented with supraliminal displays of the same 40 faces that had been flashed subliminally (**Figure 2**). Depending on the experimental condition (varied between subjects), faces were combined with either the same written occupations, congruous occupations or incongruous occupations. Ten of the 20 control faces that had been combined with letter strings for subliminal presentations were now presented with a high-wage occupation and 10 with a low-wage occupation (control condition). Each faceoccupation pair was presented for 4.5 s. The incidental encoding task required participants to give a subjective, intuitive response of how well an occupation matched a person (poor match, poor-average match, good-average match, good match).

#### Decision Task and Explicit Memory Test

occupation; The individuals depicted in this figure gave written informed consent to publication of their image.

We assessed the influence of unconsciously and consciously acquired face-occupation pairs on income ratings regarding these same persons (**Figure 2**). The previously presented 40 faces were displayed again, each for 5 s and without written occupations, for conscious inspection. Participants were asked to decide on the suspected income of each person on a scale ranging from 1 to 4: low (1), low-average (2), high-average (3), high (4; response by key press). They were encouraged to respond spontaneously and intuitively and to focus on the depicted person (rather than on the previously presented supraliminal occupation). The intuitive strategy and the focus on the depicted person should allow the penetrance of the influence of subliminally presented, unconsciously acquired information on decision-making. We used a rating scale rather than a binary response format because we were interested in measuring the average influence of both subliminally and supraliminally encoded occupations on decision-making (occupations can conflict in terms of the associated wage, yielding in combination an average wage) and because the presented occupations were associated with an either high or low wage but not extreme endpoints of wages. Scales allow for more variance and are therefore better suited for the measurement of implicit, unconscious influences. Furthermore, the persons whose income should be rated were selected to look neither wealthy nor poor. Hence, without any knowledge of a person's occupation participants would be inclined to select a mid-range income. Following each income decision, participants were required to try to remember and say the occupation they had earlier read below the face of the person. They were asked to indicate whether they randomly guessed the occupation ("guess"), had a hunch but felt uncertain about their answer ("unsure"), or felt sure they remembered the occupation ("sure"). This cued recall test served to determine the quality of conscious episodic memory.

#### Assessment of Decision-Making Strategy

Following the experiment, participants were asked whether they had used a specific strategy to decide on the income (open-ended question). This interview should reveal the actually employed strategies in the income rating task.

#### Awareness Test

At the end of the experiment, we determined subjective awareness of subliminal stimuli. Participants were asked about any potential awareness of the study purpose and the presence of subliminal stimuli with a series of increasingly specific questions. Finally, we measured awareness of subliminal faces and subliminal occupation words using two objective forcedchoice awareness tests (as described in Ruch et al., 2016, experiment 1). We implemented 20 trials for each test to achieve equal test strength as in the experiment where 20 trials constituted an experimental condition. A new set of 20 faceoccupation combinations was used for subliminal encoding in both the face-awareness test and the occupation awareness test. In both awareness tests, a trial consisted of the subliminal presentation of a face-occupation combination over 6 s (as in the experiment) immediately followed by either a forced-choice between two supraliminally shown faces (target face and a foil face) or between the category "high-wage occupation" and "lowwage occupation." If the chosen wage-category corresponded to the written occupation flashed with a face, the answer was scored as correct. Subliminal stimuli were presented using the same masking paradigm and attention task as in the experiment. Instructions required participants to try to consciously discern the subliminally flashed faces and words, or fragments thereof, in order to inform choices. The order of the two awareness tests was counterbalanced between participants. The percentage of correctly identified faces and wage categories was the dependent variable.

## Materials

We used 80 grayscale images of male faces from the FERET database (Phillips et al., 1998, 2000) which received average income ratings by students in a pilot study that we conducted in preparation of this study. Of the 80 faces, 40 were used in the experiment (20 in the experimental and 20 in the control condition) and 40 in the two awareness tests (20 targets and 20 foils in the forced-choice test for face awareness). Luminance and contrast of all images were equalized. Faces were assigned to 8 lists of 10 faces arranging for an equal distribution of mean income ratings (based on the pilot data), emotional facial expressions, facial hair, and age across lists. Across participants, each face was presented an equal number of times in the experiment and the awareness tests. Within the experiment, each face was presented an equal number of times with a high- and a low-wage occupation and with a consonant string (control condition) for subliminal encoding and with an identical, a congruous and an incongruous occupation for conscious encoding.

We came up with 40 pairs of semantically congruous occupations—such as "director" and "manager" or "postman" and "messenger." Twenty pairs corresponded to common highwage occupations (e.g., director and manager) and 20 pairs to common low-wage occupations (e.g., postman and messenger). The typical wage of occupations was assessed in a pilot study (N = 34) in which participants estimated the income each occupation would yield on a 5-point scale ranging from −2 (low) to +2 (high). The estimated income of high-wage occupations ranged from 0.17 to 1.60 (scale from −2 to+2) with a mean of 0.88 (SD = 0.07). The income of low-wage occupations ranged from −0.42 to −1.61 with a mean of −0.92 (SD = 0.05). Assuming that average income ratings on a 5-point scale are normally distributed, we estimated that high-wage occupation words reflected the upper 41% and low-wage occupations the lower 34% of all possible wages. The total of 80 occupation words were assigned to 4 lists of 10 high-wage and 10 low-wage occupations, of which two lists contained congruous occupations required for the congruous condition in the experiment (A and A′ and B and B′ ). In the incongruous condition, highand low-wage occupations were swapped between faces from subliminal to supraliminal encoding to induce semantic conflict. A and B lists were used in the experiment and awareness tests, respectively, in a counterbalanced fashion. Words in the four lists were matched regarding mean logarithmic word frequency (retrieved from the Leipzig Corpora Collection, http:// corpora.uni-leipzig.de/) and character count. The 20 consonant strings required for the subliminal presentations in the control condition were built by randomly assembling the consonants contained in the selected occupation words. Consonant strings corresponded to occupation words regarding their character count. **Supplementary Table 1** provides a list of all occupation words, with word frequency, character count, and estimated average income (as rated by an independent sample of 34 participants). **Supplementary Table 2** provides a list of all consonant strings.

Stimuli were presented using the software Presentation <sup>R</sup> (Version 16.2, Neurobehavioral Systems, http://www.neurobs. com) on a standard Dell Notebook running Windows XP. A Digital Light Processing (DLP) projector BenQ© SP831 projected the face-occupation pairs (occupation written below face) onto a screen in front of participants in a visual angle of 15◦ width with a resolution of 1,024 × 768 at a refresh rate of 60 Hz.

Responses were recorded with a standard USB keyboard. Participants responded using their right hand, index to little finger, tapping the four neighboring keys [c], [v], [b], [n].

## RESULTS

#### Data Analysis

The dependent variable in the decision task was the subjects' mean income rating (four possible answers: 1 = low, 2 = lowaverage, 3 = high-average, 4 = high) for the 10 faces of each of the four within-subject conditions, i.e., for faces consciously encoded with high- vs. low-wage occupations and unconsciously encoded with occupations vs. consonant strings. The dependent variable in the cued recall test was the percentage of correctly recalled occupations for each condition. Each dependent variable was used in a 2 × 2 × 3 repeated measures ANOVA with the two within-subject factors Occupation/Consonant (subliminal occupation word vs. consonant string) and High-/low-wage (referring to occupations presented supraliminally for conscious encoding) and the between-subjects factor Congruency (identical, congruous, and incongruous).

We assessed normality of the two dependent variables separately for each of the four within-subject conditions, i.e., for faces consciously encoded with high- vs. low-wage occupations and unconsciously encoded with occupations vs. consonant strings. Although, Kolmogorov–Smirnov tests suggested that mean income ratings deviated from normality [all D(89) > 0.09, all p < 0.05], histograms, Q-Q plots, and skewness (range: [−1, 1]) and kurtosis (range: [−2, 2]) values indicated a normal distribution. We therefore used parametric tests to examine our hypotheses. Note, however, that the observed pattern of results (see Section Decision-Making) was replicated when we standardized the raw income ratings per participant (ztransformation) to obtain mean ratings that satisfied all criteria of normality [Kolmogorov–Smirnov tests: all D(89) < 0.08, all p > 0.20, range of skewness & kurtosis: [−1, 1]]. The pattern of results was also replicated when using non-parametric repeated measures ANOVAs as implemented in the function ezPerm of the R-package "ez" (Lawrence, 2016). Percentages of recalled occupations were not normally distributed due to the participants' low recall performance (see Section Conscious Retrieval). Non-normality was suggested by histograms, Q-Q plots, and highly significant Kolmogorov–Smirnov tests [all D(89) > 0.21, all p < 0.001]. The low overall recall performance prevented a meaningful analysis and interpretation of the influence of subliminal information on the conscious recall of supraliminal information.

### Attention Task/Unconscious Encoding

Mean hit rate in the attention task was 88.20% (SD = 9.65) which indicates that participants closely attended to the center of the images provided for the attention task and for subliminal encoding.

## Conscious Retrieval

Participants recalled the occupations of 10.09% of the supraliminally displayed face-occupation pairs (SD = 7.35%). This low recall performance (10% means 4 of the 40 learned face-occupation pairs) probably owes to the incidental encoding instruction, the large amount of learning material, and the fact that only one encoding run was provided. Recall performance was neither affected by wage (factor High-/low-wage) nor Congruency (identical, congruous, incongruous) as suggested by the non-significance of all main effects and all interaction terms in the 2 × 2 × 3 ANOVA (all p > 0.134).

### Decision-Making

Decision-making was significantly influenced by both the supraliminal and the subliminal face-occupation pairs. Income decisions were strongly biased by the wage associated with supraliminal occupations [main effect of High-/low-wage: F(1, 86) = 13.187, p < 0.001, η 2 <sup>G</sup> = 0.033]: participants gave significantly higher income ratings to faces that were supraliminally encoded with high- vs. low-wage occupations. Importantly, this bias was moderated by subliminal occupations as suggested by the significant three-way interaction between Occupation/consonant, High-/low-wage, and Congruency: F(2, 86) = 3.529, p = 0.034, η 2 G = 0.012. Hence, subliminal and supraliminal information both contributed to participants' income decisions. No other main effect or interaction term reached significance (all p > 0.200) in this ANOVA.

To further examine the interaction between subliminal and supraliminal memory formation, we analyzed the differences in mean income ratings between faces supraliminally encoded with high vs. low-wage occupations separately for the experimental and the control condition (faces subliminally presented with occupations vs. consonant strings) and for each congruency group (see **Table 1** for an overview of the mean ratings, and **Table 2**, tests 1–8, and **Figure 3A** for the difference scores). Large positive difference scores suggest a strong decision bias toward the wage category of supraliminal occupations. We examined the changes in these difference scores between experimental and control faces (**Table 2**, test 9–12) and across congruency groups (**Table 3**, tests 1–6), and we contrasted the changes in difference scores between experimental and control faces across all congruency groups (three-way interactions, see **Table 3**, tests 7–9). We performed two-tailed t-tests for all difference scores and contrasts to assess their significance. We further calculated Bays Factors (BF) for these tests to find out whether the data favored the null hypotheses H0 for t-tests that failed to reach significance or for tests where we expected H0 to be true. Bayes Factors were estimated using a half-normal prior distribution with a mode of 0 (reflecting H0, i.e., no difference in income ratings or in difference scores) and a standard deviation reflecting the expected effect size under H1 (Dienes, 2008, 2014). Because participants remembered only ∼10% of the supraliminally displayed occupations, we assumed that consciously and unconsciously encoded occupations would influence decision-making with similar effect sizes. Two recent experiments investigating long-term effects of subliminal processing on decision-making (Ruch et al., 2016) yielded an average effect size of 0.4 (effect size was calculated by dividing the reported t-values obtained for each of the two experiments by the square root of the respective sample size and then computing the average of the two scores). By multiplying this effect size with the pooled standard deviation of the differences in income ratings obtained in the present study (SD = 0.33), we arrived at an expected difference in income ratings of 0.13. This value was used as expected effect size given H1 to calculate Bayes Factors.

Mean income ratings revealed that unconscious and conscious encoding of identical occupations had increased the decision bias, whereas encoding of incongruous as well as congruous occupations reduced the decision bias compared to the control condition, in which conscious encoding alone could influence decision outcomes (**Figure 3**). Most importantly, decisions were strongly biased toward the wage category of supraliminally processed occupations following the unconscious encoding of identical occupations [**Figure 3A**; **Table 2**, test 1: t(31) = 3.40, p = 0.002, BF = 97.44]. The decision bias was marginal or even reversed following the encoding of congruous [**Table 2**, test 2: t(28) = 1.03, p = 0.294, BF = 1.03] and incongruous occupations [**Table 2**, test 3: t(27) = −0.21, p = 0.838, BF = 0.44], respectively. Although bias in the incongruous group was negative and did not significantly differ from zero, the Bayes factor of 0.44 suggests that the data were only 2:1 in favor of the null hypothesis, indicating merely anecdotal evidence for the absence of any bias. Importantly decision bias was stronger in the identical group compared to the congruous [**Table 3**, test 1, t(59) = 1.86, p = 0.068, BF = 3.47] and incongruous group [**Table 3**, test 2, t(58) = 2.34, p = 0.023, BF = 6.77]. In the control condition (subliminal presentation of consonant strings), decisions were

TABLE 1 | Mean income ratings (M) and standard deviations (SD) for each congruency group and each condition (faces consciously encoded with high- vs. low-wage occupations and unconsciously encoded with occupations vs. consonant strings).


TABLE 2 | Decision bias toward consciously encoded occupations.


Differences between mean income ratings for faces consciously encoded with high- vs. low-wage occupations are indicated. Difference scores reflect decision bias toward the wage category of the consciously encoded occupations. Scores are reported separately for both unconscious encoding conditions (occupations and consonants) and for each congruency group (identical, congruous, incongruous, and congruous and incongruous combined). Decision bias is further contrasted between faces unconsciously encoded with occupations vs. consonants (tests 9–12). These contrasts reflect the unconscious influence on decision bias; Two-tailed t-tests with uncorrected p-values and Bayes Factors are reported. Bayes factors were calculated using a half-normal prior distribution with mode = 0 (reflecting H0) and standard deviation = 0.13 (reflecting the expected effect size under the alternative hypothesis H1) as suggested by Dienes (2014). See text for source of H1.

between-subjects manipulated experimental condition (identical, congruous, incongruous) and the within-subject manipulated unconscious encoding condition (faces with occupations vs. consonants). The figure illustrates a strong decision bias following unconscious and conscious encoding of identical information, and a reduced or absent bias following encoding of congruous or incongruous unconscious and conscious information. (B) Plotted is the change in the bias toward consciously encoded occupations ([high]-[low]) for faces that were unconsciously encoded with occupations instead of consonant strings (i.e., [bias for faces encoded with occupations]-[bias for faces encoded with consonant-strings]). This change reflects the unconscious influence on decision bias: positive scores indicate increased, negative scores decreased decision bias toward the consciously seen occupations due to subliminal encoding of face-occupation pairs. The figure illustrates an increased decision bias following unconscious and conscious encoding of identical occupations, and a reduced bias following encoding of non-identical (congruous or incongruous) information. Displayed are group means with SEMs. #p < 0.10, \*p < 0.05 (uncorrected two-tailed t-tests).

biased toward consciously encoded occupations (**Table 2**, tests 5–7, albeit not significantly so for the identical group) and did not significantly differ between congruency groups (**Table 3**, tests 4–6, all p > 0.39, all BF < 0.73). The change in decision bias between faces that had been presented subliminally with occupations vs. consonant strings failed to reach significance for each congruency group (**Table 2**, tests 9–11). If the data of the congruous and incongruous group were pooled in a post-hoc


Between-group differences in decision bias toward consciously encoded occupations (differences of the difference between income ratings for faces consciously encoded with high vs. low-wage occupations) are reported separately for faces unconsciously encoded with occupations and consonants. Furthermore, the contrast in the decision bias between faces unconsciously encoded with occupations vs. consonant strings is compared between groups; Two-tailed t-tests with uncorrected p-values and Bayes Factors are reported. Bayes factors were calculated using a half-normal prior distribution with mode = 0 (reflecting H0) and with standard deviation = 0.13 (reflecting the expected effect size under the alternative hypothesis H1) as suggested by Dienes (2014). See text for source of H1.

analysis, decision bias was significantly reduced compared to the control condition [**Table 2**, test 12, t(56) = −2.26, p = 0.028, BF = 6.47]. Importantly, the change in decision bias between experimental and control stimuli differed significantly between the identical group and both the congruous and the incongruous groups (both p < 0.04, both BF > 4.55; **Figure 3B** and **Table 3**, tests 7–8).

In short, decision bias was increased following the subliminal and supraliminal processing of identical occupations, but not semantically congruous occupations. Encoding of congruous and incongruous occupations reduced the decision bias compared to the control condition in which faces were subliminally paired with consonant strings. Although, these findings reveal that subliminal stimulation altered subsequent decisions, the pattern of results might suggest that participants did not encode the semantic content of subliminal words and did not form semantically precise relational memories for face-occupation pairs.

#### Assessment of Decision-Making Strategy

Participants were instructed to decide spontaneously and intuitively and to focus on the depicted person (not the occupation they had earlier read below the face) when making their income decision. These instructions should allow the penetrance of the influence of available unconscious, implicit knowledge on decision-making. Note however, that income decisions in all trials were immediately followed by a direct test of explicit memory: participants were required to remember the supraliminally presented occupation. This second task might have motivated participants to first try to retrieve the person's occupation and then to rate the income accordingly. This explicit-memory-informed strategy might reduce the impact of subliminally presented faceoccupation pairs on decision-making. We therefore tried to determine the strategies that participants had used to form their decisions.

We performed a qualitative analysis of participants' answers to the post-experimentally posed question of what strategy they had used to rate the income. 70.1% of participants reported having focused on aspects of the depicted person (e.g., charisma or emotional expression, age, cleanliness, facial features such as hairline, similarity with familiar persons, attractiveness). 34.8% of participants explicitly mentioned that they had responded intuitively (note that responses were non-exclusive). Only 12.4% of participants reported that they had tried to remember the associated occupation first only to then deduce the income of the person. The qualitative pattern of results for the decisionmaking task did not change when these participants were excluded from the analysis: the critical three-way interaction between Occupation/consonant, High-/low-wage, and Congruency on income decisions remained significant [F(2, 75) = 3.224, p = 0.045]. This suggests that findings were not affected by the subset of participants who employed a memory-based decision strategy.

To determine participants' decision strategy in a more objective way, we calculated their "decision accuracy," i.e., the percentage of decisions that corresponded to the wage-category of the supraliminal occupation (% of ratings 3 and 4 for high- and 1 and 2 for low-wage occupations). We determined each participant's accuracy averaged across all faces as well as separately for faces for which occupation retrieval succeeded vs. failed. Classification accuracy averaged across all faces was low (53.0%, SEM = 0.9%) but exceeded chance level of 50% [t(88) = 3.26, p = 0.002]. Hence, decisions were only weakly influenced by supraliminally seen occupations, which suggests that participants decided intuitively. Importantly, accuracy was much higher for faces for which participants succeeded vs. failed to remember the supraliminally presented occupation [76.6 ± 3.1% vs. 50.6 ± 1.0%; main effect of Retrieval: F(1, 81) = 59.715, p < 0.001]. Accuracy of income ratings (based on supraliminally presented occupations) exceeded chance level only if occupations were remembered [t(83) = 8.56, p < 0.001. vs. t(88) = 0.55, p = 0.583]. This suggests that participants relied on the wage-category of supraliminally presented occupations only in trials, where encoding and later explicit retrieval were successful. However, the explicit retrieval may have been delayed to the second task, where it was demanded. This would reduce the influence of supraliminally formed memories at that moment to a preconscious effect that influenced intuitive decisions along with any unconscious effect due to subliminal encoding. The fact that decision accuracy was only 75% even when occupations were remembered speaks in favor of this assumption.

The rather weak influence of supraliminally seen occupation words on income decisions as well as the fact that only a few of these occupations were remembered might suggest that participants employed an intuitive rather than a memorybased decision strategy. Nevertheless, it is still possible that the mere attempt to retrieve the face-associated occupation altered decision-making even if retrieval failed. To assess this possible interaction between the explicit memory test and the decisionmaking task, we analyzed the correlation between income decisions and the wage-category of subsequently reported but incorrect occupation words at the single-trial level. Only trials in which participants named an incorrect occupation that was not related to the face-associated supraliminal occupation were analyzed. Furthermore, only those occupations were analyzed for which the associated wage-category could be determined, i.e., occupations that were very similar to one of the occupation words used for this study. Income ratings were strongly correlated with the wage-category of occupations reported in the explicit memory test even if participants failed to recall the correct occupation (r<sup>S</sup> = 0.516, p < 0.001, N = 1,902 trials). This might suggest that participants tried to come up with an occupation from which they could deduce the income of a face instead of deciding intuitively. However, it is equally likely that participants intuitively decided on the income of a face and then tried to find a matching occupation in the memory test. The fact that participants had to decide rapidly (within 5 s) speaks for the latter interpretation.

#### Assessment of Potentially Confounding Factors

Because Congruency was manipulated across subjects, it is important to verify that participants' general decision-making style was comparable across groups and that income decisions varied due to subliminal and supraliminal stimulation, not due to a sampling bias. Congruency groups neither differed in their overall income-ratings nor in the way they were influenced by supraliminally presented information, i.e., in their general decision bias. This was suggested by the fact that neither the between-subjects effect Congruency [F(2, 86) = 1.536, p = 0.221, η 2 <sup>G</sup> = 0.014), nor the interaction of Congruency with the wagecategory (High-/low) of supraliminal occupations [F(2, 86) = 1.051, p = 0.354, η 2 <sup>G</sup> = 0.005] reached significance in the overall ANOVA. As reported earlier, only the three-way interaction Occupation/consonant, High-/low-wage, and Congruency reached significance. Hence, the content of subliminal stimuli, not group membership per-se, moderated the observed differences in decision-making across Congruency groups. This finding was confirmed if the same ANOVA was performed for only those faces that were subliminally encoded with consonant strings (control condition): only the main effect of High-/lowwage reached significance [F(2, 86) = 11.952, p < 0.001, η 2 G = 0.046]. Congruency and the interaction between Congruency and High-/Low-wage again failed to reach significance (all p > 0.468). Hence, participants in the different congruency groups gave similar income ratings and showed a similar decision bias for faces that were not associated with a (identical, congruous, or incongruous) subliminal occupation. Only for faces subliminally presented with occupations, did the interaction between Congruency and High-/Low-wage reach significance [F(2, 86) = 3.337, p = 0.040, η 2 <sup>G</sup> = 0.069]. Hence, the decision bias varied across Congruency groups only with respect to subliminally presented occupation words, not due to an overall group bias.

Furthermore, none of the following variables differed significantly between Congruency groups: response latencies during decision making [F(2, 86) = 0.558, p = 0.575], explicit recall performance [F(2, 86) = 0.221, p = 0.802], age [F(2, 85) = 1.180, p = 0.312], hit rate on the attention task given during unconscious encoding [F(2, 86) = 0.551, p = 0.578), gender distribution [χ 2 (2) = 0.784, p = 0.676], awareness of subliminal stimuli (see Section Awareness Test), self-reported decisionmaking strategy [intuition: χ 2 (2) = 4.26, p = 0.119; focusing on aspects of faces or persons: χ 2 (2) = 0.107, p = 0.948, focusing on associated occupation: χ 2 (2) = 0.205, p = 0.903], and decision accuracy (see Section Assessment of Decision-Making Strategy), i.e., the percentage of income decisions that corresponded to the wage-category of supraliminally seen occupations (all p > 0.418). This further suggests that participant groups were highly comparable.

#### Awareness Test

The inquiry into subjective awareness revealed that no participant had suspected subliminal stimuli or noticed faces, words, or fragments thereof between masks. The objective awareness tests corroborated this result. Performance on the objective face and occupation awareness test did not significantly deviate from the chance level of 50% [face awareness test: mean accuracy = 50.17%, 95% CI [47.65, 52.70], t(86) = 0.136, p = 0.892; occupation awareness test: mean accuracy = 49.11%, 95% CI [47.02, 51.20], t(88) = −0.844, p = 0.401]. Note that non-significance does not necessarily mean that participants were truly unaware (i.e., that the null hypothesis is true). It is possible that our tests were not sensitive enough to distinguish between the null and alternative hypothesis (Dienes, 2014). To assess if there was substantial evidence for the absence of awareness, we calculated Bayes Factors (BF) for the two t-tests. We assumed that awareness for subliminal stimuli would yield a recognition performance of about 5% above chance level. This is the performance achieved in tests that assessed indirect influences of subliminal face-occupations pairs on decisionmaking (about 6% in (Duss et al., 2011); about 4% in Ruch et al., 2016). We therefore used a half-normal prior distribution with a mode of 0% (reflecting chance-level performance) and a standard deviation of 5% (expected performance in case of awareness) to calculate BFs (Dienes, 2008, 2014). This yielded a BF of 0.27 for the awareness of faces (mean = 0.17% above chance, standard error of sample SE = 1.27%) and 0.12 for the awareness of occupations (mean = −0.89%, SE = 1.05%). Bayes Factors were below 1/3, suggesting that there is substantial evidence for the null assumption that participants were not aware of masked faces and occupation words. Furthermore, performance in neither of the two tests differed between the condition groups (identical, congruous, incongruous): face awareness test F(2, 84) = 0.993, p = 0.375, η 2 <sup>p</sup> = 0.023; occupation awareness test F(2, 84) = 0.092, p = 0.912, η 2 <sup>p</sup> = 0.002. Hence, between-group differences discovered in the experiment are unlikely to reflect differences in the awareness of subliminal stimuli.

# DISCUSSION

Decision-making was influenced by both subliminally and supraliminally displayed combinations of faces and written occupations. Income decisions were biased toward the wage category of supraliminally presented occupations, and this bias was either increased or reduced by subliminal encoding, depending on whether subliminal and supraliminal occupations were identical or not. Identical subliminal and supraliminal occupations added up, yielding the strongest observed bias on income decisions, whereas non-identical (incongruous and semantically congruous) occupations canceled each other out, leaving no clearly discernible or even a numerically inverted decision bias (illustrated in **Figure 2B**). This finding speaks to the power and longevity of subliminally encoded memories because at least 30 min passed between subliminal encoding and decision-making. Post-experimental awareness tests demonstrated that subliminal faces and occupations were presented below the objective threshold of conscious awareness. We deduce from this finding that the subliminal face-occupation combinations presented in the experiment guided decision-making completely outside of conscious awareness. Because participants remembered only about 10% of the supraliminally provided occupation– face combinations, choice behavior in many trials was likely also influenced by supraliminally provided information that was forgotten and hence only unconsciously available. This further emphasizes the role of non-conscious memories in decision-making.

Our results might suggest that subliminally presented faces and occupations were not bound into semantically precise relational memories but merely primed subsequent conscious processing of words and faces due to enhanced perceptual fluency. In fact, the subliminal presentation of faces paired with occupation words instead of consonant strings increased the influence of supraliminal face-occupation pairs on subsequent decisions only if subliminal and supraliminal occupations were identical. Subliminal presentation of non-identical but semantically congruous occupations that reflected the same wage category and a similar field of work (e.g., "historian" and "archeologist") reduced the supraliminal influence on decisions. This might suggest that subliminal stimulation merely enhanced the perceptual fluency of distinct faces and word forms without leaving a relational memory trace. According to this perspective, an enhanced word and face fluency would facilitate the subsequent conscious encoding and retrieval of identical faces and words which would boost the supraliminal influence on income decisions. Changes in perceptual fluency due to subliminal stimulation might also account for the finding that the supraliminal influence on decision-making was impaired following subliminal presentation of faces paired with non-identical (congruous and incongruous) occupation words compared to consonant strings (control). If meaningful subliminal words undergo more cognitive processing than nonwords, words might reduce the resources available for the processing of subliminal faces. Hence, subliminal faces might be better processed in the control than the experimental condition, which would facilitate encoding and retrieval of supraliminal faces and would enhance the supraliminal influence on income decisions. This scenario of perceptual priming thus accounts for both subliminal effects—the enhanced decision bias for identical face-occupation pairs and the reduced decision bias for non-identical pairs. This parsimonious interpretation of results contradicts, however, the well-established fact that onetrial subliminal priming is short-lived and limited to the primeadjacent target (for a review see Bodner and Masson, 2014). So far, only a few studies reported longer lasting subliminal priming effects. But these could not fully exclude prime awareness (Francken et al., 2011; Muscarella et al., 2013) and they provided only neuronal but no behavioral evidence for priming (Gaillard et al., 2007) or repeatedly presented the same stimulus over many encoding episodes to achieve long-term priming (Chen et al., 2009). Yet, here we observed long-lasting subliminal influences on the behavior in participants who were completely unaware of subliminal stimulation and who received only one subliminal encoding trial. Hence, this pattern of results is atypical for subliminal perceptual face and word priming. Yet, it nicely lines up with previous findings of unconscious relational learning from subliminal stimuli (e.g., Duss et al., 2011; Ruch et al., 2016), as argued below.

Although priming alone might account for our findings, we favor the possibility that subliminally presented face-occupation pairs were actually stored in semantically precise relational memories that guided later decisions. Such a scenario is suggested by previous evidence from experiments with similar designs (Duss et al., 2011; Reber and Henke, 2011; Reber et al., 2012, 2014; Ruch et al., 2016). For example, the subliminal presentation of face-occupation pairs was found to shape subsequent conscious semantic decisions regarding the income, education, and creativity of these same individuals (Duss et al., 2011; Ruch et al., 2016). The subliminal presentation of foreign words with translation words (e.g., "gumpel = dog") also helped participants distinguishing between supraliminally displayed correct and incorrect translations of the same foreign words (e.g., "gumpel = hound" vs. "gumpel = spear," see Ruch et al., 2016). As in the present study, only one or two subliminal encoding episodes for each stimulus pair were required to shape decisions that were delayed by up to half an hour. Hence subliminal stimulus pairs were rapidly encoded into lasting memories. Relational encoding of semantic and perceptual associations calls on the hippocampus (Henke et al., 1997; Holdstock et al., 2002; Prince et al., 2005), which is able to rapidly encode information and store relations for long-term. Perceptual priming on the other hand depends on neocortex (Henson, 2003), which requires many identical learning episodes to form lasting memories for single items (McClelland et al., 1995). This explains why subliminal priming studies, most often using single-item stimuli, yielded short-lived subliminal effects (Forster et al., 1990; Ferrand, 1996; Greenwald et al., 1996), whereas studies presenting pairs of stimuli that trigger relational processing yielded lasting subliminal influences. The assumption that long-lasting unconscious associations were formed after a single presentation of a subliminal face-occupation combination contradicts the classic view that consciousness is a precondition for rapid relational learning (Moscovitch, 1995; Squire and Zola, 1996; Tulving, 2002; Mitchell et al., 2009). It is, however, in line with much research implying the contrary (Henke, 2010; Dew and Cabeza, 2011; Hannula and Greene, 2012; Olsen et al., 2012)—in particular that humans can rapidly form and later retrieve unconsciously formed semantic associations (Duss et al., 2011; Ruch et al., 2016) by way of hippocampus (Degonda et al., 2005; Reber et al., 2012; Duss et al., 2014; Züst et al., 2015).

Our assumption that subliminally displayed faces and words yielded semantically precise relational memories raises the question why subliminal stimulation had enhanced the influence of conscious encoding on decision-making in the identical but not the congruous condition. Congruous occupations shared the same wage-category and a similar field of work. However, congruous occupations were not synonymous (e.g., "historian" vs. "archeologist," or "tailor" vs. "shoemaker"), i.e., they still reflected different professional activities and were associated with different incomes (see **Supplementary Table 1** for a list of all occupation words). The semantic distance between congruous occupations was probably too large or too salient for participants to synthesize congruous unconscious and conscious faceoccupation memories (e.g., "person X is a tailor" and "person X is a shoemaker") into an unambiguous mental representation. As a consequence, participants may have failed to integrate the two "congruous" associations to infer a high or a low income.

If unconscious relational learning rather than priming is assumed to account for our results, the question about the mechanism of subliminal influences on decision-making arises. Did subliminally and supraliminally formed memories exert independent effects on decision-making? Did subliminally formed memories modulate the subsequent encoding of supraliminal information? Were subliminally and supraliminally formed memories integrated into one single memory representation or two separate but semantically overlapping memory representations? The current experimental design and the present data cannot distinguish between these mechanisms. To unravel the precise nature of interactions between overlapping unconscious and conscious relational memories, sophisticated neuroimaging techniques in combination with multivariate pattern classifier analyses will be required (Chadwick et al., 2011).

The scope of our findings may be limited by several aspects of the experimental design. First, the main factor of interest—congruency between subliminal and supraliminal information—was varied between rather than within subjects. This opens up for the possibility that individual differences in an unknown variable (e.g., the ability to recognize faces; Wilhelm et al., 2010) rather than congruency between subliminal and supraliminal content accounted for the observed differences between participant groups. However, neither responses in the control condition (which was identical for all participants), nor any of the possible confounding factors that we analyzed varied systematically between participant groups. Hence, there is solid ground to assume that the content of subliminal and supraliminal stimuli rather than a sampling bias accounted for the present results. A further problem is the combination of the implicit income rating task with the explicit memory task within each trial because it might motivate participants to solve the implicit task using explicit memory. This explicit memory-based decision strategy might have prevented participants from responding spontaneously and intuitively, which would have reduced unconscious influences on decisionmaking. However, only a few participants reported to have employed an explicit memory-based income rating strategy. Finally, our findings are limited by the fact that explicit memory for supraliminally presented information was poor. If the conscious encoding of novel face-occupation associations yielded poor memories, unconscious encoding might have failed, too. Yet, we have repeatedly shown that humans are capable of unconsciously acquiring novel information about subliminally presented face-occupation pairs (Degonda et al., 2005; Duss et al., 2011; Züst et al., 2015; Ruch et al., 2016). Nevertheless, future studies on conscious-unconscious interactions in decision-making may be well-advised to (i) employ within-subjects designs, to (ii) implement independent test-sessions for the assessment of intuitive decision-making and conscious memory retrieval, and to (iii) use stimulus-material that is easier to memorize than novel faces and occupation words.

Our findings confirm the view that unconsciously processed information can be rapidly encoded into long-term memory (Gaillard et al., 2007; Chen et al., 2009; Duss et al., 2011; Chong et al., 2014; Ruch et al., 2016). Ruch et al. (2016) had already reported that subliminally presented faces combined with written occupations were stored in memory for almost half an hour to bias delayed decisions about the income of the depicted persons. Here, subliminally presented face-occupation pairs altered income decisions even if the same faces were later presented supraliminally along with potentially conflicting occupations. This indicates that unconsciously acquired knowledge is robust. Our findings further suggest that decisions are not determined by conscious processes alone but may simultaneously benefit from conscious and unconscious knowledge. While some scholars argue that learning and decision-making are strictly governed by consciousness (e.g., Shanks, 2010; Newell and Shanks, 2014) our data support the view that unconscious processes play a larger role in everyday cognitive functioning than commonly assumed (Rosenthal, 2008; Soon et al., 2008; van Gaal et al., 2012; Hassin, 2013).

### AUTHOR CONTRIBUTIONS

SR and KH developed the study concept and designed the experiment. SR collected and analyzed the data. KH and SR contributed to the final analyses and the interpretation of the data. BH wrote a first draft of the manuscript, KH and SR wrote the final report. All authors approved the final version of the manuscript for submission.

# FUNDING

This work was supported by Swiss National Science Foundation Grant K-13K1-119953 to KH.

#### REFERENCES


# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg. 2017.01542/full#supplementary-material

Supplementary Video 1 | Video of one subliminal encoding trial. Video must be played at 60 fps to provide an accurate representation of the stimulus sequence. The embedded subliminal image consists of a male face and the written word "Manager." A schematized image of a face instead of one of the original faces is used. The individual depicted in this video gave written informed consent to publication of his image.

Supplementary Video 2 | Video highlighting the subliminal image in one subliminal encoding trial. This video is identical to Supplementary Video 1, but the contrast of stimulus masks was reduced to enhance the visibility of the subliminal images. These masks were not used in our experiment. The individual depicted in this video gave written informed consent to publication of his image.

Supplementary Table 1 | List of all occupation words used in the current study. Word-frequency obtained from the Leipzig Corpora Collection (http://corpora.unileipzig.de/), word length (number of letters), and mean estimated income as rated by an independent sample of 34 participants are indicated. Words are grouped by stimulus lists (A, A′ , B, and B′ ) that were used to counterbalance stimuli across conditions, and by wage-category. Pairs of semantically congruous occupation words are labeled with the same identification number.

Supplementary Table 2 | List of all consonant strings with corresponding word lengths (number of letters).


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer GH and handling Editor declared their shared affiliation.

Copyright © 2017 Ruch, Herbert and Henke. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# **Influence of Suboptimally and Optimally Presented Affective Pictures and Words on Consumption-Related Behavior**

*Piotr Winkielman1,2\* and Yekaterina Gogolushko1*

<sup>1</sup> Department of Psychology, University of California, San Diego, San Diego, CA, United States, <sup>2</sup> Faculty of Psychology, SWPS University of Social Sciences and Humanities, Warsaw, Poland

Affective stimuli can influence immediate reactions as well as spontaneous behaviors. Much evidence for such influence comes from studies of facial expressions. However, it is unclear whether these effects hold for other affective stimuli, and how the amount of stimulus processing changes the nature of the influence. This paper addresses these issues by comparing the influence on consumption behaviors of emotional pictures and valence-matched words presented at suboptimal and supraliminal durations. In Experiment 1, both suboptimal and supraliminal emotional facial expressions influenced consumption in an affect-congruent, assimilative way. In Experiment 2, pictures of both high- and low-frequency emotional objects congruently influenced consumption. In comparison, words tended to produce incongruent effects. We discuss these findings in light of privileged access theories, which hold that pictures better convey affective meaning than words, and embodiment theories, which hold that pictures better elicit somatosensory and motor responses.

#### *Edited by:*

Marcus Rothkirch, Charité Universitätsmedizin Berlin, Germany

#### *Reviewed by:*

Zoltan Dienes, University of Sussex, United Kingdom Chris Loersch, University of Colorado Boulder, United States

> *\*Correspondence:* Piotr Winkielman pwinkiel@ucsd.edu

#### *Specialty section:*

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

*Received:* 31 December 2016 *Accepted:* 12 December 2017 *Published:* 29 January 2018

#### *Citation:*

Winkielman P and Gogolushko Y (2018) Influence of Suboptimally and Optimally Presented Affective Pictures and Words on Consumption-Related Behavior. Front. Psychol. 8:2261. doi: 10.3389/fpsyg.2017.02261 **Keywords: affect, priming, emotion, consciousness, behavior**

# **INTRODUCTION**

What is the relationship between affect and cognition? Earlier debates saw some strong competing claims about the separation of the presumed "systems," primacy of one system over the other, and the minimal processing necessary to trigger affective vs. cognitive reactions (Lazarus, 1984; Zajonc, 1984, 2001). Recent years have witnessed a growing consensus that affective and cognitive processes are tightly intertwined, in terms of both their psychological function and neural substrates, and the effort moved to understand the mechanisms of this connection (Clore and Colcombe, 2003; Pessoa and Adolphs, 2011; LeDoux, 2012; Winkielman et al., 2015a; Barrett, 2017).

In empirical contributions, rather than simply contrasting processing with and without awareness to test for "primacy," the effort moved on to manipulations of stimulus and task variables that may highlight the affective and the cognitive component of processing. Along with manipulations of stimulus visibility, such manipulations help reveal possible differences in "hot" vs. "cold" contributions, specify the role of consciousness, and examine whether these differences actually matter in terms of actual behavior. In this paper we contribute to this effort by exploring the influence of affective pictures and affective words on participants' spontaneous behavior toward a novel stimulus, and their subjective experience. Before we outline the current studies, we offer some background on the topic of affective influence.

# **Affective Influence**

Affective and emotional influence is a topic of a large and growing literature (for reviews, see Winkielman et al., 2007, 2015b; Lerner et al., 2015). Many studies in this literature examine how exposure to an affective stimulus (e.g., valenced picture) changes evaluative responses to a neutral target that immediately follows (Niedenthal, 1990; Murphy and Zajonc, 1993; Winkielman et al., 1997; Payne et al., 2005). These studies have found that an affective stimulus can influence a subsequent response, with direction of the influence depending on several factors. Many studies report affectively congruent, or assimilative effects—with valence of the prime facilitating responses of similar valence. Thus, in the Fazio et al. (1986) experiments, presentation of a positive word (e.g., wedding) shortened classification of another positive word (e.g., puppy). Similarly, in several experiments, subliminally1 presented happy faces, as opposed to angry faces, enhanced ratings of neutral Chinese characters (Murphy and Zajonc, 1993; Winkielman et al., 1997). Such congruent effects are especially likely when primes are subliminal, or supraliminal but unobtrusive (e.g., participants perform some irrelevant task on a visible prime). However, researchers have also observed affectively incongruent, or contrast effects, when the prime facilitates responses opposite of its valence. Such incongruent effects can reflect correction processes, as when the prime is blatant (Murphy and Zajonc, 1993) or evaluatively extreme (Glaser and Banaji, 1999) as well as participants' response strategies (Kiefer et al., 2017). Importantly, these transitions from assimilation-to-contrasts can also be due to automatic mechanisms that operate even with subliminal primes (Hermans et al., 1994), including the dynamics of activation and saturation (Irwin et al., 2010). As such, the study of affective influences is informative about the dynamics of less and more conscious or attentive modes of processing.

# **Affective Influences on Spontaneous Behavior**

The just mentioned studies explored the influence of affective stimuli on responses that are (i) produced immediately following the prime, and (ii) relatively conceptual (e.g., evaluative categorization or judgment). However, over the last decade or so, there has been a growing interest in influences on behaviors that are (i) temporally distant from the priming episode and (ii) spontaneous and unconstrained. Much of this research involves an unobtrusive activation of a "cold" cognitive concept (e.g., by flashing the word "bet") and later assessment of spontaneous behaviors (e.g., a decision to gamble). There is a recent debate about these phenomena (Cesario, 2014). Some worry about whether they should be called "priming," are better characterized as "implicit memory" (e.g., source confusion), or are just a form of "unobtrusive suggestions" and "implicit influence." Others worry about the empirical robustness of some of these "priming" phenomena. Yet, there is evidence that at least some of such effects can be reliably obtained (Payne et al., 2016).

In our earlier work, we have explored whether exposure to affective, rather than cognitive, stimuli can also influence a spontaneous behavior. Winkielman et al. (2005) exposed participants to a longer series (eight exposures) of subliminally presented happy and angry faces under the disguise of a gender classification task. Following the "priming" (affect induction) episode, participants were given a large pitcher with a novel beverage and were asked to simply pour themselves as much beverage as they wanted and to drink as much as they wanted. We chose this task because of a large literature showing that willingness to explore novel items, including foods and drinks, is a sensitive indicator of the organism's affective state (for review, see Winkielman et al., 2007). As expected, after being primed with happy faces participants poured more beverage and drank more of it than participants primed with angry faces. Interestingly, despite eliciting affect-congruent changes in spontaneous behavior, the primes had no effect on participants' conscious mood (Winkielman and Berridge, 2004; for a similar result see also Winkielman et al., 1997; Zemack-Rugar et al., 2007). Subsequent research has found the impact of visually suppressed emotional faces on a variety of reactions, behaviors, judgments, and decisions (e.g., Sweeny et al., 2009; Bornemann et al., 2012; Almeida et al., 2013; for a review see Axelrod et al., 2014).

#### **Generality and Mechanisms**

But, how general are these behavioral effects, and what are their mechanisms? In the current studies we extend our previous research by comparing the influence of briefly and optimally presented affective pictures (faces and scenes) with the influence of valenced-matched words. This comparison can help us to not only understand the parameters of affective influences on behavior, but also may inform the debates about the connection between affect and cognition, and the role of consciousness.

#### **Accounts of Affective Influence**

One standard way of thinking about affective influences on behavior is the standard associative memory framework (Bower, 1991). Primes activate valence-congruent material in memory which then guides the interpretation and response to the target stimulus according to principles of applicability (Higgins et al., 1977). This "cold" semantic account can explain some cases of affective influence (Forgas, 2002), but there are also reasons to consider "hot" alternatives. Some arguments for this come from neuroscience research highlighting an important role of affective circuitry in guiding perceptual, attentional, memorial, and action-oriented processes (Phelps, 2005; Pessoa, 2013). More pertinent here are arguments from psychology. One piece of evidence is the relatively greater impact of affective than descriptive dimensions of the same priming stimulus. For example, Murphy and Zajonc (1993) reported that under subliminal exposures, judgments of neutral targets were influenced by the prime's affective dimensions (the valence of facial expression) but not by descriptive dimensions (face

<sup>1</sup>To avoid strong claims about the unconscious, Murphy and Zajonc (1993) used the terms "suboptimal" to refer to very brief presentation conditions. In this paper, we treat the terms "subliminal" and "suboptimal" as equivalent, but also wish to avoid making strong claims about the existing literature and our own presentation conditions.

gender or age). However, it is possible that these findings are limited to faces, especially to faces presented at short durations due to privileged processing of faces in general (Farah et al., 1998) or affective faces in particular (Öhman, 2002). In fact, other studies found comparable priming effects on lexical decisions with affect- and gender-related words presented at minimal durations (Greenwald et al., 1996). Further, the relative ease of obtaining affective, rather than descriptive priming might sometimes be due to the experimenter's selection of stimuli. Specifically, if affective stimuli form more homogenous sets (i.e., stimuli are clearly positive or clearly negative) than descriptive stimuli, the extraction of the affective dimension will be facilitated (Storbeck and Robinson, 2004). On the other hand, the "hot" interpretation of Murphy and Murphy and Zajonc's (1993) effects is consistent with recent evidence that autistic participants (with atypical affective functioning) do not show the typical advantage in processing affective, as opposed to gender, information from faces (Clark et al., 2008). Finally, the "hot" account is supported by observations that affective stimuli can sometimes influence behaviors that tap a person's affective state, but not descriptive measures that tap accessibility of semantic concepts. For example, as mentioned earlier, Winkielman et al. (2005) observed priming effects on hedonic behavior (pouring and drinking), but not on mood ratings. Presumably, if the underlying process involved semantic activation of affective concepts, then exposure to affective primes should have also influenced participants' responses on descriptive measures (Higgins et al., 1977). However, it is possible that some behavioral measures are more sensitive to priming (cognitive or affective) than more descriptive measures, such as self-reports.

#### **Embodied Accounts**

Recently, a different perspective on phenomena of cognitive and affective influence has been offered by the framework of embodied or grounded cognition (Barsalou, 1999, 2010; Niedenthal et al., 2005; Niedenthal, 2007; Winkielman et al., 2015b). This framework acknowledges that processing can proceed in a "disembodied" fashion, relying solely on the semantic memory system. However, processing can also be "embodied," where perceivers recruit somatosensory states that occur during the actual experience with objects in the world. Importantly, the degree of "embodiment" depends on the nature of the stimulus (e.g., perceptual vs. conceptual) and task demands (e.g., shallow vs. deep processing). That is, stimuli vary in the degree they activate somatosensory resources, and response production can also differentially draw on somatosensory resources (Barsalou, 2010). Translated into the domain of affective influence, the embodiment framework suggests that affective influence can occur in a "disembodied" way (i.e., involve only activation of relevant evaluative concepts) or in an "embodied" way (i.e., involve engagement of genuine affective reactions). One interesting empirical prediction from this account is that affective stimuli that recruit an embodied response should exert greater influence on certain kinds of behavior than comparable affective stimuli that do not recruit an embodied response. Specifically, stimuli that elicit somatosensory reactions should influence behaviors that require participants to engage in an "embodied simulation," and thus draw on the primed somatosensory resources. In contrast, behaviors that do not require embodied simulation should not be influenced (see Kan et al., 2003; Solomon and Barsalou, 2004). For example, one should see embodiment effects on processes that lead to action to be taken with the stimulus (e.g., deciding how much to pour and drink), but not on purely conceptual responses (e.g., simple ratings).

# **Current Studies**

The current studies sought to compare the influence of affective pictures and words presented at different durations on behavior using the procedure from Winkielman et al. (2005). Specifically, we predicted that pictures would have a stronger influence on consumption behavior than matched emotional words. This prediction was based on two considerations.

The embodiment approach predicts greater influence of pictures because bodily responses are more easily engaged by affective information presented by pictures rather than by written words. For example, Larsen et al. (2003) found stronger electromyographic (EMG) responses to pictures (IAPS set) than words (ANEW set), even though the stimuli were matched in self-reported valence and arousal (Bradley and Lang, 1999; Lang et al., 2005). Similarly neuroimaging studies show that pictures of emotional scenes and emotional facial expressions induce robust activations in the amygdala and related structures (Hariri et al., 2002; Norris et al., 2004). In fact, facial expressions elicit these activations even when presented unobtrusively (Critchley et al., 2000) or unconsciously (Whalen et al., 1998; Morris et al., 1999). In contrast, studies with emotional words reveal much weaker effects, unless researchers use very strongly charged stimuli (Isenberg et al., 1999) or very sensitive measurement techniques, such as intracranial recording directly in the amygdala (Naccache et al., 2005).

Another reason why pictures should have greater influence than words comes from the literature suggesting that pictures have a privileged access to the cognitive system (Potter et al., 1986; Glaser, 1992), including the network representing affective information (De Houwer and Hermans, 1994). Consistent with this assumption, several cognitive studies found that pictures are more effective as primes, and a recent study found a stronger impact of pictures than words in an evaluative priming task (Spruyt et al., 2002). We come back to the issue of similarities and differences between the embodiment and cognitive access account in the discussion.

We examined the differences between pictures and words in two studies. In both studies, the primary manipulation was the valence (positive vs. negative) and format (picture vs. word) of the affective prime. Specifically, Experiment 1 compared the impact of emotional facial expressions vs. valence-matched emotional words, whereas Experiment 2 compared the impact of pictures of various emotional objects to valence-matched words representing the same objects. For both experiments, we predicted greater behavioral impact of affective pictures than words. The experiments also examined several other issues, as follows.

We also wanted to examine the role of awareness and processing amount on the impact of pictures and words. As discussed earlier, on some views awareness may lead to a reversal of affective influence—switching from assimilation to contrast. Alternatively, greater awareness could enhance the impact of the stimulus due to greater processing. Finally, we could observe comparable effects due to early nature of processing of emotional facial expressions (e.g., Critchley et al., 2000).

The processing manipulation also speaks to possible concerns that picture stimuli are easier to physically perceive or comprehend than word stimuli, thus explaining their greater effects on behavior (Glaser, 1992). To address these questions, we added two manipulations. In Experiment 1, we manipulated stimulus duration, with some stimuli presented subliminally and other stimuli supraliminally (with priming made unobtrusive by focusing participants on a valence-irrelevant dimension). In Experiment 2, we manipulated stimulus frequency, with some stimuli representing highly frequent, easily recognizable everyday objects and other stimuli representing less frequent objects. If the difference in behavioral impact of pictures and words holds across duration and frequency manipulations, this suggests that the difference cannot be simply due to perceptibility or comprehension.

Finally, in both experiments, we measured participants' motivational state (hunger and thirst). Our earlier research found that affective priming of consumption behavior with facial expressions was stronger for thirsty participants (Winkielman et al., 2005). Thus, we sought to examine whether Experiment 1 would replicate this effect using facial expressions, and whether Experiment 2 would yield a similar effect using emotional scenes (e.g., dog, gun) which, unlike facial expressions, are less likely to be processed via low-level circuitry and might be less directly tied to approach-avoidance.

The primary dependent measure in both experiments was consumption behavior—the amount of a novel beverage that participants poured and drank. We selected consumption behavior for several reasons. First, earlier work in our lab and other labs showed that consumption behavior is sensitive to manipulation of affective state (Laeng et al., 1993; Strahan et al., 2003; Winkielman et al., 2005). Second, it is important to explore the impact of affective priming on a relatively consequential behavior (a voluntary decision to ingest a novel beverage of uncertain taste and composition. Furthermore, drinking allows one to assess the impact of affective priming with "real-world" units such as volume and price. Finally, as in earlier studies, we also measured changes in subjective experience—mood and arousal, but because our previous studies found no effects on these measures, we predicted no differences here. We return to this issue later.

### **EXPERIMENT 1**

This study compared the impact of subliminal and supraliminal facial expressions and valence-matched words on consumption behavior and measures of subjective experience. We expected that facial expressions would have a stronger impact than words regardless of presentation duration.

# **Method**

#### **Participants and Procedure**

The study was approved by UCSD Human Research Protection Program. Fifty undergraduates (19 males, 31 females, mean age = 21.1 years) participated for extra credit. Five additional participants were run, but they were replaced because they reported on a post-experimental questionnaire that they saw the subliminal primes (see below). Participants were told that the study involved a computer task, a tasting task, and a rating task, all performed in individual rooms outside the view of the experimenter. The sequence of experimental events, explained in detail shortly, was as follows. First, participants completed a consent form and a pre-task questionnaire. Second, the participants performed four iterations of a classification task on a computer and the beverage tasting and rating task. Finally, participants filled out a post-experimental survey and were thanked and debriefed.

#### **Pre-task Questionnaire**

The pre-task questionnaire assessed participants' baseline affective and motivational state with the following questions: "How hungry are you at the moment?" (0 = not at all, 10 = extremely), "How thirsty are you at the moment?" (0 = not at all, 10 = extremely), "How many hours has passed since your last meal?" (0.5–10 h), "How large was your meal?" (1 = snack, 3 = large), "How much of a sweet tooth do you have?" (0 = not at all, 10 = very much), "How do you feel right now, at this very moment?" (−5 = unpleasant, 5 = pleasant), "How much arousal do you feel right now, at this very moment?" (−5 = low, 5 = high).

#### **Classification (Priming) Task**

Participants were exposed to affective primes under the disguise of a categorization task. We told participants that they would see different stimuli quickly presented in succession, and should ignore all but the last stimulus. If the stimulus was a face, participants judged whether it was male or female, and if it was a word, whether it was an animal or an object.

The structure of a trial was as follows (**Figure 1**, top panel). Each trial started with a small 700-ms fixation cross, followed by 200-ms large cross that served as a pre-mask. The prime was then presented for either 10 ms (subliminal) or 200 ms (supraliminal). Finally, the prime was followed by a 1,000-ms neutral post-mask, which participants were to classify as quickly as possible. After one block of eight trials, participants received the consumption and rating task. In total, there were four blocks of valenced primes, counterbalanced across subjects: (i) positive words, (ii) negative words, (iii) positive faces, and (iv) negative faces. The duration of the prime was manipulated between subjects.

#### **Primes and Mask Stimuli**

Face stimuli came from the JACFEE series (Matsumoto and Ekman, 1988). Affective primes were 8 negative and 8 positive faces and masks were 16 neutral faces. Half of the faces were Caucasian and half Asian, half male and half female. All faces were shown in black and white and their size was approximately 15 cm2. Word stimuli came from the ANEW set (Bradley and Lang, 1999) and from a set developed by Storbeck and Robinson

(2004). Affective primes were 8 positive and 8 negative words and masks were 16 neutral words. Half the words were of animals and half were of objects. All word stimuli were in black font against a white background with a size of approximately 8–15 cm wide and 2.5 cm high. Images were shown on a 48-cm monitor, approximately 50 cm away from the participant. The monitor and graphics card were capable of supporting 100 Hz refresh rate which allowed for presentation of 10 ms stimuli (as limited by precision of E-Prime software).

All facial and word stimuli were matched on valence (within 0.5 SD) based on a pre-test in which 16 participants rated a larger set of stimuli on a 1–9 scale (1 = extremely negative, 9 = extremely positive). The exact stimuli and mean ratings are reported in Appendix 1 (left panel).

#### **Beverage Task**

After each round of the classification task, participants were asked to pour and drink the beverage, and to rate their mood and arousal, with task order counterbalanced across subjects. Finally, they were asked to rate the beverage using four questions: "How delicious is the drink?"(0 = not at all to 10 = extremely delicious), "How much of this drink would you like to drink right now?"(0 = none to 6 = pints), "How much would you pay for the drink?"(1 = \$0.10 to 10 = \$1.00), and "How sweet is this drink?" (0 = not at all to 10 = very much).

As in previous studies, the beverage was made out of Kool-Aid lemon-lime flavored powder, water, and sugar (see Winkielman et al., 2005 for details). To ensure that the beverage was unfamiliar for each condition, we varied sugar and powder proportions. We used four 2-liter pitchers (labeled A, B, C, and D) that contained approximately 600 ml of the beverage at the beginning of the study. Participants poured the drink into 250 ml cups (labeled A, B, C, and D). After each participant left, we used an electronic scale to weigh the amount of beverage left in the pitcher and in the cup, which allowed us to determine how many grams of liquid the participant poured and drank.

#### **Feelings Rating Task**

The feelings rating task included the following questions: "How do you feel right now, at this very moment?" (−5 = unpleasant to 5 = pleasant), "How much arousal do you feel right now, at this very moment?"(−5 = low, 5 = high). The feelings rating task also included a supplemental measure that asked participants to rate 22 emotions currently felt (1 = not at all to 5 = extremely) using a PANAS scale supplemented by 2 items (happy and angry). These data are not reported for Experiment 1 because they were unfortunately lost due to experimenter error. However, in previous studies using this supplemental measure, and in Experiment 2, we found no significant effects.

#### **Post-experimental Survey**

The final questionnaire tested participants' impression of the experiment and tested for any suspicions about the experiment priming or consumption tasks. Participants were asked about any unusual aspects of the procedure, any briefly flashed pictures, the drinks, and changes in their mood.

#### **Results**

We analyzed how valence of the prime (positive vs. negative), format of prime stimulus (faces vs. words) and prime duration (subliminal vs. supraliminal) influenced measures of (i) consumption behavior, (ii) subjective experience, and (iii) drink ratings. The relevant means are presented in **Table 1**. Data underlying these analyses are available under the following link: http://pages.ucsd.edu/~pwinkiel/koolaid-picture-words.zip.

#### **Consumption Behaviors: Pouring and Drinking**

We tested the influence of priming on two different types of consumption behavior: pouring and drinking. These two DVs were analyzed with a four-way mixed MANOVA with three within-subject factors: behavior type (pouring and drinking), prime valence (positive and negative), prime format (faces and words), and a between-subject factor of prime duration2. This MANOVA found no effect of prime duration and a theoretically uninteresting main effect for behavior type, showing that participants poured more than they drank, *F*(1, 48) = 41.30, *p* < 0.001. More interesting, there was a valence by stimulus format interaction, *F*(1, 48) = 4.56, *p* = 0.038. To understand this interaction we ran MANOVAS testing separately the impact of pictures and the impact of words (with all other factors included). The MANOVA on pictures revealed that consumption behaviors (pouring and drinking) were greater after positive than negative faces, *F*(1, 48) = 5.25, *p* =0.026. In contrast, the MANOVA on words revealed that the impact of valence was not significant (*F* = 1.04), with the direction of means suggesting a decrease of consumption behavior after positive words.

The above analysis did not reveal a significant three-way interactions involving behavior type (pouring vs. drinking). However, as discussed shortly, these two measures were differently impacted by thirst. Furthermore, we wanted to probe the nature of the priming effect on each individual measure of consumption behavior. We also dropped duration from these additional analyses, as we found no effects earlier (and in later analyses). The results are illustrated in **Figure 2** (left panel). On pouring, the 2-way interaction of valence and stimulus format was marginal, *F*(1, 49) = 3.06, *p* = 0.087. Simple effects analysis revealed that participants tended to pour more after positive than negative faces, *t*(49) = 1.7, *p* = 0.095, with no significant difference after positive than negative words (*t* > 1). On drinking, the valence by stimulus format interaction was also marginal, *F*(1, 49) = 3.91, *p* = 0.054. Simple effects analysis revealed that participants drank more after positive than negative faces, [*t*(49) = 2.07, *p* = 0.043], and tended to drink less after priming with positive than negative words, *t*(49) = 1.07, *p* = 0.29.

#### *Effect of thirst*

Our previous work showed that priming with emotional faces had the strongest effect on thirsty participants (Winkielman et al., 2005). Therefore, we conducted an analysis with 5 factors. Three within factors: behavior type (pouring and drinking), prime valence (positive vs. negative), and stimulus format (picture format). Two between factors: duration (subliminal, supraliminal), and thirst (high vs. low, determined by a median split). This analysis again revealed a main effect of behavior type, *F*(1, 46) = 39.64 (pouring higher than drinking), a 2-way interaction of valence and format, *F*(1, 46) = 3.78, *p* = 0.058, a 3 way interaction of valence, format and thirst, *F*(1, 46) = 6.75, *p* = 0.013, and a 4-way interaction of behavior type, valence, format, and thirst, *F*(1, 46) = 7.285, *p* = 0.01. Duration did not enter in any effects so it was dropped from subsequent analyses designed to further understand the patterns. To make things simpler, and because behavior type interacted with other factors, additional analyses were conducted separately on drinking and on pouring.

On drinking, there was a 3-way interaction of valence, stimulus format, and thirst, *F*(1, 48) = 14, *p* < 0.01. We

<sup>2</sup>Preliminary analyses revealed no effects of the blocks order, so main analyses collapsed across this variable.

**TABLE 1 |** Means and standard errors of critical dependent measures in Experiment 1.


Consumption behavior was measured in grams. Experience on −5 to + 5 scale (not at all, very much). Ratings of deliciousness and sweetness on 0–10 scale. Wanting on 0–6 scale reflecting desired amount, and paying on 10 cents to 1 dollar scale.

decomposed this interaction next by separate analyses of valence and thirst on different prime formats. Using only faces, the analysis revealed a 2-way interaction of valence and thirst, *F*(1, 48) = 7.69, *p* < 0.01. Thirsty participants drank more after positive faces than negative faces, [*t*(27) = 3.00, *p* < 0.01]. For nonthirsty participants, facial valence did not significantly influence drinking (*t* < 1). Using only words, the analysis yielded a 2 way interaction of valence and thirst, *F*(1, 48) = 7.60, *p* < 0.01. Interestingly, the simple effects were in opposite direction to the effects of faces. Thirsty participants drank less after positive words than negative words, *t*(27) = −2.37, *p* < 0.05. This contrast effect was unexpected and we will return to it in the discussion. For non-thirsty participants, word primes did not influence drinking, *t* < 1.7.

Because of possible distortions inherent in median-split analyses of continuous variables, we also correlated participants' level of thirst with a change score reflecting the impact of the prime on their drinking behavior (the within-subject difference in drinking after positive vs. negative primes). Consistent with the median-split analyses, greater thirst predicted greater congruent impact of facial primes, *r*(50) = 0.36, *p* < 0.05, and greater incongruent impact of word primes, *r*(50) = −0.30, *p* < 0.05.

Finally, no significant main effects or interactions with thirst were obtained on pouring. Continuous measure of thirst also did not predict the impact of facial or word primes on pouring.

#### **Ratings of Subjective Experience and Ratings of Drinks**

Priming effects on subjective experience and ratings of drinks were analyzed with a three-way ANOVA with the within-subject factors of prime valence, prime format, and duration. On the mood rating, there were no significant effects. On arousal, there was an unexpected main effect for the stimulus type, with participants exposed to words reporting higher arousal ratings, *F*(1, 48) = 7.92, *p* = 0.01. For ratings of drinks, there were no significant effects, as in earlier research involving similar "pouring and drinking" procedure (Winkielman et al., 2005, Study 1).

Finally, as above, we redid the same analyses adding thirst. There were no significant thirst effects on ratings of mood and drinks, and no significant interaction between valence of the prime and thirst (all *F*s < 1).

#### **Discussion**

Experiment 1 obtained results that are reasonably consistent with our predictions and earlier work. Facial expressions influenced consumption behavior in a valence-congruent manner, with participants drinking more after positive (happy) than negative (angry) faces. As in earlier work, the influence of facial expressions was amplified by thirst. As previously, we found no significant effect of facial expressions on subjective experience, though obviously proving the null effect of faces on subjective experience would require evidence for H0 (Bayes factor favoring the absence of an effect) ideally with much larger samples. So, for now the conclusion is very tentative.

Surprisingly, we did not find significant main effects or significant interactions involving stimulus duration—whether the primes were subliminal (near threshold) or supraliminal (visible, though unobtrusive). Though unexpected, this echoes some reports of comparable behavioral effects for subliminal and visible but unobtrusive primes (Bargh, 1992) and some neuroimagining findings of comparable activation effects for subliminal and unobtrusive presentations of emotional facial expressions (Critchley et al., 2000). But again, we would need stronger evidence to confidently claim the absence of the duration effect.

Interestingly, in comparison with faces, words tended to have a valence-incongruent influence. There was one significant contrast effect on drinking for thirsty participants, but other contrast effects were only at the level of a tendency. We will return to this observation in the discussion, though obviously this fragile pattern needs to be replicated.

Finally, on ratings of subjective experience, participants reported greater arousal after words than faces. However, this non-intuitive finding should be interpreted with caution as we only balanced the words and faces on pretest ratings of valence, but not on arousal. We address this issue in Experiment 2 by using a differently standardized stimulus sets.

## **EXPERIMENT 2**

Experiment 1 left several questions unanswered. One issue is whether consumption behavior can be influenced by affective pictures other than faces. This is important as faces are unique in several respects. For one, faces are extremely frequent stimuli for which people develop processing expertise (Farah et al., 1998). Further, facial expressions might influence affective responses via low-level mechanisms tuned to detection of rudimentary features (Morris et al., 1998; Vuilleumier et al., 2003). Finally, facial expressions, especially smiles and frowns, are frequently used gestures of approval or disapproval, including in the domain of consumption (Klinnert et al., 1983). To address the issue of possible specificity for facial expressions, in Experiment 2 we selected pictures from the standardized IAPS set that included emotional objects or scenes that should not receive privileged processing (e.g., dog, gun, astronaut, and dentist). We then selected words from the standardized ANEW that matched the object in the pictures and were similar in valence and arousal.

To further address the possible role of the amount of experience with the stimulus, we also manipulated frequency of words and matched pictures. Some studies suggest that priming effects are more easily obtained with high frequency words, due to their greater initial activation (McClelland and Rumelhart, 1981). Similarly, previous studies that obtained priming effects with pictures tended to use familiar, high frequency objects (e.g., Dell'Acqua and Grainger, 1999).

In addition, because in Experiment 1 we observed no effects of presentation duration, all primes in Experiment 2 were presented supraliminally, but unobtrusively. Making both picture and word primes fully visible also reduces the possibility that any differences in their influence have to do with the relative salience and complexity of the word and picture stimuli.

Finally, one potential methodological concern about Experiment 1 is that the face classification task (male/female) was different from the word classification task (object/animal). It is possible that word classification along the object-animal dimensions is more difficult than face classification along the gender dimension. This could result in more distraction from the affective meaning of the word stimuli relative to the face stimuli, or deeper, more careful processing of word stimuli. To address this concern, in Experiment 2 participants judged all stimuli on the same dimension—whether they contained a living or a non-living object.

#### **Method**

#### **Participants and Procedure**

Forty-eight undergraduates (12 males, 36 females, mean age = 20.7 years) participated for extra credit. The cover story, the equipment, the order of events, and the general procedure were similar to Experiment 1, except for the classification task. As shown in **Figure 1** (bottom panel), the affective prime was shown for 200 ms, which participants classified on the living/non-living dimension. Because the primes were always supraliminal, we eliminated the pre-mask cross as well as the neutral post-mask. Eight priming trials were arranged in four within-subjects counterbalanced blocks that were: (i) positive and high frequency, (ii) positive and low frequency, (iii) negative and high frequency, and (iv) negative and low frequency. Format of prime stimulus (words or pictures) was manipulated between subjects, with half the subjects in the words condition and half in the picture condition.

#### *Word stimuli*

Word stimuli as well as their valence, arousal, and frequency ratings came from the ANEW set by Bradley and Lang (1999) and are listed in the Appendix 1 (right panel). Half of the words represented living things and half represented non-living things. Words ranged in width from 8 to 15 cm and were all 2.5 cm high.

#### *Picture stimuli*

Picture stimuli along with their valence and arousal ratings came from IAPS set by Lang et al. (2005) and are listed in the Appendix 1 (right panel). Half of the pictures represented living things and half represented non-living things. Picture stimuli were presented in a full screen view on a 21-inch (48-cm) computer monitor.

# **Results**

We analyzed how prime valence (positive vs. negative), prime format (faces vs. words), and prime frequency (low vs. high) influenced consumption behavior, subjective experience, and drink ratings. Data underlying these analyses are available under the following link: http://pages.ucsd.edu/~pwinkiel/koolaidpicture-words.zip. Means are presented in **Table 2**.

#### **Consumption Behavior: Pouring and Drinking**

Priming effects on consumption behaviors (pouring and drinking) were analyzed with a four-way mixed MANOVA with three within-subject factors—type of behavior (pouring vs. drinking), prime valence (positive vs. negative), and prime frequency (high vs. low), as well as the between-subjects factor of prime format (pictures vs. words). There was a theoretically uninteresting main effect for behavior type, showing that participants poured more than they drank, *F*(1, 46) = 31.26, *p* < 0.001. More interestingly, there was a prime format by valence interaction, *F*(1, 46) = 6.87, *p* < 0.05. This reflects that positive vs. negative stimuli had different impact on consumption behaviors as a function of the type of prime.

Though we did not observe an interaction with the behavior type in this analysis (but see below), we wanted to further probe the nature of these effects on the individual measures of pouring and drinking. The results are illustrated in **Figure 2** (right panel). On pouring, there was a valence by stimulus format interaction, *F*(1, 46) = 5.82, *p* < 0.05. Participants tended to pour more after priming with positive pictures than with negative pictures (*p* = 0.13), but *less* after priming with positive words than with negative words (*p* < 0.07). There was also a valence by stimulus format interaction on drinking, *F*(1, 46) = 6.65, *p* < 0.05. Participants drank more after priming with positive pictures than with negative pictures (*p* < 0.05). They also tended to drink *less* after priming with positive words than with negative words, but this effect was far from significance (*p* = 0.21).

Finally, like in Experiment 1, we examined possible thirst effects by dividing participants into high-thirst and low-thirst groups (via median split) and adding this factor to the analysis. On pouring and on drinking, the effects of valence did not vary by the level of thirst. For completeness, we also conducted a full, five-way MANOVA with behavior type, prime valence, frequency, prime format, and thirst level. This analysis revealed a complex 4-way interaction, *F*(1, 44) = 4.12, *p* = 0.048. However, diagnosing this interaction revealed that it was primarily driven by different impact of stimulus frequency across different levels of other factors, so we will not decompose it here as it is not theoretically relevant. Likewise, the analysis with thirst as a continuous measure yielded no significant correlation between participants' thirst and the impact of primes on pouring or drinking.

#### **Ratings of Subjective Experience and Ratings of Drinks**

For ratings of subjective experience and ratings of drinks, no significant effects were revealed by a three-way MANOVA of prime valence, prime stimulus format, and frequency. There was a marginal 2-way interaction between valence and stimulus format on deliciousness, *F*(1, 46) = 3.98, *p* < 0.06, but no simple effects were significant and this interaction was not obtained on other ratings.

#### **Discussion**

The main results of Experiment 2 were similar to Experiment 1. Priming with emotional pictures influenced consumption behavior in a valence-congruent fashion, with increased consumption after priming with positive than negative pictures. In comparison, priming with valence-matched words yielded trends in the opposite, valence-incongruent direction, with *decreased* consumption after positive than negative words. Similar to Experiment 1, priming had no significant effect on either subjective experience (mood and arousal) or on drink ratings. Interestingly, the frequency of the stimuli did not significantly modify the effects. Finally, unlike Experiment 1, there were no significant effects of thirst, even on priming with emotional pictures.

## **GENERAL DISCUSSION**

The main finding of the current experiments is that unobtrusive exposure to emotional facial expressions (Experiment 1) and emotional pictures (Experiment 2) influenced participants' consumption behavior in a valence-congruent way. In comparison, the exposure to emotional words tended to lead to the opposite, valence-incongruent contrast effects. However, only one of those contrast effects was significant and required that participants were thirsty (Experiment 1). The divergent patterns of influence were obtained even though words were matched on valence with pictures of emotional facial expressions (Experiment 1), and on valence and arousal with pictures of emotional objects (Experiment 2). Before we discuss the implications of this main finding, let us summarize and discuss some secondary findings.

**TABLE 2 |** Means and standard errors of critical dependent measures in Experiment 2.


Consumption behavior was measured in grams. Experience on −5 to + 5 scale (not at all, very much). Ratings of deliciousness and sweetness on 0–10 scale. Wanting on 0–6 scale reflecting desired amount, and paying on 10 cents to 1 dollar scale.

Several potentially interesting effects were statistically nonsignificant. Without treating these non-significant results as evidential, we point out their potential relevance. In Experiment 1 the effect of presentation duration (subliminal vs. supraliminal, but unobtrusive) was non-significant. The non-significant effect of duration is consistent with a range of effects in this literature (including quite small ones) suggesting that priming direction is determined not by awareness of the prime *per se*, but by awareness of the prime influence (Bargh, 1992). It also matches some neuroscience data which found little difference between subliminal and supraliminal, but unobtrusive presentations of facial expressions on limbic activation (Whalen et al., 1998; Critchley et al., 2000). The non-significant effect of presentation duration reduces the concern that any differences between pictures and words were due to sheer perceptibility. In Experiment 2, the stimulus frequency effect was also statistically non-significant. This non-significant effect fits the notion that the relative amount of participants' experience with words and pictures stimuli is not the reason for their different influence.

In both experiments, we found no significant effects of stimulus valence on subjective experience, even for participants who showed reliable valence effects on the consumption behavior. This finding fits earlier reports that affective stimuli, can influence behavior without eliciting changes in conscious experience (Winkielman et al., 1997, 2005; Zemack-Rugar et al., 2007; Bornemann et al., 2012). Theoretically, this result fits the idea of "unconscious emotion" (Winkielman et al., 2011; Smith and Lane, 2016). However, this claim requires much further testing because phenomenal changes could be revealed with more sensitive measures of affective experience (Schooler et al., 2015), under conditions that promote the emergence of state awareness (Morsella, 2005), or in participants attuned to their bodily state (Bornemann et al., 2012). Clearly, it is also possible that accumulating data will reveal support for this null hypothesis (i.e., Bayes factor favoring the null). At this point, any conclusion about "unconscious emotion" in this paradigm is tentative.

Finally, consistent with earlier research, participants' motivational state—thirst—modified the impact of facial expressions on consumption (Winkielman et al., 2005). However, we found no statistically significant effect of thirst on the impact of emotional IAPS pictures containing objects and scenes. This non-significant finding was unexpected and could simply reflect differences in baseline thirst levels or drinking levels across studies, or overall levels of task motivation (Gendolla, 2000). Unfortunately, in the current research we did not compare facial expressions and IAPS in the same experiment. But, if such findings were confirmed by future studies, they could be interpreted in light of arguments that facial expressions have an advantage in modulating low-level affective and motivational mechanisms (Whalen et al., 1998; Vuilleumier et al., 2003). Consistently, faces are good triggers of "approach-avoidance" tendencies (Klinnert et al., 1983; Marsh et al., 2005; but see Phaf et al., 2014). Future research may explore this possibility.

#### **Pictures and Words**

Our main finding was the valence-congruent influence of affective pictures, but not words, on spontaneous behavior. Two theoretical accounts may provide explanation for this finding.

One interpretation is offered by the framework of grounded or embodied cognition (Barsalou, 1999). As discussed in the introduction, this framework assumes that processing of affective stimuli can be purely conceptual, or can be accompanied by somatosensory reactions (Niedenthal, 2007; Winkielman et al., 2015b). If an incidental stimulus elicits a somatosensory reaction, and the mechanisms underlying a particular behavior draw on the same set of somatosensory resources, one should observe a congruent influence (Winkielman et al., 2007; Knutson et al., 2008). This interpretation is consistent with existing literature, discussed earlier, showing that viewing pictures of emotional faces or emotional scenes leads to greater physiological effects than viewing emotion words (Harris et al., 2003; Larsen et al., 2003).

Why do emotional pictures exert this influence but not words? One possibility is that the evolutionarily "old" affect system is particularly suited for dealing with pictorial information. A picture of a smiley face, a juicy cake, a slithering snake, or a bloody knife may be quickly interpreted by early visual areas connected to subcortical structures involved in generation of physiological responses (de Gelder et al., 2011). In comparison, words are polysemous—even simple nouns like "cake" or "gun" can appear in contexts that flip its affective meaning (cake=dung, guns=biceps). So, before a word is able to elicit a robust physiological response, it needs to be processed more deeply by more advanced mechanisms, and perhaps even "translated" into a specific sensory representation (Paivio, 1971; Vandenberghe et al., 1996; Naccache et al., 2005; Niedenthal et al., 2009). Furthermore, without the physiological response to guide subsequent processing in a valence-congruent fashion, words might be subject to more "cold" cognitive operations.

A different theoretical perspective suggests that the power of affective pictures comes not from their ability to trigger physiological responses, but from their privileged access to the cognitive semantic system (Potter et al., 1986; Glaser, 1992), including the semantic network representing evaluative information (De Houwer and Hermans, 1994). Consistent with this assumption, some studies found stronger impact of pictures and words in a fast evaluative priming task (Spruyt et al., 2002, but see Kiefer et al., 2017). However, this "privileged semantic access", unlike the "embodiment" explanation, does not predict greater impact of pictures on consumption behavior a finding we observed. It also predicts greater impact of primes on evaluation of drink qualities and evaluation of one's mood—findings we failed to observe. Further, the privileged semantic access account would predict greater impact of supraliminal, than subliminal words, and high-frequency than low-frequency words—again, findings that we failed to observe. On the other hand, this verdict against this explanation needs to be tentative until future studies directly examine the relative availability of semantic concepts after picture and word priming and the relative sensitivity of behavioral and self-report measures.

More generally, the privileged semantic access explanation, may on the first glance seem contradictory to the embodied explanation. However, it can be seen as compatible. After all, theories of embodied cognition propose that semantic processing is supported by somatosensory simulation involving "perception-like" processes. In fact, categorization literature suggests that when participants try to access a deep meaning of an emotional concept, they generate concrete images and develop a specific "situated" conceptualization (Barsalou, 1999; Niedenthal et al., 2005; Oosterwijk et al., 2015). However, that simulation process takes motivation, effort, and time (Wilson-Mendenhall et al., 2011). From that perspective, pictures offer "pre-packaged" situated conceptualizations—an idea which explains their privileged access and greater spontaneous impact on physiology.

One weakness of both the embodiment and the "privileged access" explanations is that they both predict that the impact of words should simply be weaker primes than pictures. However, even though most of the individual effects were not significant, a general tendency was for words to have an opposite effect. One speculative explanation is that because our words were concrete nouns, referring to specific exemplars rather than general traits, they might have triggered comparison contrast operations, thus creating a response tendency opposite to the prime implications (Dijksterhuis et al., 1998). On the other hand, emotional pictures also represent concrete exemplars, albeit their exemplar status might be qualified by a broad affective activation.

Finally, it is important to acknowledge that words can be powerful triggers of affect and actions. Occasionally, a word may be worth a thousand pictures. Humans are a symbolic species and much of their affairs, including those of the heart, take root and express in symbolically-mediated interactions (e.g., via political manifestos, love letters, or arguments presented in journal articles). Words can do that because they "invite and guide" us to develop mental models that real images may not match (Barsalou, 1999; Bergen, 2012). As a result, words can trigger powerful emotional reactions, with all their physiological consequences. Some of them require a deep understanding of the implications of what has been said ("when you left house today, you forgot to turn off the hot iron"). Other words, such as taboo words, terms of endearment, or native-language emotional terms can work directly through previously pre-computed meanings (Harris et al., 2003; Baumeister et al., 2017). Future research may focus on specifying different conditions under which words and pictures are influential as affective primes of judgment and action, and explore more precisely the underlying psychological and physiological mechanisms. In any case, we hope that the words and pictures presented in this article offer an initial contribution toward this goal, and help us move toward better integration of theories of cognition and emotion.

### **ETHICS STATEMENT**

This study was carried out in accordance with the recommendations of UCSD Human Research Protection Program with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the HRPP human subject committee.

## **AUTHOR CONTRIBUTIONS**

YG collected, entered, did preliminary analyses of the data and described the procedures and methods. PW conceptualized the study and wrote the introduction, results, and discussion sections.

# **REFERENCES**


# **FUNDING**

PW was supported by UCSD Academic Senate Grant and by SWPS BST research fund.

### **ACKNOWLEDGMENTS**

Data were collected by the second author and parts were reported in her UCSD Honor's Thesis (unpublished). This work is supported by Academic Senate Grant from UCSD. Data and materials from this research are available and hosted on UCSD server under the following link: http://pages.ucsd. edu/~pwinkiel/koolaid-picture-words.zip. The second author was unreachable during the preparation of the final version of this article.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2018 Winkielman and Gogolushko. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# **APPENDIX**


# Interplay between Narrative and Bodily Self in Access to Consciousness: No Difference between Self- and Non-self Attributes

Jean-Paul Noel1,2,3, Olaf Blanke1,2,4, Andrea Serino1,2,5 and Roy Salomon1,2,6 \*

<sup>1</sup> Laboratory of Cognitive Neuroscience, Faculty of Life Science, Brain Mind Institute, Ecole Polytechnique Federale de Lausanne, Lausanne, Switzerland, <sup>2</sup> Center for Neuroprosthetics, Ecole Polytechnique Federale de Lausanne, Lausanne, Switzerland, <sup>3</sup> Vanderbilt Brain Institute, Vanderbilt University, Nashville, TN, USA, <sup>4</sup> Department of Neurology, University Hospital, Geneva, Switzerland, <sup>5</sup> Department of Psychology, Alma Mater Studiorum – Università di Bologna, Bologna, Italy, <sup>6</sup> Gonda Multidisciplinary Brain Research Center, Bar-Ilan University, Ramat Gan, Israel

#### Edited by:

Guido Hesselmann, Charité, Germany

#### Reviewed by:

Timo Stein, University of Trento, Italy Bianca Van Kemenade, University of Marburg, Germany

> \*Correspondence: Roy Salomon Roy.salomon@epfl.ch

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 30 October 2016 Accepted: 12 January 2017 Published: 31 January 2017

#### Citation:

Noel J-P, Blanke O, Serino A and Salomon R (2017) Interplay between Narrative and Bodily Self in Access to Consciousness: No Difference between Self- and Non-self Attributes. Front. Psychol. 8:72. doi: 10.3389/fpsyg.2017.00072 The construct of the "self" is conceived as being fundamental in promoting survival. As such, extensive studies have documented preferential processing of self-relevant stimuli. For example, attributes that relate to the self are better encoded and retrieved, and are more readily consciously perceived. The preferential processing of self-relevant information, however, appears to be especially true for physical (e.g., faces), as opposed to psychological (e.g., traits), conceptions of the self. Here, we test whether semantic attributes that participants judge as self-relevant are further processed unconsciously than attributes that were not judged as self-relevant. In Experiment 1, a continuous flash suppression paradigm was employed with "self" and "non-self" attribute words being presented subliminally, and we asked participants to categorize unseen words as either self-related or not. In a second experiment, we attempted to boost putative preferential self-processing by relation to its physical conception, that is, one's own body. To this aim, we repeated Experiment 1 while administrating acoustic stimuli either close or far from the body, i.e., within or outside peripersonal space. Results of both Experiment 1 and 2 demonstrate no difference in breaking suppression for self and non-self words. Additionally, we found that while participants were able to process the physical location of the unseen words (above or below fixation) they were not able to categorize these as self-relevant or not. Finally, results showed that sounds presented in the extra-personal space elicited a more stringent response criterion for "self" in the process of categorizing unseen visual stimuli. This shift in criterion as a consequence of sound location was restricted to the self, as no such effect was observed in the categorization of attributes occurring above or below fixation. Overall, our findings seem to indicate that subliminally presented stimuli are not semantically processed, at least inasmuch as to be categorized as self-relevant or not. However, we do demonstrate that the distance at which acoustic stimuli are presented may alter the balance between self- and non-self biases.

Keywords: self, criterion bias, peripersonal space, multisensory, audio, consciousness

# INTRODUCTION

fpsyg-08-00072 January 27, 2017 Time: 15:14 # 2

The inherent complexity of the construct of the self, has led to distinct conceptualizations typically including several levels of "selfhood" (James, 1890; Damasio, 1996, 1999; Parnas, 2003; Northoff and Bermpohl, 2004; Zahavi, 2005). At a foundational level of the self is a pre-semantic and nonconceptual understanding of the bodily self, which may serve as a scaffold for more complex and abstracts understandings of the self (Damasio, 1996, 1999; Gallagher, 2005; Legrand, 2006, 2007; Blanke and Metzinger, 2009). The pre-reflective level of self-representation has been suggested to be derived from multisensory-motor correspondences (Blanke, 2012; Blanke et al., 2015) and is thought to be developed at an early age (Rochat, 1995, 2003; Filippetti et al., 2013). It is only at later stages of development that higher-level semantic representations of the self – such as the narrative or autobiographical self (Northoff and Bermpohl, 2004; Northoff et al., 2006) – representing reflective conceptualizations of the self across time and space are established (Rochat, 1995, 2003; Damasio, 1996, 1999; Baumeister, 1998). Thus, a core self-representation is suggested to be grounded in the bodily self, while semantic and psychological representations of the self are developed later, putatively scaffolded upon the bodily-self representation, and involving memory and cognitive constructs of oneself across time and space (Arzy et al., 2008; Peer et al., 2015).

Both the bodily and the narrative conceptions of the self have been described as a fundamental construct bearing on human survival (Keenan et al., 2000; Humphrey, 2006) and correspondingly, the processing of self-relevant information has been shown to enjoy a privileged status (Yu and Blake, 1992; Tong and Nakayama, 1999; Sugiura et al., 2000; Salomon et al., 2011, 2013). At the sensory level, an example of this privileged selfprocessing can be observed in the so-called "cocktail party effect" (Cherry, 1953; Wood and Cowan, 1995) in which one's name is automatically picked up by the auditory system among a sea of sensory noise (see Schäfer et al., 2016, for a recent demonstration of self-prioritization effects across various sensory modalities). A similar "self-reference effect" (Rogers et al., 1977) has been described for mnemonic processes in which information that is encoded as being self-relevant is a posteriori better retained (Sui and Humphreys, 2015). Further, it has recently been shown that even an arbitrary and externally imposed relationship between stimuli and the self causes enhanced processing of these stimuli (Sui et al., 2012, 2013; Stein et al., 2016). Thus, seemingly both at the sensory and at higher-order cognitive levels, selfrelated stimuli may enjoy of enhanced or prioritized access to awareness.

The bifurcation between the study of the bodily-self and a narrative-self has nonetheless arguably led to neglecting possible interactions between the sensory signals which are at the basis of the bodily self and higher-order level processes relating to the narrative self (see Canzoneri et al., 2016 for a recent exception).

Regarding the bodily-self, recent studies have shown that coupling masked visual stimuli with related bodily signals affects the emergence of the visual stimuli into awareness. Such effects have been shown for proprioceptive (Salomon et al., 2013), tactile (Lunghi et al., 2010; Lunghi and Alais, 2013; Salomon et al., 2015a), vestibular (Salomon et al., 2015b), and interoceptive signals (Salomon et al., 2016b). Our group, for example, has demonstrated that pairing unseen visual stimuli with a congruent body posture causes the unseen visual stimuli to break continuous flash suppression (CFS; Tsuchiya and Koch, 2005; Jiang et al., 2007; Stein et al., 2011) faster than when the stimuli is paired with incongruent proprioceptive signals (Salomon et al., 2013). That is, when participants are presented with highly visible and salient stimuli to their dominant eye (e.g., "Mondrians"), and visual stimuli with a lesser contrast or salience to their non-dominant eye (e.g., CFS paradigm), this latter stimulus will break the Mondrian suppression and thus access consciousness more readily when paired with a congruent (vs. incongruent) body posture. Further, the co-localization of sound and visual stimuli has equally been demonstrated to thrust this latter one into awareness under a breaking CFS paradigm (Aller et al., 2015). Similarly, using a related visual dichoptic presentation paradigm, namely, binocular rivalry (see Alais and Blake, 2005, for review), Lunghi et al. have demonstrated striking interactions between the administration of touch and/or audiotactile stimulation and congruently presented visual stimuli in propelling the latter one into awareness (Lunghi and Alais, 2013; Lunghi et al., 2014). Further, sensory signals presented not only at the body, but also near the body, in the peripersonal space (PPS; Serino et al., 2015), have been demonstrated to heavily influence self-processing (Blanke et al., 2015; Noel et al., 2015a, 2016; Salomon et al., 2016a), and to reference conceptual processing (Canzoneri et al., 2016). Thus, sensory signals from the body or within the PPS may affect the processing of visual and self-related stimuli (see Faivre et al., 2015 for review).

On the other hand, in relation to the preferential processing of the semantic self, the scope and limits of unconscious semantic processing have revealed contradictory results – arguably in great measure as a consequence of the utilization of different experimental paradigms (Hesselmann and Moors, 2015), but also due to distinct levels of association between physical and psychological conceptions of the self. While researchers employing unconscious priming (which may arguably primarily index non-conscious processing, as opposed to access to consciousness, as CFS does; Stein et al., 2011) have demonstrated powerful effects of unconscious social primes influencing posterior behavior (Bargh et al., 1996; Dijksterhuis and Van Knippenberg, 2000), these subliminal effects are not always present – in particular when scrutinizing the possibility of observing subliminal semantic processing (Zimba and Blake, 1983; Blake, 1988; Kang et al., 2011). In fact, researchers have argued that the default stance should be not to expect much (self or non-self) semantic unconscious processing during incongruent dichoptic presentation (Hesselmann and Moors, 2015), and Blake (1988) inclusively reported that even the first name of a specific observer, which represents a fundamental constituent of the semantic self (Joubert, 1991) proved insufficient to alter dominance of one percept (self) over another (non-self) under a condition of dichoptic stimulation. Tacikowski and Ehrsson

(2016) have recently reported a similar effect, in that even one's own name, when masked, was insufficient to act as a self-prime.

Contrarily, recent studies utilizing a breaking CFS paradigm have demonstrated a host of effects presumably relying on unconscious semantic processing. Sklar et al. (2012), for instance, have shown that semantically incoherent expressions broke suppression (that is, "emerged" into visual awareness) more readily than semantically coherent expressions. Similarly, these researchers showed that increasing the negative affect of expressions lowered suppression time significantly, and that effortful arithmetic equations can be solved without awareness (Sklar et al., 2012). Yet other groups have demonstrated that emotional information is processed during suppression (Yang et al., 2007), scene congruency information can be extracted in the absence of visual awareness (Mudrik et al., 2011), and that word meaning (Costello et al., 2009) and word valence (Yang and Yeh, 2011) can be processed unconsciously. Further, particularly with regard access to consciousness of self-relevant information under the context of breaking CFS, Geng et al. (2012) have demonstrated that one's own face is detected more rapidly than other faces, while Stein et al. (2016), did not show the analogous effect when a visual stimulus (gabor) was arbitrarily labeled "You" as opposed to "Other" (Stein et al., 2016).

Overall, the recent demonstrations of semantic processing under CFS beget the question whether an unconscious semantic representation of the self can be processed unconsciously. The prospect of revealing semantic self-processing unconsciously has been troublesome, but evidence indicates that this prospect may be bolstered by investigating semantic self-processing in combination with lower-level understandings of the bodily self. Here we investigated if participants are (i) able to process self-relevant semantic stimuli unconsciously and (ii) if this is modulated by task irrelevant stimuli, which tap into a specific multisensory dimension of the bodily self, i.e., the PPS. In a first experiment, we present participants with self-attributed personality trait to their non-dominant eye, while presenting dynamic high-contrast stimuli to their dominant one in order to mask the self-attribute. Participants were then to categorize the presented but unseen words as either self-related or not. We tested whether participants were able to semantically process self-related information inasmuch to appropriately categorize the words as self-relevant or not. Successful categorization of unseen words as either self-related or not is taken to index unconscious self-processing and is thus the dependent variable of interest here. That is, we questioned whether subjectively invisible words could be nonetheless accurately classified as self vs. non-self. In a second experiment, we tested whether such putative self-effect could be modulated by a low-level manipulation prioritizing the representation of the bodily self. One key aspect of bodily self is its link to the body and to the space immediately surrounding it, i.e., the PPS (Blanke, 2012; Blanke et al., 2015; Noel et al., 2015a, 2016; Canzoneri et al., 2016; Salomon et al., 2016a). Thus, here, we tested the hypothesis that presenting an acoustic stimuli within, as opposed to outside, PPS may bolster unconscious semantic self-processing. In this manner, we aimed at testing a direct link between bodily and narrative representations of the self and how they interact to modulate awareness.

Continuous flash suppression was utilized in order to mask presented stimuli as it has been suggested that this paradigm allows for some sort of non-conscious physical/perceptual (Geng et al., 2012), but not the psychological/conceptual self-processing (Stein et al., 2016). Hence, it represents an ideal candidate to reveal prioritized non-conscious processing by the psychological self, when this latter one is related to its physical conception. Lastly, self-rated personality attributes were used as these tap into the core of the psychological self while equally being malleable in time and self-selected by the participants themselves. That is, in contrast to other experiments that have used a fixed set of stimuli (e.g., Blake, 1988), here we aimed at having a set of words that were always the same yet for some participants fell within the "self " category, while for other fell in the "non-self " category.

#### MATERIALS AND METHODS

#### Participants

Forty-five native French speakers (nine females, mean age = 22.2 years old, range = 18–30 years old) completed a self-attribution questionnaire (see below), out of which 23 (six females, mean age = 21.4 years old, range = 18–27 years old) partook in both Experiment 1 and Experiment 2. The rest of participants were discarded from participation in the experimental phase, as their self-reports (see below) showed a bias in the attributes they categorized as "self " or "non-self " in terms of either the word-length, or the frequency of the select word in the French language. All participants were right-handed, had normal or corrected-to-normal visual acuity, and reported normal hearing. The study was approved by the Brain Mind Institute Ethics Committee for Human Behavioral Research of the EPFL, and conducted in line with the Declaration of Helsinki. All participants gave informed consent prior to participation and were remunerated with 20 Swiss Francs for their time.

## Self-attribution Reports

#### Materials, Procedure, and Analyses

Anderson's Likableness ratings of 555 personality-trait words (Anderson, 1968) were sorted from most to least likable (according to the mean values in Anderson, 1968), and the middle 200 words (word 178: hopeful to word 378: frustrated<sup>1</sup> ) were translated into French by three native French speakers. If translators did not independently agree on the translation of an attribute, they were asked to reach a consensus. This middle section of Anderson's personality-trait words was selected in order to increase between-subject variability in self-attribution reports (see below) while also minimizing the within-word likability variance.

Forty-five participants completed, online and at their own time of convenience, a self-attribution questionnaire, which

<sup>1</sup>http://psy2.ucsd.edu/~pwinkiel/words.txt

was constituted of the translated 200 personality-trait words. Participants were asked to rate on a Likert Scale (1 = completely not-self to 7 = completely self) whether the attributes described themselves (self) or not (non-self), and whether they liked the word (also on a Likert Scale). No further instruction was given to participants completing the questionnaire in terms of what they should consider when judging whether a particular attribute described them or not. Word order was randomly shuffled between participants. Upon receiving participant's response to the self-attribution questionnaire, for each subject individually, the list of word was sorted from most to least "self," and the top and lower 50 words were compared by means of a paired t-test (alpha set at 0.05, two-tailed) for word-length, frequency in the French language (word frequency was obtained from http: //www.lexique.org/listes/liste\_mots.php), and likability. If for a particular subject the group of self and non-self words differed any of the variables stated above (length, frequency, or likability), the participant was not invited to partake in Experiments 1 and 2.

# Experiment 1

#### Materials and Apparatus

Visual stimuli consisted of high-contrast dynamic noise patches suppressors (Mondrians; Jiang and He, 2006; Hesselmann and Malach, 2011) and target stimuli. The target stimuli consisted of one of the personality-trait words (either "self " or "non-self ") in black Times New Roman, font 12 (example in **Figure 1**; "Hopeful"). Background was white. Mondrians (noise stimuli) were flashed at 10 Hz to the participants' dominant eye, and the targets were presented simultaneously to the other eye. A red fixation point was presented to both eyes. The target words we presented either above or below the fixation point in a randomized fashion. Stimuli were presented using ExpyVR<sup>2</sup> , an in-house custom-built multimedia stimuli presentation software developed with Python 2.6 and the Open Graphics Library v.2.2. The stimuli were viewed via a Head-Mounted Display (HMD: VR1280, Immersion Inc, SXGA, 60◦ diagonal field of view, refresh rate 60 Hz). Participants' responses were gathered via button-press on a gamepad (XBOX 360 controller, Microsoft, Redmond, WA, 215 Hz sampling rate).

#### Procedure

Ten to 15 days after completion of the self-attribution report, subjects were brought into the lab for completion of the experimental phase (Experiments 1 and 2, which were ran in a counter-balanced order across participants). Participants were first tested for ocular dominance via the Miles test (Miles, 1930). Then, they were instructed that on each trial, in addition to the fixation point, which they were to gaze at, they would see a random pattern of colorful squares. They were equally told that, during some trials they would additionally see "a sequence of letters." They were instructed to press, during the trial, a button on a gamepad if they saw this additional stimulus. Those trials were indexed as having broke suppression, and removed from analyses for the stimuli categorization and location questions (see below), as we are interested in trials for which no subjective experience of the words was reported (i.e., unconscious self-processing). Then, as portrayed in **Figure 1** (upper), subjects were informed that upon completion of a trial, they would see the fixation point changing color: first blue, which indicated participants were to answer to the question "Was the additional stimuli self-related or not?", then green, which indicated that subjects were to answer to the question "Was the additional stimuli presented above or below fixation?" The questions were always asked in this order, and participants were reassured that if they had not seen anything (apart from the Mondrians and the fixation point) they were to simply guess. The question related to stimuli location (above or below fixation) was employed as a probe that some minimal form of processing was present when suppressing self-attributes. Participants responded via button-press, which mapping (self/non-self and above/below) was randomized across participants.

Trial presentation (Mondrians and personality-trait words) lasted for 3 s. Contrast of the target stimuli was ramped linearly from zero to full contrast over the 1 s. Participants were given unlimited time to answer both questions (Self-Attribution and Stimulus Location). Inter-trial interval was randomly shuffled between 1 and 1.5 s. The experiment consisted of 200 trials (presentation of each of the 100 participant-specific personalitytrait attributes – 50 self and 50 non-self – twice. Once above and once below fixation). Total experimental time was about 30 min.

#### Analyses

Participants' responses were first analyzed in terms of percentage of trials in which subjects broke suppression for self vs. non-self words, as well as for words presented above vs. below fixation. Then, subsequently to discarding the trials in which participants broke suppression (e.g., participants were conscious of the stimuli), Signal Detection Theory (SDT) analysis (sensitivity – d<sup>0</sup> , and criterion – c) was applied to both the Self-Attribution and Stimulus Location questions. Null effects were assessed using JZS Bayes factor (BF) tests with default prior scales (Rouder et al., 2012) so that a BF < 0.33 implies substantial evidence for the null hypothesis, 0.33 < B < 3 suggests insensitivity of the data, and B > 3 implies substantial evidence for the alternative hypothesis (see Dienes, 2011).

#### Experiment 2

The material and apparatus, as well as the procedure and analyses used in and applied to Experiment 2 (which, again, was counterbalance in order with Experiment 1 across participants) followed largely those of Experiment 1, for the exception of the following.

In order to probe at whether an implicit association with the bodily self, by delivery of auditory stimuli within the PPS (Noel et al., 2015a; Serino et al., 2015; Salomon et al., 2016a), would boost non-conscious self-processing, acoustic stimuli (white noise) were presented both close to, and far from, the body. Specifically, as depicted in **Figure 1** (lower), a speaker (Near) was placed 20 cm in front of the participants in the anterior-posterior axis, at his or her midline and at sternum level, while a second speaker (Far) was placed 100 cm away (at the same horizontal and elevation level). The speakers emitted white noise for the last second of trial presentation. They were both regulated in

<sup>2</sup>http://lnco.epfl.ch/expyvr

fixation?". In Experiment 2, the same protocol as in Experiment 1 was repeated, but additionally a white noise sound was presented for a second (the last second of the attribute presentation) either from the near or from the far speaker.

order to produce 70 dB(A) of sound measured at the participants' ear. White noise was utilized (as opposed to more naturalistic stimuli) as this stimuli has been demonstrated to most effectively drive neurons encoding for PPS (Fogassi et al., 1996), and the onset of the acoustic stimuli was offset with respect of onset of visual stimuli in order to minimize the effect audio onset can have in thrusting a visual stimuli into awareness (Aller et al., 2015). Experiment 2, as Experiment 1, also consisted of 200 trials (presentation of each of the 100 participant-specific personalitytrait attributes twice. Once while a near sound was presented, and once while a far sound was presented (randomized on a trial by trial fashion). Total experimental time was about 30 min.

# RESULTS

# Experiment 1

#### Breaking Suppression

A repeated measures ANOVA was ran on the percentage of occasions in which self vs. non-self words broke suppression, as well as for words presented above vs. below fixation. Results revealed no significant difference between levels for neither Self-Attribution [F(1,21) = 0.241, p = 0.83] or Stimulus Location [F(1,21) = 0.832, p = 0.64], nor an interaction between these variables [F(1,21) = 0.828, p = 0.40]. Bayesian statistics demonstrated a BF of 0.23 in the case of the former and 1.15

in the case of the latter. Thus, in the case of the frequency with which self vs. non-self words broke suppression, data implies substantial evidence for the null hypothesis. Overall, stimuli broke suppression in about a fifth of the trials (self: M = 20.0%, SE = 6.4%; non-self: M = 19.7%, SE = 6.4%; above: M = 19.0%, SE = 6.5%; below: M = 20.7%, SE = 6.3%).

#### Self-Attribution

Sensitivity and criterion analysis on participants' response to the question; "Was the additional stimuli self-related or not?" was effectuated by means of purpose-made scripts in MATLAB (equations follow Macmillan and Creelman, 2005) and subsequent one-sample t-test comparison to zero (meaning no sensitivity and no bias, respectively for d' and c). The "Self " category was defined as "target," while the "Non-Self " category was used as "noise." Results revealed that participants were not able to discriminate between self and non-self words [d'; M = −0.49, SEM = 0.27; t(22) = −1.35, p = 0.32], nor did they show any bias [c; M = 0.03, SEM = 0.06; t(22) = −0.60, p = 0.55]. In the case of d', Bayesian statistical analysis suggests a lack of sensitivity data (BF = 0.82), nonetheless, in the case of the response bias Bayesian statistics provided substantial evidence for the null hypothesis (BF = 0.25).

#### Stimulus Location

In terms of subjects' responses to the question about Stimulus Location; "Was the additional stimuli presented above or below fixation?", and defining "Above" as "target" and "Below" as "noise," findings revealed that participants were indeed able to discriminate between the elevation of the personality traits presented [d'; M = 1.24, SEM = 0.19, t(22) = 5.82, p < 0.001] and did not show a criterion bias [c; M = 0.10, SEM = 0.05; t(22) = −1.24, p = 0.12]. In the case of the latter, the Bayes factor (BF = 1.1) seems to indicate a lack of sensitivity in the data.

# Experiment 2

#### Breaking Suppression

Repeated-measures ANOVAs were conducted on the percentage of occasions in which self vs. non-self words broke suppression, as well as for word location (Above vs. Below fixation), as a function of the location of sound presentation (Near or Far). Concerning the frequency in which self vs. non-Self words broke suppression as a function of the distance of sound presentation, results revealed no main effect, neither for Self-Attribution [F(1,21) = 0.83, p = 0.37], or Sound Location [F(1,21) = 0.04, p = 0.92], nor an interaction between these variables [F(1,21) = 0.99, p = 0.33]. In the case of the main effect Bayesian statistics demonstrated insensitive data (all BFs > 0.62), while in the case of the interaction these statistics provide substantial evidence for the null hypothesis (BF = 0.18).

A similar pattern of results was observed for the frequency with which words presented Above vs. Bellow fixation broke suppression, as a function of Sound Location. The repeated-measures ANOVA showed no main effect of Word Location [F(1,21) = 0.94, p = 0.35] or Sound Location [F(1,21) = 0.745, p = 0.41], nor an interaction between these variables [F(1,21) = 0.56, p = 0.57]. Bayesian Factors demonstrated substantial evidence for the null effect in the case of breaking suppression for attributes presented above vs. below fixation (BF = 0.22), while in the other cases Bayesian factors demonstrated the insensitivity of the data (all BF > 0.38).

#### Self-Attribution

As for Experiment 1, the analysis of the self-attribution question was performed by SDT. We compare sensitivity and criterion bias across Sound Locations by means of a paired-samples t-test, and subsequently, if Sound Location significantly modified either the sensitivity or criterion to Self or Non-self attributed words, we compared these parameters to the zero-baseline. As portrayed in **Figure 2A**, results revealed that regardless of the Sound Location, participants were not able to discriminate between Self and Non-Self words [t(21) = −0.58, p = 0.56; d 0 near, M = −0.14, SEM = 0.19; d 0 far, <sup>M</sup> = −0.08, SEM <sup>=</sup> 0.16], as in Experiment 1. Further, Bayesian statistics demonstrated substantial evidence for the null hypothesis, both for the case of close sounds (BF = 0.29) and far sounds (BF = 0.26).

In terms of the criterion, however, as shown in **Figure 2B**, findings did demonstrate a significant difference between Sound Location levels [t(21) = 3.76, p < 0.01], revealing a more conservative approach for identifying "Self " when a sound was presented Far (M = 0.07, SEM = 0.02), as opposed to when the sound was presented Near (M = −0.003, SEM = 0.05) to the participant. Comparisons to zero, indicated that the "far criterion," was indeed significantly different from no bias [t(21) = 1.80, p = 0.03], while the "near criterion" was not [t(21) = −0.05, p = 0.95]. These findings are further supported by the demonstration that in the case of the latter (near sounds), but not the former (far sounds), Bayesian statistics provided substantial evidence in favor of the null hypothesis (BF = 0.22).

#### Stimulus Location

With regard whether the target stimuli was presented above or below fixation, analysis demonstrated no significant difference between Sound Location levels [t(21) = −0.31, p = 0.75, BF = 0.32]. It must be noted, however, that as depicted in **Figure 2C**, and replicating the results from Experiment 1, participants did prove to be overall sensitive to the location of the target stimuli [d'; M = 1.05, SEM = 0.29, t(21) = 4.622, p < 0.001, two-tailed one-sample t-test vs. zero].

Lastly, contrarily to the case of the Self-Attribution reports, and as shown in **Figure 2D**, participants did not show a systematic bias in their response to the elevation of target stimuli as a consequence of Sound Location [t(21) = 0.338, p = 0.73]. BF was equal to 0.32, providing evidence for the postulation that participants were equally likely to categorize unseen visual stimuli as happening above or below fixation regardless of whether an acoustic stimulus was presented near or far from them.

## DISCUSSION

In the present study, we first posed the question whether self-relevant (vs. non-self relevant) semantic information is preferentially unconsciously processed. In addition, we tested

from them (D). Error bars represent ±1 SEM. <sup>∗</sup>p < 0.05; ∗∗p < 0.01.

whether such self-effect could be modulated by priming a bodilyself representation via the presentation of a near sound within PPS.

In line with previous studies investigating semantic processing outside awareness (Zimba and Blake, 1983; Blake and Logothetis, 2002; Dehaene et al., 2006), our results indicate that masked words were not processed at the semantic level – at least inasmuch as to reveal a self-processing advantage (see Tacikowski and Ehrsson, 2016, as well as Stein, et al., 2016). This was true both in the case when no sounds were presented, and when sounds were presented either close to or far from the participant. It is important to highlight, however, that participants were able to correctly localize the elevation of the presented unseen visual stimuli, and thus, the lack of evidence for semantic processing is apparent even in light of clear evidence for lower-level subliminal processing. Blake (1988) also reported that even the first name of a specific observer, which represents a fundamental constituent of the semantic self (Joubert, 1991), proved insufficient to alter dominance of one percept (self) over another (non-self) under a condition of dichoptic stimulation. In contrast to the traditional view, however, an array of recent studies employing the b-CFS paradigm seem to demonstrate high-level processing of stimuli suppressed from conscious experience. Most notably, Jiang et al. (2007) showed that words that were visually familiar to a specific population's lexicon broke suppression more readily than foreign words. Costello et al. (2009), similarly demonstrated that semantic priming subsequently accelerated breaking suppression of related words, and finally Yang and Yeh (2011) reported that the affective connotation of an unseen word altered the duration for which

they could be maintained under suppression. It is under this context – one that has previously indicated the presence of semantic processing outside of awareness (Jiang et al., 2007; Costello et al., 2009) and even evidencing congruency priming under complete unawareness (Faivre et al., 2014; but see Noel et al., 2015b), that the current findings are important. It may be that the familiarity of words and their affective connotation can be processed and even integrated outside awareness, yet self-related and non-self related words are not differentially prioritized, at least inasmuch as this processing would entail a sensitive categorization of self and non-self words into their appropriate categories unconsciously. That is, although generally studies employing dichoptic presentation methods other than CFS have reported limited unconscious semantic processing (e.g., Blake, 1988), the recent surge in popularity of the CFS method and the numerous findings indicating unconscious processing under this paradigm has renewed interest in unveiling unconscious semantic processing. In the current study, we thus utilize the CFS method in order to mask self-attributes, and our results demonstrate no prioritized unconscious processing of the self vs. the non-self words.

On the other hand, our results revealed an interesting result relating to a shift in the criterion bias as a consequence of the location of sound presentation. That is, while in Experiment 1 no criterion bias was present, findings from Experiment 2 revealed a more conservative self-categorization approach (or a more liberal other-categorization) when sounds were presented outside, as opposed to inside, their PPS. That is, individuals required more evidence in order to judge that the unseen word presented was "self " when a sound was presented far as opposed to near. This differential response bias as a consequence of sound location was specific for the self dimension, as we did not observe a differential response bias among sound locations for the indication of whether subliminal visual stimuli were presented above or below fixation. That is, arguably, the self-other distinction process was modified when an auditory stimulus was provided far from the self. This effect is reminiscent of behavioral (Tajadura-Jiménez et al., 2013; Teneggi et al., 2013; Noel et al., 2015a; Salomon et al., 2016a) and neuroimaging (Brozzoli et al., 2012, 2013) multisensory studies demonstrating a self-other differentiation process grounded on the representation of PPS. PPS, indeed, might represent a multisensory-motor representation of the self in interaction with the environment (Noel et al., 2014; Galli et al., 2015; Serino et al., 2015), and the PPS boundaries might define a first level of selfother distinction (Noel et al., 2016). In fact, Blanke et al. (2015) hypothesized that changes in the size of PPS neurons receptive fields might underpin alterations in states of bodily self-consciousness, and we have recently demonstrated that subjective changes in self-location co-vary with changes (i.e., spatial translations) in the representation of PPS (Noel et al., 2015a) and even for unconscious manipulations of bodily self consciousness (Salomon et al., 2016a). Here, we show that priming the PPS boundary by presenting an auditory stimulus within the PPS, as contrasted to a stimulus outside the PPS, can modify the criterion with which individuals categorize stimuli as self-related or other-related. That is, the administration of spatially restricted exteroceptive stimuli either within or outside the PPS alters cognitive processes with regard the self. We find this observation particularly interesting as it seems to imply a continuous and dynamic re-assessment to the self as external objects and events come in contact with our sensory modules. Importantly, this change in criterion as a consequence of sound location, however, did not modulate the emergence of self-attributes into awareness, but changed the post-perceptual criterion participants used to determine whether the invisible word was self-related or not.

# CONCLUSION

It appears that though bodily representations of the self are able to modulate access to visual awareness (Lunghi et al., 2010; Lunghi and Alais, 2013; Salomon et al., 2015b), they do not enhance non-conscious processing of the narrative self (or at least association with the PPS does not). Importantly, self-attributes did not emerge into awareness in a prioritized way, even when presented in synchrony with exteroceptive signals that did alter self-bias. This lack of effect could putatively emanate from an inability of unconscious semantic processing, and not from a lack of enhanced self-processing. Nonetheless this option is in contrast to the results of recent studies demonstrating semantic processing of subliminal stimuli masked by means of CFS (Yang and Yeh, 2011; Prioli and Kahan, 2015). In conclusion, our results demonstrate no evidence for differential processing of masked semantic self-relevant stimuli, but do demonstrate a shift in criterion bias as a function of whether stimuli are presented in the peri- or extra-personal space.

# ETHICS STATEMENT

The study was approved by the Brain Mind Institute Ethics Committee for Human Behavioral Research of the EPFL, and conducted in line with the Declaration of Helsinki. All participants gave informed consent prior to participation and were remunerated with 20 Swiss Francs for their time.

# AUTHOR CONTRIBUTIONS

OB, AS, and RS conceived of the study, which was performed by J-PN. J-PN analyzed the data and wrote the manuscript, which was then edited by AS and RS. All authors approved the final version of the manuscript.

# ACKNOWLEDGMENTS

RS was supported by the National Center of Competence in Research (NCCR) "SYNAPSY – The Synaptic Bases of Mental Diseases" financed by the Swiss National Science Foundation (No. 51AU40\_125759). J-PN was supported by a Fulbright Scholarship from the US Institute of International Education.

# REFERENCES

fpsyg-08-00072 January 27, 2017 Time: 15:14 # 9



suppression? Front. Hum. Neurosci. 5:167. doi: 10.3389/fnhum.2011. 00167 doi: 10.3389/fnhum.2011.00167


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Noel, Blanke, Serino and Salomon. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Between-Subject Variability in the Breaking Continuous Flash Suppression Paradigm: Potential Causes, Consequences, and Solutions

#### Surya Gayet <sup>1</sup> \* and Timo Stein2, 3

<sup>1</sup> Department of Experimental Psychology, Utrecht Univeristy, Utrecht, Netherlands, <sup>2</sup> Center for Mind/Brain Sciences, University of Trento, Rovereto, Italy, <sup>3</sup> Amsterdam Brain and Cognition, University of Amsterdam, Amsterdam, Netherlands

#### Edited by:

Guido Hesselmann, Charité Universitätsmedizin Berlin, Germany

#### Reviewed by:

Pieter Moors, KU Leuven, Belgium Zhicheng Lin, Ohio State University at Columbus, USA

> \*Correspondence: Surya Gayet s.gayet@uu.nl

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 17 January 2017 Accepted: 08 March 2017 Published: 27 March 2017

#### Citation:

Gayet S and Stein T (2017) Between-Subject Variability in the Breaking Continuous Flash Suppression Paradigm: Potential Causes, Consequences, and Solutions. Front. Psychol. 8:437. doi: 10.3389/fpsyg.2017.00437 A recent focus in the field of consciousness research involves investigating the propensity of initially non-conscious visual information to gain access to consciousness. A critical tool for measuring conscious access is the so-called breaking continuous flash suppression paradigm (b-CFS). In this paradigm, a high contrast dynamic pattern is presented to one eye, thereby temporarily suppressing a target stimulus that is presented to the other eye. The time it takes for observers to report (e.g., the location of) the initially suppressed stimulus provides a measure of conscious access. Typical observations in b-CFS studies include the finding that upright faces are released from suppression faster than inverted faces, and the finding that stimuli that match the current content of visual working memory are released from suppression faster than mismatching stimuli. Interestingly, the extent to which observers exhibit these effects varies extensively (in the range of hundreds of milliseconds). By re-analyzing existing datasets and a new dataset we establish that the difference in RTs between conditions in b-CFS tasks (i.e., the effect of interest) is highly correlated with participants' overall suppression durations, and with their trial-to-trial variability in RTs. We advocate the usage of a simple latency- normalization method, which (1) removes the between-subject variability in suppression duration from the effect of interest, while (2) providing distributions of RT differences that are better suited for parametric testing. We next compare this latency-normalization method to two other transformations that are widely applied on within-subject RT data (z-transformations and log-transformations). Finally, we tentatively discuss how trial-to-trial variability and overall suppression duration might relate to prolonged phases of shallow suppression that are more prone to modulations of conscious access.

Keywords: consciousness, visual awareness, continuous flash suppression, binocular rivalry, individual differences, response times, normalization, assumption of normality

# INTRODUCTION

Consciousness researchers have many psychophysical tools at their disposal to render visual input invisible to the observer (for reviews, see Kim and Blake, 2005; Breitmeyer, 2015; Overgaard, 2015). One method in particular, however, has been increasingly dominant in the recent consciousness literature. Continuous flash suppression (CFS; Tsuchiya and Koch, 2005) consists of presenting a high contrast dynamic masking stimulus to one eye, thereby causing prolonged suppression of a lower contrast static target stimulus presented to the other eye. In 2015 alone, more than 40 studies have been published that use CFS. In particular, many studies use the time it takes for an initially suppressed target stimulus to overcome CFS as a measure of conscious access. This method is referred to as "breaking continuous flash suppression" (b-CFS; Jiang et al., 2007; Stein et al., 2011a; for a review, see Gayet et al., 2014).

Recent findings using this method have sparked the debate about the extent to which visual input is processed in the absence of consciousness (e.g., Hassin, 2013; Gayet et al., 2014; Yang et al., 2014; Hesselmann and Moors, 2015). Response times (RTs) in b-CFS experiments are lower than in traditional detection tasks, where information is not interocularly suppressed. Simultaneously, raw RT differences between experimental conditions are typically larger (in absolute time) than those typically observed with traditional detection tasks (e.g., comparing Stein et al., 2011a, with Lewis and Ellis, 2003). Eyeballing individual participants' b-CFS data (e.g., in Jiang et al., 2007; Wang et al., 2012) reveals an interesting data pattern: participants with slower overall RTs across conditions typically exhibit a larger RT difference between conditions. A number of recent b-CFS studies explicitly reported this correlation between overall RT and raw RT difference between conditions (e.g., Gayet et al., 2016a,b). Similarly, it has been demonstrated that artificially lengthening the RTs (e.g., by reducing the contrast of the target or by using more potent masks) induces larger raw RT differences (Stein et al., 2011a). As such, the relatively slow RTs in the b-CFS paradigm might play a role in the paradigm's potency for uncovering (small) differences in processing strength between stimulus conditions, which would drown in noise using traditional methods. Another interesting observation is that the trial-totrial variability in RTs in b-CFS experiments is higher than in traditional detection tasks where information is not interocularly suppressed (i.e., within-participant variance is typically much larger in b-CFS conditions than in control conditions involving no interocular competition). Greater variability in detection times implies that observers have greater uncertainty regarding the appearance of a target stimulus. This might also play a role in the b-CFS paradigm's sensitivity for detecting differences between conditions (for a discussion, see Stein et al., 2011a). Considering that the b-CFS paradigm is widely employed to answer important theoretical and philosophical questions on consciousness, it is important to better understand the mechanisms that play a role in the potency of the b-CFS method in uncovering differences in visual processing between experimental conditions.

The purpose of this article is 3-fold. First, we empirically establish the abovementioned relationships between overall RTs, trial-to-trial variability and raw RT differences within the b-CFS paradigm. For this, we use two of the most replicated findings of the b-CFS literature: face inversion and VWM boost. For those effects, we demonstrate that a participant's raw RT difference between two experimental conditions is strongly correlated with (A) the participant's overall RT across conditions and (B) the participant's trial-to-trial variability in RTs. As it is well established that overall RTs correlate with RT variability (Wagenmakers and Brown, 2007), a correlation of both metrics with RT differences is expected. Second, we suggest that this observation is of particular relevance in the b-CFS paradigm, where between-subject variability in RTs stems from both interindividual differences in interocular suppression durations, as well as interindividual differences in response speed after the interocular conflict is resolved. Following this consideration, we provide a clear-cut solution for handling data in which such a correlation is observed. The proposed latency-normalization procedure, which removes between-subject variability of-nointerest, has been used in only a few previous b-CFS studies. Here, we show for the first time that (1) this latency-normalization procedure yields RT differences that more closely approximate a normal distribution, and (2) we make the case that it provides increased sensitivity for detecting differences between experimental conditions. Third, we tentatively propose that the correlation between overall RTs, trial-to-trial variability in RTs, and raw RT differences between conditions could reflect the role of perceptual uncertainty in modulating RTs in the b-CFS paradigm. Finally, we discuss how this putative role of perceptual uncertainty could allow for reconciling contradictory and surprising findings that have been observed in recent b-CFS studies.

For these purposes, we retrieved data from three existing b-CFS datasets that we deemed most representative of the b-CFS paradigm. Following Gayet et al. (2014) we distinguished between two types of manipulations in the b-CFS paradigm. The first type of manipulations in b-CFS paradigms consists of manipulations of stimulus content, in which two different stimuli are compared in their propensity to reach conscious access (e.g., faces vs. houses). Arguably, the most reliable manipulation of stimulus content in the b-CFS literature is the finding that upright faces are released from interocular suppression faster than inverted faces (e.g., Jiang et al., 2007; Stein et al., 2011a). This face inversion effect is of particular interest because it involves the comparison of stimuli with identical pixel values, differing only in their spatial orientation on the screen. This approach thus allows for ruling out many low-level stimulus confounds that could explain differences in suppression durations between image categories (Stein et al., 2012). The second type of manipulations in b-CFS paradigms consists of manipulations of stimulus context, in which conscious access of a single stimulus is assessed as a function of its relation with consciously accessible information (e.g., a target following a congruent vs. an incongruent cue). With this approach, by definition, stimuli in different conditions are identical, such that low-level stimulus contributions cannot account for any difference in suppression durations between conditions. The most reliable manipulation of stimulus context arguably consists of manipulating the concurrent content of visual working memory. Applying this manipulation revealed that visual input that matches concurrently memorized visual features (such as a color or shape) is released from interocular suppression faster than visual input that mismatches the concurrently memorized content (Gayet et al., 2013, 2016a; Pan et al., 2014; van Moorselaar et al., 2015, 2016; Gayet, 2016).

# METHODS

#### Datasets

For the present set of analyses, we used three b-CFS datasets, one from an experiment that has not yet been published, and two datasets were retrieved from published studies (Gayet et al., 2013; Gayet, 2016). The first dataset comprises of a b-CFS experiment in which RTs to upright faces were compared with RTs to inverted faces (see Appendix for a complete description of the Methods). For the sake of brevity, this dataset will be referred to as the "Face Inversion" experiment. It revealed that upright faces are released from interocular suppression faster than inverted faces. The second dataset comprises of a b-CFS experiment in which RTs to colored targets are compared between the case in which they match and the case in which they mismatch a color that is concurrently maintained in visual working memory for a subsequent recall task. Similarly, the third dataset comprises of a b-CFS experiment in which RTs to geometrical target shapes are compared between the case in which they match and the case in which they mismatch a geometrical shape that is concurrently maintained in visual working memory. These datasets will be referred to as the "Color" experiment and the "Shape" experiment respectively (see Gayet et al., 2013; Gayet, 2016, for a complete description of the Methods in these two experiments). In short, the Color and Shape Experiments revealed that identical visual stimuli are released from interocular suppression faster when they match a stimulus that is concurrently maintained in visual working memory (for a subsequent recall task). The left part of **Table 1** provides general information on these three experiments, including the number of participants, the overall mean RT and the trial-to-trial variability (see the "metrics" paragraph below).

# Metrics

In order to investigate the relation between overall RTs, trial-to-trial RT variability and the RT difference between experimental conditions, we established the following metrics. Each Participant's overall RT (RTOVERALL) was defined as the average of the median RTs in each experimental condition that elicited a RT difference (i.e., upright and inverted, or matching and mismatching conditions, but also left and right eye, different target locations, and differently colored or shaped targets). Medians were used to account for the skewness in the distributions of raw RTs, and for including trials that yielded no response within the time window (reflecting the longest suppression durations). The trial-to-trial variability was computed as the average of the SDs of each experimental condition that elicited a RT difference, hence yielding a withincondition SD (or SDWITHIN), which only reflects RT variability that is not caused by the experimental manipulations. The raw RT difference was computed by subtracting each participants' median RT to upright faces (or matching targets) from the median RT to inverted faces (or mismatching targets).

## Analysis

We computed standard Pearson correlations, which were regarded as significant in case the p-value was below the threshold of α = 0.05 divided by the number of parallel comparisons, following the Bonferroni correction. In addition, we ran onesided Bayesian correlation analyses, testing for the evidence or absence of a positive correlation (using JASP Team, 2016; Beta prior width of 1).

# RESULTS I: BREAKING CONTINUOUS FLASH SUPPRESSION

### Overall RTS

The results of the main correlational analyses are depicted in **Figure 1A**. In all three datasets reported here, participants' overall RTs correlated with the difference in raw RTs between experimental conditions. This was true for the Face Experiment, R = 0.912, p < 0.001, BF+<sup>0</sup> = 7 ∗ 1071, for the Color Experiment, R = 0.726, p < 0.001, BF+<sup>0</sup> = 2 ∗ 10<sup>4</sup> , and for the Shape Experiment, R = 0.687, p = 0.001, BF+<sup>0</sup> = 38. This correlation has been reported in the b-CFS literature on a number of other occasions as well (e.g., Gayet et al., 2016a,b), and can also be observed when eyeballing individual participant's data in a number of other b-CFS studies (e.g., Jiang et al., 2007; Wang et al., 2012). Thus, this appears to be a common phenomenon in b-CFS data.

### A Simple Latency-Normalization Procedure

The present results show that, for different experimental manipulations in the b-CFS paradigm, participants with slower overall RTs also show a stronger RT difference between experimental conditions. In other words, it is possible that part of the between-subject variability in the RT difference between experimental conditions (i.e., the researcher's effect of interest) stems from between-subject variability in overall RTs (i.e., which is not related to the effect of interest). Consequently, when conducting statistical analyses to establish whether an effect of interest is present or absent, statistical power is reduced by between-subject variability that is unrelated to the effect of interest. A simple way to isolate the effect of interest from participants' overall response speed is to normalize the RT difference for each participant, such that it reflects a proportional (rather than absolute) difference in RTs caused by the experimental manipulation. This can be achieved by dividing the difference in RT between two conditions by the overall RT:

$$
\Delta RT\_{\text{NORMALIZED}} = 100 \ast \,\,\frac{RT\_A - RT\_B}{RT\_{OVERAL}}
$$

A similar approach has been proposed by Tsuchiya et al. (2009), by Stein (2012); Stein et al. (2011b, 2012), and by Gayet et al. (2016a); Gelbard-Sagiv et al. (2016).

In **Table 1**, frequentist and Bayesian test-statistics, as well as standardized effect sizes (Cohen's d) are provided for the raw RT difference and the normalized RT difference of all three datasets. This reveals that the latency-normalization procedure increased the sensitivity to detect differences between experimental conditions in all three datasets, as revealed by increased t-statistics and larger standardized effect sizes (Cohen's d) relative to the tests performed on the raw RT differences. This advantage of the normalization procedure was also observed in

FIGURE 1 | Results of the main correlational analyses in all three datasets. (A) Depicts the correlation between participants' overall RT (x-axis) and the RT difference between experimental conditions (y-axis) for all three datasets. (B) Depicts the correlation between the within-condition variability of RTs (x-axis) and the RT difference between experimental conditions (y-axis) for all three datasets. All correlations depicted here were significant after Bonferroni correction, with all p's < 0.005. Plots were adapted from JASP Team (2016) output.


\*Significant violation of normality according to the Shapiro-Wilk normality test.

Gayet et al. (2016b; Supplementary Materials). Thus, although we do not provide statistical confirmation of this improvement, the increase in test values is in line with the reasoning that removing a source of between subject variability of no interest (i.e., between subject variability in overall RTs) increases sensitivity for detecting an effect of interest (i.e., an RT difference between experimental conditions).

A second major asset of normalizing the RT differences in b-CFS experiments pertains to the normality of the distribution of RT differences. Indeed, the difference in raw RTs between experimental conditions tends to be skewed toward the tail end of the RT distribution (**Figure 2A**). For instance, in the Color experiment data, the assumption of normality is violated, according to the Shapiro-Wilk test, W = 0.848, p < 0.001.<sup>1</sup> After the normalization procedure described above, however, this is no longer the case, W = 0.954, p = 0.155 (**Figure 2B**). Similarly, in the Shape experiment, the distribution of raw RT differences violated the assumption of normality, W = 0.774, p < 0.001, whereas the distribution of normalized RT differences did not, W = 0.951, p = 0.406. Again, in the Face Inversion experiment, the distribution of raw RT differences violated the assumption of normality W = 0.857, p < 0.001, whereas this was not (or much less) the case with the distribution of normalized RT differences, W = 0.966, p = 0.067. The distributions of the RT differences after the normalization procedure are depicted in **Figure 2B**. The reduction in the rightward tail-end of the RT difference by the latency-normalization procedure implies that this procedure reduces the amount of Type II errors (i.e., failing to detect an existing difference between conditions). The fact that the latencynormalized data more closely followed a normal distribution also implies greater control of Type I error rate compared to the skewed distribution of raw RT differences. Taken together, the latency-based normalization procedure offers an improved way to conduct parametric tests on b-CFS data. In addition, it reveals that the proportional (normalized) modulation of RTs in the b-CFS paradigm is approximately normally distributed (at least for the experimental manipulations provided here).

# A Comparison with Other Normalization Procedures

In order to evaluate the latency-normalization method, we also carried out two other transformations on our data sets that are widely applied to within-subject RT data: log-transformation and z-transformation (for a review, see Bush et al., 1993). Specifically, we aimed to investigate whether, within the b-CFS paradigm, RT differences between conditions would (1) yield larger effect sizes, and (2) more closely approximate a normal distribution, following these transformations as well. Ztransformations were obtained by dividing the raw RT difference between two conditions A and B (e.g., upright and inverted faces), by the standard deviation of the RT difference between these two conditions, within each participant:

$$
\Delta \text{RT}\_{\text{Z-TRANSFORMED}} = \frac{RT\_A - RT\_B}{SD\_{A-B}}
$$

The within-participant standard deviation for the difference in RTs between two conditions (SDA-B) was computed as follows:

SDA−<sup>B</sup> = s (N<sup>A</sup> − 1) ∗ SD<sup>2</sup> <sup>A</sup> <sup>+</sup> (N<sup>B</sup> <sup>−</sup> 1) <sup>∗</sup> SD<sup>2</sup> B (N<sup>A</sup> − 1) + (N<sup>B</sup> − 1)

where N<sup>A</sup> is the number of trials in condition A, and SD<sup>A</sup> is the standard deviation of the RTs in condition A. The logtransformed RT difference was obtained by first taking the logarithm with base ten of all RTs, and then computing the difference between the average<sup>2</sup> log-transformed RTs of the two conditions:

$$
\Delta \text{RT}\_{\text{LOG-TRANSFORDED}} = \log\_{10}(RT\_A) - \log\_{10}(RT\_B)
$$

Across our three data sets, the three transformations applied to the RT difference increased the sensitivity to detect the difference between experimental conditions (see **Table 1**), except for the z-transformation in the Shape experiment. Overall, although we did not compare the methods statistically, the test statistics and effect sizes for the RT difference was numerically larger after applying the log- and the latencybased normalization procedures than after applying the ztransformation procedure. In addition, the RT difference more closely approximated a normal distribution after the logtransformation and latency-normalization procedure than after the z-transformation procedure. After z-transformation, the assumption of normality was still violated in 2 out of 3 data sets. On the basis of these observations, we encourage the usage of either the latency-based normalization procedure or log-transformations to b-CFS data before conducting statistical analyses, to increase sensitivity by reducing type II error-rate.

# RESULTS II: A MONOCULAR CONTROL CONDITION

#### Within-Condition RT Variability

The results of the correlation analyses between the withincondition variability in RTs (i.e., SDWITHIN) and the raw RT difference between experimental conditions is depicted in **Figure 1B**. These analyses revealed significant correlations between the within-condition variability in RTs and the raw

<sup>1</sup> Individually, none of the three experiments of Gayet et al. (2013), which together constitute the data of the Color Experiment described here, violated the assumption of sphericity (all p's > 0.2). As such, the usage of parametric tests for these three separate experiments was allowed, although arguably suboptimal in retrospect.

<sup>2</sup> In the Face experiment, the CFS masks were ramped down to zero contrast over the course of a trial, ensuring that observers would eventually perceive the target on each trial, allowing them to provide a response within the time limit. The color experiment had a relatively long time limit of ten seconds, so that targets were responded to within the time limit in virtually all trials in this experiment as well. Hence, for these two experiments a mean log-RT could be computed for each condition. In the Shape experiment, however, participants failed to respond within the time limit on an average of 19 trials (SD = 18) out of 144. These trials, which yielded 'infinite' suppression durations, could only be included in the analyses by using medians as an averaging metric (for an elaborate discussion on this approach, see Gayet et al., 2016b, Section 2.5).

FIGURE 2 | Distributions of the raw (A), latency-normalized (B), z-transformed (C), and Log10-transformed (D), difference in RTs between experimental conditions in all three experiments. These graphs reveal that the latency-normalization procedure had the largest impact on the RT differences of those participants that showed extremely large differences in RTs between conditions. In addition, these graphs demonstrate that after the normalization procedure, the log-transformation and (to a lesser extent) the z-transformation, the distribution of RT differences more closely followed a normal distribution. Plots were adapted from JASP Team (2016) output.

RT difference between experimental conditions in the Face Experiment, R = 0.693, p < 0.001, BF+<sup>0</sup> = 1 ∗ 10<sup>8</sup> , in the Color Experiment, R = 0.465, p = 0.005, BF+<sup>0</sup> = 9, and in the Shape Experiment, R = 0.625, p = 0.004, BF+<sup>0</sup> = 13. Given that larger within-condition variability in RTs implies greater uncertainty, it is tentative to conclude that perceptual uncertainty positively correlates with the propensity of an experimental manipulation to affect RTs in a b-CFS paradigm. Arguably, the z-transformation described above increased effect sizes for RT differences by removing this source of between-subject variability from the RT difference between experimental conditions. We advocate caution in considering this interpretation, however, as it might be a spurious correlation emerging from the correlation between overall RTs and RT variability (e.g., Wagenmakers and Brown, 2007), which we also find in the Face Experiment, R = 0.797, p < 0.001, BF<sup>10</sup> = 4 ∗ 1023, in the Color Experiment, R = 0.736, p < 0.001, BF<sup>10</sup> = 4 ∗ 10<sup>4</sup> , and in the Shape Experiment, R = 0.659, p = 0.002, BF<sup>10</sup> = 23.

#### Sources of RT Variability

It should be made explicit that we do not know what part of the between-subject variability in overall (i.e., median) RTs stems from individual differences in suppression durations, and what part stems from individual differences in response speed after the interocular conflict is resolved. Similarly, we do not know what part of the within-subject variability in RTs (i.e., the withincondition SD) stems from trial-to-trial differences in suppression durations, and what part stems from trial-to-trial differences in response speed after the interocular conflict is resolved. Finally, it is debated whether it is possible to unequivocally separate the contribution of these two processes that constitute b-CFS RTs (e.g., Stein et al., 2011a; Stein and Sterzer, 2014).

In order to separate the two, we capitalized on the fact that, in the Face Inversion experiment, the same participants also performed a control condition. In b-CFS control conditions, similar stimuli as in the CFS condition are used but no interocular suppression is induced. Hence, RTs in these control conditions offer insights into processes that take place after the interocular conflict is resolved. In these control conditions, both the CFS masks and the target stimuli are presented to both eyes (binocular control condition), or the CFS mask is presented to the same eye as the target stimulus, with the target stimulus displayed on top of the mask (monocular control condition). However, these control conditions typically yield null results, even when comparing stimuli that are well known to yield detection differences in other non-CFS detection paradigms, such as upright vs. inverted faces. The present Face Inversion experiment therefore adopted an improved control condition, in which face targets were blended into phase-scrambled face images (i.e., phase-scrambled noise, see Appendix for details). Face targets and phase-scrambled noise were presented to one eye, while the other eye was presented with the gray background only, such that no interocular suppression was induced.

In the Face Inversion experiment, the advantage for upright faces over inverted faces was also observed in the control condition but was smaller (9 vs. 26%) than in the CFS condition, t(64) = 8.999, p < 0.001, BF<sup>10</sup> = 3 ∗ 1010. Overall RTs, in contrast, did not significantly differ between the CFS condition and the control condition, t(64) = 1.506, p = 0.137, BF<sup>01</sup> = 3. These findings, summarized in **Table 2**, show that the upright face advantage is more pronounced in the CFS condition than in the control condition, even though overall RTs were successfully matched between the two (for a discussion on why this is important, see Stein et al., 2011a). Taken together, upright faces are detected faster than inverted faces when there is no interocular suppression (e.g., after the interocular competition is resolved) but, the larger RT difference in the CFS condition suggests that, on top of this, upright faces are also released from interocular suppression faster than inverted faces.

Similar to the CFS condition, the RT difference between upright and inverted faces in the control condition correlated positively with the overall response speed, R = 0.320, p = 0.009, BF+<sup>0</sup> = 9, and with the within-condition SD, R = 0.306, p = 0.013, BF+<sup>0</sup> = 6. These correlations (depicted in **Figures 3A,B** respectively) were less reliable and smaller in magnitude, however, than those observed in the CFS condition.

The within-condition SD (reflecting trial-to-trial variability) was much smaller in the control condition (M = 292 ms, SD = 143) than in the CFS condition (M = 1,298 ms, SD = 835), t(64) = 10.373, p < 0.001, BF<sup>10</sup> = 8 ∗ 1013. This observation is not surprising, considering that, in the CFS condition, our metric of within-condition variability captured both variability in suppression duration and variability in response speed after resolution of the interocular conflict, whereas in the control condition (variability in) suppression durations did not contribute to within-condition variability. The large difference in magnitude between the within-condition variability observed with and without interocular competition, however, indicates that the within-condition SD observed in the CFS condition mostly reflected trial-to-trial variability in suppression durations. In line with this view, the raw difference in RT between upright and inverted faces in the CFS condition did not correlate with


\*Significant violation of normality according to the Shapiro-Wilk normality test (none observed).

the within-condition variability in the control condition, R = 0.016, p = 0.902, BF<sup>01</sup> = 6 (see **Figure 3C**). This observation provides additional grounds for the notion that the relation between RT differences and trial-to-trial variability in RTs in the CFS condition (reported in the previous section) is primarily driven by trial-to-trial variability in suppression durations.

# GENERAL DISCUSSION

#### Brief Summary

Using three representative b-CFS datasets, we established that longer RTs relate to larger RTs differences between experimental conditions in the b-CFS paradigm. This may be considered a nuisance, because part of the between-subject variance that will affect statistical testing for a difference between experimental conditions (i.e., the effect of interest), does not reflect individual differences in the effect of interest, but rather reflects individual differences in overall RTs. Usually, these individual differences in overall RTs that are not specific to the experimental conditions, are of no interest to the research question. This relation between overall RTs and RT differences is probably inherent to RT data, and therefore not specific to the b-CFS paradigm (i.e., it might be a general law, following from the increase in RT variability that accompanies an increase in RT; Wagenmakers and Brown, 2007). It is of particular importance in the b-CFS paradigm, however, for two main reasons. First, in the b-CFS paradigm, overall RTs reflect both (1) the duration of interocular suppression and (2) the response speed to the stimulus after the interocular conflict is resolved. Both processes could independently elicit longer overall RT latencies and thus larger RT differences between experimental conditions than in experiments not involving CFS. Second, b-CFS is believed to be a highly potent measure to discern differences between experimental conditions because it yields large raw RT differences between conditions, with relatively few trials (e.g., compare the face inversion effect of Stein et al., 2011a, using b-CFS, with that observed in traditional detection paradigms reviewed in Lewis and Edmonds, 2003). At the same time, however, artificially stretching out RT differences between conditions, for example by presenting target stimuli at low contrast or by increasing depth of suppression through high mask contrast, also causes noise to be stretched out, allowing for more Type I errors (i.e., false positives) of a larger magnitude. For these reasons, it should become standard procedure in b-CFS experiments that examine differences between experimental conditions (the effect of interest) to investigate whether the effect of interest correlates with participants' mean RTs, and remove this between-subject variability of non-interest from the effect of interest.

# A Simple Solution

We encourage the usage of a simple normalization method (for similar approaches, see Tsuchiya et al., 2009; Stein et al., 2011b, 2012; Stein, 2012; Gayet et al., 2016a,b) to remove the influence of participants' overall RTs from the effect of interest. Using this method provides three clear benefits. First, removal of uninteresting between-subject variability provides a more precise measure of the effect of interest. Indeed, across all three data sets, test-statistics and effect sizes were improved after applying this normalization method to the RT differences. Second, the distribution of normalized RT difference approximates normality to a greater extent than the distribution of raw RT differences. This allows for conducting parametric tests, by transforming the data in a way that is tailored to the specifics of a participant's RT distribution, rather than being a generic approximation (e.g., such as with log-transformations). Third, normalized RT differences provide an intuitive measure of how an experimental manipulation affected participants' behavior, in the form of a proportional difference (reduction) in RTs caused by the experimental manipulation. This is an advantage over, for instance, log-transformed or z-transformed RT differences.

Log-transformed RT differences yielded increases in effect size, and approximations of normality, that were comparable to that of normalized RT differences. Therefore, log-transformations are an equally good solution for dealing with non-normal RT data with large between subject variability in the b-CFS paradigm. Z-transformations, on the other hand, were predominantly less successful in generating normally distributed RT differences.

# Suppression Durations

As we argued above, RTs in a b-CFS paradigm comprise both the duration of interocular suppression, and the response speed to the stimulus after the interocular conflict is resolved. Similarly, the individual differences in (trial-to-trial variability in) RTs that we analyzed in the present study reflect variability in suppression durations, as well as variability in response speed after the interocular conflict is resolved. We suggest that the relation between individual differences in RTs and individual differences in the effect of interest, which we report here, are largely specific to CFS, reflecting individual differences in suppression durations. This notion is supported by the finding that RT differences in conditions that include interocular suppression (the CFS condition) do not correlate with individual differences in RTs drawn from conditions that do not include interocular suppression (i.e., a control condition). Another argument stems from the magnitude of the effects of interest and the magnitude of individual differences in RTs (i.e., in interocular suppression conditions). These are much larger than those typically observed in non-suppression conditions, suggesting that suppression durations constitute a large portion of the eventual RTs and RT differences. Taken together, these observations suggest that individual differences in overall RTs primarily reflect individual differences in suppression durations.

# The Role of Response Bias

In this study, we observed that trial-to-trial variability was compellingly larger in the b-CFS suppression condition compared to the b-CFS control condition. This is potentially problematic, as the absence (or reduction) of an experimental effect in b-CFS control conditions is typically interpreted as evidence that an experimental effect in the CFS condition was not caused by anything happening after the interocular conflict was resolved, and thus reflects a difference in suppression duration. The difference in trial-to-trial variability between suppression and control conditions, however, could be associated with differences in perceptual uncertainty, such that response criteria might differ between suppression and control conditions. From this perspective, even the absence of an effect in a b-CFS control condition, does not necessarily preclude that an effect of interest in the b-CFS suppression condition was caused by a difference in response criteria. For example, more familiar stimuli, such as upright faces, might be more readily responded to in a state of perceptual uncertainty than less familiar stimuli, such as inverted faces (Stein et al., 2011a). In the control conditions, however, the lower levels of uncertainty could still be insufficient for the different response criteria between these conditions to modulate RTs. Future b-CFS studies could rule out the potential influence of such response biases by using non-speeded, signal detection-based bias-free protocols in which presentation times are fixed (e.g., Kaunitz et al., 2013; Lupyan and Ward, 2013; Hedger et al., 2015).

# The Role of Perceptual Transitions

In b-CFS, an initially fully invisible stimulus eventually becomes fully visible. However, this emergence into conscious awareness is not an abrupt, all-or-none phenomenon, but a gradual one (Stein et al., 2011a), and some stimulus properties become available to consciousness earlier than others. For example, recent observations have shown that, under conditions of prolonged CFS, periods of partial awareness might arise, in which some stimulus properties (such as color) are available to consciousness, whereas other are not (such as orientations; Zadbood et al., 2011; Yang and Blake, 2012). Thus, prolonged periods of continuous flash suppression might enable distinct stages of suppression strength that vary in the degree of susceptibility to experimental manipulations. This is in line with the idea that interocular competition is modulated at different levels throughout the visual processing hierarchy (Blake and Logothetis, 2002). A recent study showed that stimuli under CFS elicited high-level behavioral priming effects only under conditions of partial awareness, but not when fully suppressed (Gelbard-Sagiv et al., 2016). Similarly, a recent study demonstrated that manipulations of attention only affected dominance durations in a binocular rivalry paradigm around the time of perceptual transitions (Dieter et al., 2015). Another binocular rivalry study showed that the detection performance of monocular probes followed gradual changes in consciousness, rather than being dichotomous (Alais et al., 2010). Thus, variations in suppression strength, as are likely to occur over the course of a b-CFS (or binocular rivalry) trial, might enable different processes to modulate suppression durations of initially fully suppressed visual input.

Both fluctuations in suppression strength during individual trials, as well as the duration of perceptual transitions may vary between observers. It is thus conceivable that our current metrics (the trial-to-trial variability in RTs, and the overall RTs) relate to the duration of transitory percepts—with limited depth of suppression—in individual participants. Accordingly, certain experimental manipulations should exert more influence on the RTs of participants with longer transitory percepts (or longer periods of shallow suppression) than on the RTs of participants with shorter transitory percepts (or shorter periods of shallow suppression). Thus, longer periods of shallow suppression may allow for a greater extent of non-conscious processing, such that larger differences in detection times can be expected for participants with longer periods of shallow suppression. The correlation between longer overall RTs (and larger trial-totrial variability) and the RT difference between experimental conditions observed in this study fits well with this idea.

From a neural perspective, different levels of suppression might allow for neural responses to non-conscious stimuli to travel up to different levels of the (visual) processing hierarchy, thereby enabling more cognitive functions (e.g., Grill-Spector et al., 2000; Bar et al., 2001; Kanwisher, 2001; Supèr et al., 2001; Sergent and Dehaene, 2004). This leads to the expectation that, with manipulations requiring increasing cognitive demands, the relation between our current metrics and the modulation of RTs by the experimental manipulations should become more pronounced. The general consensus was, traditionally, that interocularly suppressed information cannot be processed at a semantic or conceptual level (e.g., Zimba and Blake, 1983; for a review, see Lin and He, 2009). This view stemmed from experiments using such variants of interocular suppression as binocular rivalry (Wheatstone, 1838; Alais and Blake, 2005) and flash suppression (Wolfe, 1984). In contrast, b-CFS studies have yielded contradictory findings, with some studies demonstrating that semantic or conceptual information can drive conscious access of initially suppressed visual input (e.g., Mudrik et al., 2011; Yang and Yeh, 2011; Sklar et al., 2012) while other studies have challenged this view (e.g., Heyman and Moors, 2014; Moors et al., 2016; Rabovsky et al., 2016; Stein et al., in press). Investigating the point in time (or the suppressive strength) at which a manipulation impacts conscious access in a b-CFS paradigm might be a valuable tool to resolve this apparent conflict.

#### ETHICS STATEMENT

All data analyzed in this study was retrieved from previously published or unpublished studies. Hence, despite human participant data was used in this study, it was already collected before this study was initiated. Hence, ethical statements

#### REFERENCES


are to be found in the articles from which the data was retrieved.

## AUTHOR CONTRIBUTIONS

The study idea was developed by SG and worked out in more detail by SG and TS. Data collection and analyses were conducted by SG and TS. The manuscript was initially written by SG, and was later improved by comments and suggestions provided by TS. All authors approved the final version of the manuscript.

#### ACKNOWLEDGMENTS

The authors want to thank Pieter Moors for useful comments and insights provided during review of an earlier version of this manuscript.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg. 2017.00437/full#supplementary-material


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Gayet and Stein. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Neural Correlates of Subjective Awareness for Natural Scene Categorization of Color Photographs and Line-Drawings

Qiufang Fu1,2 \*, Yong-Jin Liu<sup>3</sup> , Zoltan Dienes<sup>4</sup> , Jianhui Wu1,5, Wenfeng Chen<sup>1</sup> and Xiaolan Fu1,2

<sup>1</sup> State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing, China, <sup>2</sup> University of Chinese Academy of Sciences, Beijing, China, <sup>3</sup> Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China, <sup>4</sup> Sackler Centre for Consciousness Science, School of Psychology, University of Sussex, Brighton, UK, <sup>5</sup> College of Psychology and Sociology, Shenzhen University, Shenzhen, China

It remains controversial whether visual awareness is correlated with early activation indicated by VAN (visual awareness negativity), as the recurrent process hypothesis theory proposes, or with later activation indicated by P3 or LP (late positive), as suggested by global workspace theories. To address this issue, a backward masking task was adopted, in which participants were first asked to categorize natural scenes of color photographs and line-drawings and then to rate the clarity of their visual experience on a Perceptual Awareness Scale (PAS). The interstimulus interval between the scene and the mask was manipulated. The behavioral results showed that categorization accuracy increased with PAS ratings for both color photographs and line-drawings, with no difference in accuracy between the two types of images for each rating, indicating that the experience rating reflected visibility. Importantly, the event-related potential (ERP) results revealed that for correct trials, the early posterior N1 and anterior P2 components changed with the PAS ratings for color photographs, but did not vary with the PAS ratings for line-drawings, indicating that the N1 and P2 do not always correlate with subjective visual awareness. Moreover, for both types of images, the anterior N2 and posterior VAN changed with the PAS ratings in a linear way, while the LP changed with the PAS ratings in a non-linear way, suggesting that these components relate to different types of subjective awareness. The results reconcile the apparently contradictory predictions of different theories and help to resolve the current debate on neural correlates of visual awareness.

Keywords: visual awareness, subjective measures, objective measures, natural scene categorization, Perceptual Awareness Scale (PAS)

# INTRODUCTION

Recently, an increasing amount of attention has been given to neural correlates of visual awareness, i.e., the subjective experience of seeing. Neural imaging studies have revealed that a distributed system, including the primary visual cortex (V1), V2, temporal, parietal, and frontal areas, plays a crucial role in the generation of visual awareness (see Koch et al., 2016 for a review; Rees, 2001; Vuilleumier et al., 2002; Salminen-Vaparanta et al., 2012). Electrophysiological studies have shown

#### Edited by:

Guido Hesselmann, Charité – Universitätsmedizin Berlin, Germany

#### Reviewed by:

Zhicheng Lin, Ohio State University, USA Liad Mudrik, Tel Aviv University, Israel

> \*Correspondence: Qiufang Fu fuqf@psych.ac.cn

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 11 August 2016 Accepted: 01 February 2017 Published: 15 February 2017

#### Citation:

Fu Q, Liu Y - J, Dienes Z, Wu J, Chen W and Fu X (2017) Neural Correlates of Subjective Awareness for Natural Scene Categorization of Color Photographs and Line-Drawings. Front. Psychol. 8:210. doi: 10.3389/fpsyg.2017.00210

that an early positive enhancement (P1) over occipital regions that were approximately 100 ms from the stimulus onset, a negative difference wave (visual awareness negativity, VAN) at the posterior regions typically occurring 150–250 ms from the stimulus onset, and a P3 or a late positive (LP) wave occurring 300 ms from the stimulus onset were probably correlated with the emergence of visual awareness (see Koivisto and Revonsuo, 2010; Revonsuo and Koivisto, 2010; Naccache et al., 2016; contrast Silverstein et al., 2015). However, there is no agreement on what the identity of the neural indicator is for the emergence of visual awareness. For example, the N1 component (e.g., Jiang et al., 2013), the P2 component (e.g., Melloni et al., 2011), and the N2 component (e.g., Koivisto and Revonsuo, 2003) have been also argued to be related to visual awareness. The identity of the neural indicator for the emergence of visual awareness is still unclear.

One of the challenges of addressing this issue is the measurement of the status of visual awareness. There are two measures of awareness or consciousness in the literature: objective and subjective measures. With objective measures, the ability to discriminate features of the world is taken to indicate whether knowledge is conscious (Fu et al., 2008; Koivisto et al., 2008). An inability to discriminate reflects the absence of conscious knowledge. Using objective measures, P1 and VAN might be found to be related to visual awareness. For example, in a two-alternative matching to sample task, participants were asked to determine whether a briefly presented "sample word" was the same or different from a second "test word" by pressing one of two keys. Melloni et al. (2007) found that a P1 like component was significantly different between visible and invisible words, indicating that the P1 component was the earliest event-related potential (ERP) component that correlated with visual awareness. Pins and Ffytche (2003) also found a P1 effect for visual awareness, but this P1 correlate was not been observed in other studies (e.g., Sergent et al., 2005; Koivisto et al., 2008; Melloni et al., 2011). Moreover, Koivisto et al. (2008) compared the neural activities for hit trials (aware) with those for miss trials (unaware) and found that the posterior VAN was an earlier ERP correlate of visual awareness. The posterior VAN correlate has been observed in other studies (e.g., Koivisto and Revonsuo, 2003; Genetti et al., 2009).

With subjective measures, the ability to report or discriminate mental states is taken to indicate whether the knowledge is conscious (Fu et al., 2008; Lamy et al., 2009). If a person says he does not see or he has no experience of the stimulus, the knowledge revealed in the behavioral forced-choice response is unconscious by these measures. Using subjective measures, P3 or LP might be found to be related to visual awareness. For example, Lamy et al. (2009) compared the neural activities for correct trials in which participants reported they had seen the stimulus (aware-correct) and correct trials in which participants reported that they merely guessed (unawarecorrect). They found that only an enhanced P3 effect with a widely distributed topography was related to subjective awareness. Similarly, Del Cul et al. (2007) also found that only the P3 amplitude distinguished visible and invisible stimuli in subjective reports. Moreover, although the studies using objective measures found VAN to be related to visual awareness, most of them also found P3 or LP (later positive) effects in visual awareness (e.g., Koivisto et al., 2008; Genetti et al., 2009).

Obviously, different measures lead to different findings in the neural correlates of visual awareness, but more importantly, they are based on different theories of consciousness. For example, subjective measures are consistent with the assumption of the higher-order thought (HOT) theory that states that a distinguishable characteristic of a conscious state is that an individual can report the state (Rosenthal, 2005; Lau and Rosenthal, 2011). According to the HOT theory, the conscious status of a mental state is most naturally measured by subjective measures. Meanwhile, the global workspace (GWS) theory takes the dynamic long-distance synchronization as a key feature for conscious condition (Dehaene and Naccache, 2001). According to the GWS theory, objective measures could be satisfied by local processors without information entering the workspace. Information in the workspace would be available to higher order thoughts, so the GWS theory naturally motivates subjective measures. In this respect, the HOT and GWS theory are similar. In contrast, a recent recurrent processing hypothesis states that localized recurrent processing is sufficient for consciousness, and while not entirely obvious theoretically, this approach has been used to motivate objective measures (Lamme, 2010). But most generally, the local recurrent processing approach attempts to move away from behavioral or introspective methods in the measure of consciousness.

The widely distributed P3 or LP correlate of visual awareness is in line with the prediction of the GWS theory, which assumes that visual awareness correlates with later activation in the frontoparietal cortices; the more local posterior VAN correlate of visual awareness is in line with the recurrent processing hypothesis, which proposes that visual awareness correlates with early activation in the visual cortex (Windey et al., 2013; Andersen et al., 2015; Koivisto and Grassini, 2016). In addition, the GWS theory assumes that visual awareness is dichotomous, while the recurrent processing hypothesis assumes that visual awareness is graded (Windey et al., 2013; Andersen et al., 2015; Koivisto and Grassini, 2016). It remains controversial whether visual awareness is correlated with early activation indicated by VAN, as the recurrent process hypothesis theory proposes, or with later activation indicated by P3 or LP, as suggested by the GWS and HOT theories, and whether the neural indicator is correlated with visual awareness in a dichotomous or graded way.

Interestingly, Timmermans et al. (2010) argued that studies supporting the GWS theory generally use complex stimuli (e.g., Del Cul et al., 2007, 2009), whereas studies supporting the recurrent processing hypothesis mostly use simple stimuli (e.g., Fahrenfort et al., 2007). Moreover, Windey et al. (2013) found that simple task produced evidence for graded visual awareness, whereas more complex task produced more dichotomous visual awareness. Thus, the findings about the neural correlates of visual awareness could be influenced by stimulus or task complexity. To the best of our knowledge, no study has adopted complex stimuli such as natural scenes to explore neural correlates of visual awareness. Natural scenes are rather complex, but people

can categorize them quickly and accurately, even with little or no attention (Li et al., 2002; Rousselet et al., 2002; Feifei et al., 2005; Otsuka and Kawaguchi, 2007), and it has been claimed that the integration of scene elements can be achieved without awareness (Mudrik et al., 2011; Mudrik and Koch, 2013), although there is inconsistent evidence for this (Faivre and Koch, 2014; Moors et al., 2016). Moreover, recently, a fMRI study found that natural scene categorization of line-drawings generated similar neural activation as color photographs, although color photographs includes more features such as color and contexts than linedrawings (Walther et al., 2011).

Therefore, to further examine what are the neural correlates of visual awareness, we adopted natural scenes of both color photographs and line-drawings in a backward masking task. In the backward masking task, the stimulus duration was identical, i.e., the physical stimulus was identical, and the interstimulus interval (ISI) between the stimulus and the mask was manipulated. It has been demonstrated that masking has little effect on the feedforward processing of a stimulus from low to high visual areas, but interferes with the recurrent processing of the stimulus from high to low visual areas that is crucial for the emergence of visual awareness (Rolls et al., 1999; Lamme et al., 2002; Lamme, 2010). To measure the status of visual awareness, we used subjective measures because they are motivated by two major psychological approaches to consciousness, that is, higher order and GWS approaches. Specifically, we used the Perceptual Awareness Scale (PAS), in which participants were asked to rate the clarity of their visual experience using a 4-point scale with the elements: no experience, brief glimpse, almost clear image, and clear image (Ramsøy and Overgaard, 2004; Sandberg et al., 2010; see also Dienes and Seth, 2010). The interpretation of PAS depends on its theoretical relation to awareness of the relevant stimulus characteristics. On one view, each PAS rating represents an increase in visual awareness in a graded way (e.g., Ramsøy and Overgaard, 2004). On another view, the distinction between, on the one hand, brief glimpse, in which the subject claims not to have seen anything of relevance, and almost clear image, on the other hand, marks a distinct change in the conscious awareness of information relevant to the task discrimination (Dienes and Seth, 2010). On the first view, a correlate of awareness should vary linearly with the PAS ratings; on the second view, relevant components should vary non-linearly with the PAS ratings. Further, if subjective visual awareness correlates with early activation in the visual cortex, the posterior VAN should vary with the PAS ratings in an appropriate way; or else, if long-distance activation among brain areas is additionally required for the emergence of subjective awareness, a widely distributed P3 or LP would vary with the PAS ratings in an appropriate way.

# MATERIALS AND METHODS

# Participants

Twenty-two college students (11 males and 11 females; mean age = 21.82 years, SD = 2.34 years) participated in this study. All of them had normal or corrected-to-normal vision, and none of them had any history of neurological or psychiatric diseases. This experiment was approved by the committee for the protection of subjects at the Institute of Psychology, Chinese Academy of Sciences. All students gave written informed consent and were paid for their attendance.

# Stimuli and Apparatus

The stimuli were images of six natural scene categories: beaches, city streets, forests, highways, mountains, and offices (see **Figure 1**). Each image had a resolution of 320 pixels × 240 pixels in two versions: color photographs and line-drawings. The linedrawings were produced by trained artists by tracing contours in the color photographs (see Walther et al., 2011). Each category had 76 to 80 different images, for a total of 475. The masks were two white noise images at two different spatial scales: one was generated at the resolutions of 320/240, and the other was generated at the resolutions of 20/15 and then resized to 320/240 pixels. Each mask was also resized to 320 pixels × 240 pixels in two versions: one was in color and the other was in grayscale. The stimulus was presented against a silver gray background on a CRT monitor with a resolution of 1280 pixels × 768 pixels and a refresh rate of 75 Hz. The distance from the participants to the screen was approximately 60 cm, and the horizontal and vertical visual angles were approximately 8.7 and 8.2 degrees, respectively.

# Procedure

Each trial began with a black fixation cross at the center of the screen for 500–950 ms at random. Then, an image was presented for 13 ms, and followed by two masks for 100 ms. Each mask was shown for 50 ms and the sequence of the two masks was randomly assigned. For color photographs, the masks were in color, and for line-drawings, the masks were in grayscale. The ISI between the image and the mask was 0, 13, 26, or 200 ms at random. After the masks, a blank was presented for 500 ms and then six category names appeared on the screen from left to right. Participants were asked to report the category of the presented image by pressing a corresponding key on the keyboard. The keys D, F, G, H, J, and K corresponded to the locations of the six category names. To prevent participants from preparing for their response before the appearance of the category names, the locations of the six category names were randomly assigned for each trial. There was no feedback about the response. After the response, they were asked to rate their experience of the stimulus, i.e., how clearly did they see the image, with four possible options from left to right on the PAS scale [(1) no experience; (2) brief glimpse; (3) almost clear experience; (4) clear experience] by pressing a corresponding key. Participants were told to select the option according to their true feelings and then to press the space key to start the next trial when they were ready. There were approximately 95 trials in each block, in which type of image (color photographs vs. line-drawings), natural scene category, and ISI were counterbalanced. There were at least 30 s of rest between any two blocks. The experiment included 10 blocks, for a total of 950 trials.

# EEG Recording and Analysis

Electroencephalograph (EEG) was recorded with a NeuroScan Synamps amplifier using a custom 64-channel cap. The left mastoid was used as an on-line reference and the algebraic average of left and right mastoids was used as off-line rereference. Two pairs of electrodes placed 1 cm above and below one eye and 1 cm lateral from the outer canthi of both eyes were used to monitor blinks and other eye movements. The EEG signals were amplified by using a band pass of 0.05 to100 Hz, with a sampling rate of 500 Hz. Electrode impedances were kept below 5 k. For the analysis of ERPs, the continuous EEG signals were low pass filtered, with a cutoff frequency at 30 Hz, and then segmented in series of epochs of 800 ms. Each epoch started 100 ms before the stimulus onset. Baseline correction was performed over the 100 ms window before the stimulus presentation. Trials containing voltages exceeding ± 80 µV were rejected. The average rejection rate was only 2.17%.

ERPs for correct trials were averaged separately for the different experience ratings of color photographs and linedrawings. Based on previous studies (for review, see Koivisto and Revonsuo, 2010; Melloni et al., 2011), the mean amplitudes of the ERPs were analyzed for P1 (100–130 ms), N1 (140–200 ms), P2 (150–190 ms), N2 (210–310 ms), VAN (240–300 ms), P3a (370– 470 ms), and LP or P3 (420–570 ms). The time windows for the statistical analysis of the ERPs were determined on the basis of the grand average waves. The mean amplitudes were computed over groups of electrodes representative of the topography of each component. For P1, N1, and VAN, a group of occipital electrodes (CB1, O1, Oz, O2, and CB2) was selected; for P2, N2, and P3a, a group of fronto-central electrodes (F3, Fz, F4, FC3, FCz, FC4, C3, Cz, and C4) was selected; for LP, a group of occipito-parietal electrodes (P3, Pz, P4, PO3, POz, PO4, O1, Oz, and O2) was selected. Each was subjected to repeated two-way ANOVA with the type of image (color photographs vs. line-drawings) and experience ratings (no experience vs. brief glimpse vs. almost clear experience vs. clear experience) as within-subject variables.

#### RESULTS

#### Inferential Strategy

fpsyg-08-00210 February 13, 2017 Time: 14:18 # 5

For all tests p-values are reported; in addition, for all one degree of freedom tests, Bayes factors, B, are reported. Bayes factors (B) were used to assess strength of evidence for H1 relative to H0 (Wagenmakers et al., 2015). Unlike null-hypothesis significance testing, Bayes factors have the advantage of distinguishing sensitive evidence for H0 from not much evidence at all (relative to the range of effects that could plausibly be obtained on H1). A B of above 3 indicates evidence for the alternative hypothesis and below 1/3 evidence for the null hypothesis. Bs between 3 and 1/3 indicate data insensitivity in distinguishing H0 and H1 (see Dienes, 2014; cf Jeffreys, 1939). Thus we will report that there was no effect only when B < 1/3, that is, when there is evidence for H0 over H1. Here, BN(0,x) refers to a Bayes factor in which the predictions of H1 were modeled as normal distribution with an SD of x (see Dienes, 2014), where x scales the size of effect that could be expected. If a rough maximum effect can be determined, x can be set as half that value (Dienes, 2014).

For proportions of different PAS ratings, with four ratings the average proportion was 0.25, so we report BN(0,0.25) . For accuracy of classification, as the range from baseline (i.e., chance level 0.17) to 1 was 0.83, a difference of 0.83 is the maximum difference that could be expected between any two conditions. Thus, we took half of 0.83 as the SD and report BN(0,0.42) . For ERP results, we took half of the roughly average difference between maximal and minimal amplitude of the ERP component over color photographs and line-drawings as x for each component. Note that the difference between maximal and minimal amplitude was never itself tested, so there is no double counting (Jeffreys, 1939). As positive trend coefficients summed to 1 and negative to −1, the expected size of linear trends is the same as a simple difference. Thus, for trend contrasts, H1 was modeled as a normal with an SD equal to half of the roughly average difference between maximal and minimal amplitudes of the ERP component, as just mentioned. With these models of H1, it so happened that whenever p < 0.05, then B > 3, and vice versa. There is no guarantee of this correspondence (Lindley, 1957), but in the current case significance testing and Bayesian analyses agreed when indicating that there was an effect.

#### Behavioral Results

As in previous studies (e.g., Ramsøy and Overgaard, 2004; Del Cul et al., 2007; Railo et al., 2015), we treated the experience rating as a within-subject factor. **Figure 2** shows the proportion and accuracy for each experience rating for both types of images. Repeated ANOVA on proportions, with experience ratings and type of image as within-subject variables, revealed a significant effect of experience ratings, F(1.62,33.97) = 4.34, p < 0.01, η 2 <sup>p</sup> = 0.17, and a significant experience ratings by type of image interaction, F(3,63) = 22.55, p < 0.001, η 2 <sup>p</sup> = 0.52. People reported having a "brief glimpse" less often for color photographs than for line-drawings, t(21) = −7.37, p < 0.001, dz = 1.61, BN(0,0.25) = 7.13 × 1010; but reported having a "clear experience" more often for color photographs than for line-drawings, t(21) = 6.69, p < 0.001, dz = 1.46, BN(0,0.25) = 9.07 × 10<sup>8</sup> . There were no differences between color photographs and linedrawings in the proportion of time they reported having "no experience," t(21) = −0.03, p = 0.98, BN(0,0.25) = 0.08, nor in the proportion of "almost clear experience" reports, t(21) = −1.43, p = 0.17, BN(0,0.25) = 0.32. Overall, the above results provided substantial evidence that people had subjective experiences with more felt clarity of color photographs than of line-drawings.

Repeated ANOVA on accuracy, with experience ratings and type of image as within-subject variables, revealed only a significant effect of experience ratings, F(1.83,38.86) = 299.49, p < 0.001, η 2 <sup>p</sup> = 0.93. For both types of images, the accuracy was higher for a "clear image" than for an "almost clear image," t(21) = 4.81, p < 0.001, dz = 1.05, BN(0,0.42) = 3.66 × 10<sup>20</sup> , higher for an "almost clear image" than for a "brief glimpse," t(21) = 13.05, p < 0.001, dz = 2.85, BN(0,0.42) = 1.58 × 10<sup>35</sup> , and higher for a "brief glimpse" than for "no experience," t(21) = 9.90, p < 0.001, dz = 2.16, BN(0,0.42) = 1.47 × 10<sup>4</sup> . There was no effect of type of image, F(1,21) = 0.09, p = 0.77, η 2 <sup>p</sup> = 0.004, BN(0,0.42) = 0.04. The two-way interaction did not reach significance, F(3,63) = 2.17, p = 0.10, η 2 <sup>p</sup> = 0.09. For both color photographs and line-drawings, the accuracy was above chance (0.17) for "no experience," t(21) = 4.11, p < 0.001, dz = 0.88, BN(0,0.42) = 2.85 × 10<sup>58</sup> , t(21) = 3.56, p < 0.01, dz = 0.76, BN(0,0.42) = 3.51 × 1026. That is, people could correctly classify natural scenes even when they reported that they did not see them at all.

#### ERP Results

Inspection of ERPs revealed seven major components: P1, N1, P2, N2, VAN, P3a, and LP (or P3). **Figure 3** shows the ERP data for each condition. **Figure 4** shows the mean amplitudes of each component for each condition. To evaluate which components were correlated with subjective awareness, repeated two-way ANOVA with experience ratings (no experience vs. brief glimpse vs. almost clear experience vs. clear experience) and type of images (color photographs vs. line-drawings) as within-subject variables was conducted for the mean amplitude of each component. Moreover, as "brief glimpse" involves some visual experience, but not of anything relevant to making the task discrimination, the difference between "brief glimpse" and "almost clear image" is when perceptual contents go from largely unconscious and to largely conscious. Even on a graded consciousness view (Ramsøy and Overgaard, 2004), this particular contrast represents a distinction in amount of conscious awareness. Thus, to further examine whether each component was a real marker of the conscious status of knowledge of a stimulus, the planned comparison between "brief glimpse" and "almost clear image" were conducted for the mean amplitude of each component.

**Table 1** summarizes significant results of the ANOVAs. As observed from **Table 1**, for posterior P1 amplitudes, the effect of

the experience ratings and interaction did not reach significance (ps > 0.31). The planned comparison revealed that there was evidence for no difference between "brief glimpse" and "almost clear experience" for color photographs, t(21) = −1.24, p = 0.23, dz = 0.26, BN(0,3) = 0.26, and marginal evidence that there was no difference for line-drawings, t(21) = 1.46, p = 0.16, dz = 0.31, BN(0,3) = 0.35. That is, the posterior P1 amplitude did not always relate to subjective awareness.

For posterior N1 and anterior P2 amplitudes, the twoway ANOVAs revealed significant experience rating effects which were modulated by the type of image. Further analysis revealed that for color photographs only, the amplitudes of posterior N1 and anterior P2 significantly varied with visual awareness, F(1.86,39.08) = 23.02, p < 0.001, η 2 <sup>p</sup> = 0.52, F(2.09,43.81) = 10.54, p < 0.001, η 2 <sup>p</sup> = 0.33, but did not for line-drawings (ps > 0.12). For N1, the planned comparison revealed that "brief glimpse" differed from "almost clear experience" for color photographs, t(21) = −5.85, p < 0.001, dz = 1.25, BN(0,3) = 4.46 × 10<sup>6</sup> , but not for linedrawings, t(21) = −0.81, p = 0.43, dz = 0.17, BN(0,3) = 0.18. For P2, the planned comparison revealed that there was no evidence for whether or not "brief glimpse" differed from "almost clear experience" for color photographs, t(21) = 1.14, p = 0.27, dz = 0.24, BN(0,4) = 0.47, but there was substantial evidence for no difference for line-drawings, t(21) = −0.07, p = 0.94, dz = 0.01, BN(0,4) = 0.08. Thus, the results indicated that the posterior N1 and anterior P2 amplitudes did not always correlate with subjective awareness.

For the later components N2, VAN, P3a, and LP, the twoway ANOVAs revealed significant experience rating effects (all ps < 0.01), which were not significantly influenced by the type of images (all ps > 0.19). For N2, the planned comparison revealed that "brief glimpse" differed from "almost clear experience" for both color photographs, t(21) = −4.03, p = 0.001, dz = 0.86, BN(0,4) = 8.20 × 10<sup>2</sup> , and line-drawings, t(21) = −3.42, p < 0.01, dz = 0.73, BN(0,4) = 69.39. For VAN, "brief glimpse" differed from "almost clear experience" for both color photographs, t(21) = 3.12, p < 0.01, dz = 0.67, BN(0,4) = 23.89, and line-drawings, t(21) = 4.06, p < 0.001, dz = 0.87, BN(0,4) = 2.35 × 10<sup>3</sup> . However, for P3a, there was not much evidence for the difference between "brief glimpse" and "almost clear experience" either way for either color photographs, t(21) = 0.84, p = 0.41, dz = 0.18, BN(0,4) = 0.35, or linedrawings, t(21) = 1.08, p = 0.29, dz = 0.23, BN(0,4) = 2.59. Finally, for LP, there was a difference between "brief glimpse" and "almost clear experience" for both color photographs, t(21) = 3.95, p = 0.001, dz = 0.84, BN(0,3) = 6.60 × 10<sup>2</sup> , and line-drawings, t(21) = 2.11, p < 0.05, dz = 0.45, BN(0,3) = 3.24. That is, the results suggested that the N2, VAN, and LP amplitudes could be related to the emergence of subjective awareness, while the evidence for P3a amplitude marking a change in subjective awareness was insensitive.

To further examine how the later components correlated with the subjective awareness, we used "no experience," "brief glimpse," and "almost clear experience" to calculate the linear trend coefficient for each of the later component (i.e., the difference between the first and last ratings). For N2, the linear trend coefficients were above zero for both color photographs and line-drawings, t(21) = 5.81, p < 0.001, dz = 1.24, BN(0,2) = 3.84 × 10<sup>5</sup> , t(21) = 3.16, p < 0.01, dz = 0.67, BN(0,2) = 27.03. For VAN, the linear trend coefficients were below zero for both types of images, t(21) = −4.16, p < 0.001, dz = 0.89, BN(0,2) = 1.13 × 10<sup>3</sup> , t(21) = −3.70, p = 0.001, dz = 0.79, BN(0,2) = 3.815 × 10<sup>2</sup> . For LP, there was not much evidence either way for whether or not the linear trend coefficients were different from zero for both types of images, t(21) = 1.64, p = 0.12, dz = 0.35, BN(0,2) = 2.56, t(21) = 1.31, p = 0.20, dz = 0.28, BN(0,2) = 1.58. Moreover, for N2, "no experience" differed from "brief glimpse" for both color photographs, t(21) = −2.80, p < 0.05, dz = 0.60, BN(0,4) = 15.88, and line-drawings, t(21) = −2.13, p < 0.05, dz = 0.45, BN(0,4) = 5.14. For VAN, "no experience" also differed from "brief glimpse" for both color photographs, t(21) = 3.90, p = 0.001, dz = 0.83, BN(0,3) = 5.84 × 10<sup>2</sup> , and line-drawings: t(21) = 2.43, p < 0.05, dz = 0.52, BN(0,3) = 6.08. But for LP, there was marginal evidence for no difference between "no experience" and "brief glimpse" for color photographs, t(21) = 0.80, p = 0.43, dz = 0.17, BN(0,3) = 0.37, and evidence for no difference between "no experience" and "brief glimpse" for line-drawings,

FIGURE 3 | Grand-average ERPs for each experience rating for color photographs and line-drawings. (A) Grand-average ERPs of correct trials at electrodes FCz, Pz, and Oz for each experience rating for color photographs and line-drawings. (B) ERP differences between every two adjacent experience ratings for color photographs and line-drawings. (C) The scalp topography of the P1, N1, P2, N2, VAN, P3a, and LP, trials of the relatively higher experience ratings minus trials of the adjacent lower experience ratings separately for color photographs and line-drawings.

TABLE 1 | Significant results of the two-way repeated ANOVAs performed over the amplitude of each component, considering experience ratings and type of images.


In each ANOVA, we report F-values with significance and η 2 p . <sup>∗</sup>p < 0.05, ∗∗p < 0.01, ∗∗∗p < 0.001.

t(21) = 0.26, p = 0.80, dz = 0.06, BN(0,3) = 0.22. The results suggested that the N2 and VAN could be related to PAS in a linear way, whereas the LP correlated with PAS in a non-linear fashion.

To explore the relationship between experience ratings and ERPs controlling for ISIs, we firstly described the distribution of ISIs cross correct trials for each experience rating for color photographs and line-drawings (see **Figure 5**). As can be seen from **Figure 5**, the number of correct trials for some ISI and experience rating combinations was rather small. As ERP effects usually require a large number of trials to measure them accurately in each condition (Luck, 2005, p. 23), "no experience" and "brief glimpse" were collapsed to form an "unconscious" category, while "almost clear experience" and "clear experience" were collapsed together for a "conscious" category, as per Melloni et al. (2011). The data from three participants were excluded because the number of trials for at least one condition was zero for them. **Figure 6** shows the ERP data for conscious and unconscious categories of each ISI for color photographs and line-drawings. An ANOVA with type of image (color photographs vs. line-drawings), experience ratings (unconscious vs. conscious), and ISI (0, 13, 26, and 200 ms) was conducted separately for amplitudes of N2, VAN, and LP. For N2, there was an experience rating effect, F(1,18) = 28.91, p < 0.001, η 2 <sup>p</sup> = 0.62,

fpsyg-08-00210 February 13, 2017 Time: 14:18 # 8

BN(0,4) = 2.15 × 10<sup>5</sup> , a significant ISI effect, F(3,54) = 20.39, p < 0.001, η 2 <sup>p</sup> = 0.53. For VAN, there was a type of image effect, F(1,18) = 26.89, p < 0.001, η 2 <sup>p</sup> = 0.60, BN(0,4) = 8.68 × 10<sup>4</sup> , a significant ISI effect, F(1.65,29.63) = 37.53, p < 0.001, η 2 <sup>p</sup> = 0.68, and a significant type of image by ISI interaction, F(2.34,42.20) = 8.71, p < 0.001, η 2 <sup>p</sup> = 0.33. For LP, there was a type of image effect, F(1,18) = 7.50, p < 0.05, η 2 <sup>p</sup> = 0.29, BN(0,3) = 8.30, an experience rating effect, F(1,18) = 17.67, p = 0.001, η 2 <sup>p</sup> = 0.50, BN(0,3) = 9.03 × 10<sup>2</sup> , and a significant ISI effect, F(2.21,39.69) = 26.12, p < 0.001, η 2 <sup>p</sup> = 0.59. The main effect of experience ratings on amplitudes indicated that the N2 and LP were each related to subjective awareness even controlling for ISI.

# DISCUSSION

Our behavioral results showed that people reported having more "clear experience" but less "brief glimpse" for color photographs than for line-drawings, suggesting that people had more experienced clarity for color photographs than for line-drawings. Importantly, there was no difference for accuracy between color photographs and line-drawings for each experience rating and the accuracy for both types of image gradually increased with rating, consistent with the experience ratings reflecting visibility. However, the accuracy for both types of images was above chance for "no experience," providing evidence for subliminal perception with an elaborated subjective scale. This may seem inconsistent with the finding of Ramsøy and Overgaard (2004), in that they found the accuracy for the stimulus form was not significantly better than chance for "no experience". However, a non-significant result is not in itself evidence for the null hypothesis.

The ERP results showed that the posterior P1 amplitude did not change with the PAS ratings, confirming previous findings that the early P1 component is not related to visual awareness (Del Cul et al., 2007; Koivisto et al., 2008; Melloni et al., 2011). Further, the posterior N1 and anterior P2 components did not vary with the PAS ratings for linedrawings at least. The N1 amplitude for color photographs decreased with increasing subjective ratings, consistent with previous results showing that the N1 amplitude decreased when visibility increased (Melloni et al., 2011). However, as the N1 component for line-drawings did not vary with subjective ratings, the N1 component cannot be a general indicator of subjective awareness. Similarly, the P2 amplitude for color photographs decreased with increasing subjective ratings, consistent with previous findings that the P2 amplitude decreased as visibility increased (Melloni et al., 2011). Nonetheless, as the P2 component for line-drawings did not change with subjective ratings, indicating that the P2 component is also not always related to subjective awareness. It has been found that N1 and P2 are associated with feature detection or integration (Hillyard and Münte, 1984; Luck and Hillyard, 1994). The features were more complex in color photographs than in line-drawings. Thus, the fact that N1 and P2 correlated with experience ratings for color photographs rather than for linedrawings may be because that the feature analysis varied with the experience ratings for color photographs but not for linedrawings.

Importantly, we found that the posterior VAN did vary with the PAS ratings linearly for both color photographs and line-drawings, which is consistent with previous findings that the posterior VAN correlated with visual awareness, as the recurrent process hypothesis theory proposes (Koivisto and Revonsuo, 2003; Koivisto et al., 2008; Genetti et al., 2009). Indeed, Koivisto and Revonsuo (2010) reviewed ERP studies on visual awareness and suggested that VAN is the most consistently observed feature across different studies. However, we found that there were significant differences for the VAN amplitudes between "no experience" and "brief glimpse" and between "brief glimpse" and "almost clear experience" for both color photographs and line-drawings. On the view that "no experience" and "brief glimpse" can be collapsed together as reflecting lack of conscious awareness of visual features relevant to the task discrimination (Dienes and Seth, 2010), the VAN amplitude increase would not be a good indicator of the emergence of subjective awareness.

Moreover, our results showed that the anterior N2 gradually decreased with the PAS ratings. The anterior N2 has been observed to correlate with conscious awareness in other cognitive tasks. For example, Eimer et al. (1996) found that the anterior

N2 enhancement for deviant stimuli compared with standard stimuli might be regarded as an indicator of the amount of explicit (conscious) knowledge in a serial reaction task. Schankin and Wascher (2007) also found that the anterior N2 were smaller for detected or aware changes than for undetected or unaware changes in Experiment 1 in a change detection task. Nonetheless, Schankin and Wascher (2007) did not observe the N2 effect when the influence of working memory was eliminated in Experiment 2, and a few studies have found that the N2 amplitudes were not significantly affected by subjectively awareness of the targets when they were simple lines or shapes (e.g., Koivisto and Revonsuo, 2003; Lamy et al., 2009). As N2 is related to an actively attended mismatch between a stimulus and a mental template (Folstein and Van Petten, 2008), the inconsistent findings may be due to the requirement difference on memory retrieval in different tasks.

Unlike VAN and N2 amplitudes, the P3a and LP (or P3) amplitudes seemed to change non-linearly with subjective ratings. For color photographs, the P3a amplitude was not larger for "brief glimpse" than for "almost clear experience," suggesting that the P3a was not sensitive to subjective awareness. As P3a reflects top-down monitoring by frontal attention mechanisms engaged to evaluate incoming stimuli (Polich, 2007), the P3a amplitude might be related to evaluating difficulty more than subjective awareness. Importantly, the LP amplitude was larger for "almost clear experience" than for "brief glimpse," but there was no difference between "brief glimpse" and "no experience," indicating that the LP amplitude is an indicator of the emergence of subjective awareness, assuming a non-linear relation between PAS and relevant awareness. The results indicated that visual awareness is correlated with later activation indicated by P3 or LP, as suggested by the GWS and HOT theories (Koivisto et al., 2008; Genetti et al., 2009).

The different patterns of VAN and LP in our findings are consistent with different stages or types of subjective awareness. It has been noted that the perception of masked displays might involve two stages: during the early stage, i.e., the first 200 to 300 ms from the stimulus onset, the brain responses detectable with ERPs increase linearly with the stimulus energy or duration, whereas in the second stage, after approximately 300 ms from the stimulus onset, the brain activity detectable with ERPs is characterized by a non-linear relation to stimulus magnitudes (see Kouider et al., 2013). Koivisto and Revonsuo (2003)suggested that the VAN is an electrophysiological correlate of phenomenal visual awareness, the subjective experience of seeing the stimulus, whereas the later LP reflects access visual awareness, the conscious evaluation of the stimulus for decisionmaking (Block, 1995, 1996). Lamme (2010) also argued that all the neural ingredients that seem to matter for the visual phenomenality are present in the localized recurrent processes (indicated by the VAN), and the widespread recurrent processes (indicated by the LP) allow to have only reportable conscious visual information. Therefore, the VAN may change linearly with the quality of visual perception, which may reflect how first order visual quality, which some may call visual phenomenal awareness, is graded or continuous; whereas the LP varies nonlinearly with PAS, and may reflect awareness of being in a relevant mental state (and therefore the state being conscious by higher order and GWS theories).

Finally, we should note some limitations in the present study. First, as different experience ratings included different proportion of trials with same ISIs, there is an issue of the influence of ISI on visibility. Ideally, the physical stimulation should remain constant when compared trials with different experience ratings. However, as a sufficient amount of responses for each experience rating was necessary, different contrasts between the target and the background (e.g., Andersen et al., 2015) or different stimulus durations (e.g., Koivisto and Grassini, 2016) were often used. In the present study, the stimulus duration was same for all experience ratings, while the ISI between the target and the mask was varied. Previous studies revealed that backward masking does not influence the feed-forward processing but interrupts the recurrent processing between high and low cortical areas which is crucial for the emergence of visual awareness (Lamme et al., 2002). Importantly, when "no experience" and "brief glimpse" were collapsed to form an "unconscious" category and "almost clear experience" and "clear experience" to form a "conscious" category, we also found that N2 and LP were each related to subjective awareness even controlling for ISI. However, further studies could still usefully explore how VAN and LP vary with experience ratings when the ISI remains unchanged so the

#### REFERENCES


control is not just statistical. Second, although we attempted to find neural correlates of subjective awareness, we could not assume that participants always provided accurate descriptions of their experience in their subjective reports (Dehaene and Naccache, 2001). Fortunately, we found that there were no significant accuracy differences between color photographs and line-drawings for each experience rating, suggesting that the rating was relatively accurate in the present study. Our findings are the first to show that the VAN amplitude increased linearly with the PAS ratings whereas the LP amplitude increased nonlinearly with PAS ratings, which is helpful to reconcile the apparently contradictory theories and to resolve the current debate on the neural correlates of visual awareness.

## ETHICS STATEMENT

This experiment was approved by the committee for the protection of subjects at the Institute of Psychology, Chinese Academy of Sciences. All students gave written informed consent and were paid for their attendance.

# AUTHOR CONTRIBUTIONS

QF, Y-JL, WC, and XF designed the experiment. QF, Y-JL, and WC prepared materials and performed the experiment. QF, ZD, and JW analyzed the data, and QF, ZD, JW, and XF wrote the paper.

# FUNDING

This work was supported by the Natural Science Foundation of China (31270024, 61521002), the National Natural Science Foundation of China and the German Research Foundation (NSFC 61621136008/DFG TRR-169), and the National Key Research and Development Plan (2016YFB1001200).

# ACKNOWLEDGMENTS

The authors thank Prof. Dirk B. Walther, the Department of Psychology, the Ohio State University, for providing us with the color photographs and line drawings first used in Walther et al. (2011) and thank Prof. Kan Zhang, the Institute of Psychology, Chinese Academy of Sciences, for allowing us to use the NeuroScan equipment of his lab.


science," in Psychological Science Under Scrutiny: Recent Challenges and Proposed Solutions, eds S. O. Lilienfeld and I. Waldman (Hoboken, NJ: John Wiley and Sons).


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Fu, Liu, Dienes, Wu, Chen and Fu. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Relative Spatial Frequency Processing Drives Hemispheric Asymmetry in Conscious Awareness

Elise A. Piazza<sup>1</sup> \* and Michael A. Silver2,3,4

<sup>1</sup> Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA, <sup>2</sup> Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, CA, USA, <sup>3</sup> School of Optometry, University of California, Berkeley, Berkeley, CA, USA, <sup>4</sup> Vision Science Graduate Group, University of California, Berkeley, Berkeley, CA, USA

Visual stimuli with different spatial frequencies (SFs) are processed asymmetrically in the two cerebral hemispheres. Specifically, low SFs are processed relatively more efficiently in the right hemisphere than the left hemisphere, whereas high SFs show the opposite pattern. In this study, we ask whether these differences between the two hemispheres reflect a low-level division that is based on absolute SF values or a flexible comparison of the SFs in the visual environment at any given time. In a recent study, we showed that conscious awareness of SF information (i.e., visual perceptual selection from multiple SFs simultaneously present in the environment) differs between the two hemispheres. Building upon that result, here we employed binocular rivalry to test whether this hemispheric asymmetry is due to absolute or relative SF processing. In each trial, participants viewed a pair of rivalrous orthogonal gratings of different SFs, presented either to the left or right of central fixation, and continuously reported which grating they perceived. We found that the hemispheric asymmetry in perception is significantly influenced by relative processing of the SFs of the simultaneously presented stimuli. For example, when a medium SF grating and a higher SF grating were presented as a rivalry pair, subjects were more likely to report that they initially perceived the medium SF grating when the rivalry pair was presented in the left visual hemifield (right hemisphere), compared to the right hemifield. However, this same medium SF grating, when it was paired in rivalry with a lower SF grating, was more likely to be perceptually selected when it was in the right visual hemifield (left hemisphere). Thus, the visual system's classification of a given SF as "low" or "high" (and therefore, which hemisphere preferentially processes that SF) depends on the other SFs that are present, demonstrating that relative SF processing contributes to hemispheric differences in visual perceptual selection.

Keywords: conscious awareness, binocular rivalry, perceptual selection, spatial frequency, hemispheric asymmetry

# INTRODUCTION

Although we may often be unaware of its impact on our everyday experiences, spatial frequency (SF) information is perceived and processed differently in the left visual field (LVF) and right visual field (RVF), due to differences in perceptual specialization between the two hemispheres (Sergent, 1982; Kitterle et al., 1990; Ivry and Robertson, 1998; Piazza and Silver, 2014). Within a stimulus

#### Edited by:

Guido Hesselmann, Charité – Universitätsmedizin Berlin, Germany

#### Reviewed by:

Christian Valuch, University of Göttingen, Germany Surya Gayet, Utrecht University, Netherlands

> \*Correspondence: Elise A. Piazza elise.piazza@gmail.com

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 08 January 2017 Accepted: 27 March 2017 Published: 19 April 2017

#### Citation:

Piazza EA and Silver MA (2017) Relative Spatial Frequency Processing Drives Hemispheric Asymmetry in Conscious Awareness. Front. Psychol. 8:559. doi: 10.3389/fpsyg.2017.00559

set, identification and discrimination of low SFs tend to be faster and more accurate for stimuli presented in the LVF, whereas high SFs are more quickly and accurately processed in the RVF. This asymmetry has been observed for both sinusoidal gratings (Christman et al., 1991; Hellige, 1993; Christman, 1997) and SF-filtered natural scenes (Peyrin et al., 2003).

Physiologically, fMRI studies show that areas in the left hemisphere respond preferentially to high SF compared with low SF stimuli, whereas the right hemisphere shows the opposite pattern (Peyrin et al., 2004; Musel et al., 2013). Additionally, EEG responses are larger in the left compared with the right hemisphere for high SF stimuli and larger in the right than the left hemisphere for low SFs (Martínez et al., 2001). Moreover, directing attention to one of two SF components of a grating while preparing to perform either a local- or global-level discrimination of a subsequently presented Navon stimulus differentially modulates the amplitude of alpha-band EEG signals in the two hemispheres (Flevaris et al., 2011). Here, we asked whether the differential filtering of a given SF in the two hemispheres for conscious awareness depends on relative processing of the set of available SFs in the environment at any given time.

Most previous behavioral studies of hemispheric asymmetries in SF processing have relied primarily upon measures of reaction times (RTs) to single stimuli that were briefly flashed in either the LVF or RVF. For example, Kitterle and Selig (1991) found that RTs for SF discrimination of two successively presented sinusoidal gratings were faster for lower SF gratings (1–2 cycles/degree, or cpd) in the LVF and for higher SF gratings (4–12 cpd) in the RVF. One study (Kitterle et al., 1992) used gratings with multiple SF components (a low fundamental frequency and higher harmonics) to compare selective processing of these components in the LVF versus RVF. However, in this study, participants were required to make a single perceptual judgment based on a particular SF component (either "Are the bars wide or narrow?" (low SF) or "Are the bars sharp or fuzzy?" (high SF)), so simultaneous perceptual processing of multiple SF components was never assessed within a given trial.

In a recent study (Piazza and Silver, 2014), we investigated the effects of hemispheric asymmetry on conscious visual representations by measuring perceptual selection from multiple SFs simultaneously present in the environment. Perceptual selection is the process of determining which of multiple possible percepts will be dominant, or consciously perceived, when visual input is consistent with multiple interpretations (e.g., the Necker cube). We used binocular rivalry, a bistable phenomenon in which two incompatible images are presented separately to the two eyes at overlapping retinal locations, resulting in perceptual alternation between the two images, even though the visual stimuli remain constant (Blake and Logothetis, 2002).

In contrast to the previous work described above, our use of binocular rivalry provided a number of advantages for studying hemispheric asymmetries in perceptual selection of SFs. First, binocular rivalry allows a direct measure of the subjective content of dynamic perceptual experience (compared to RT, which merely measures the speed of processing of a particular stimulus). Furthermore, binocular rivalry reflects conscious awareness of one of two competing images at any given moment, without the need to explicitly direct the subject's attention to a particular SF component, as in previous studies (e.g., Kitterle et al., 1992). In addition, binocular rivalry involves competition between multiple possible perceptual interpretations to resolve ambiguity in the visual inputs, a ubiquitous real-world problem that the visual system constantly faces.

In our previous study, we presented two orthogonal gratings with distinct SFs (1 and 3 cpd) to the same retinal location in the two eyes in either the left or right hemifield. We found that the lower SF grating was perceived more often when it was presented in the left hemifield (right hemisphere), whereas the higher SF showed the opposite pattern (Piazza and Silver, 2014). This result raised the intriguing question of how the visual system assigns SFs to the two hemispheres.

One possibility is that there is an absolute threshold, above which SFs are processed more efficiently by the left hemisphere than the right hemisphere, and below which they are perceived more efficiently by the right hemisphere. This would be consistent with the existence of filters in the visual system that are tuned for particular SFs (De Valois et al., 1982). Alternatively, the hemispheric preference for each SF could be evaluated relative to the other SFs that are simultaneously present in the visual scene. This would suggest the existence of a classification system for SF processing that is updated based on the set of SFs in the visual environment at any given time.

For example, given a natural scene containing predominantly high SFs (e.g., a forest), relative SF processing would result in the lower SFs in the scene (trunks and branches, as opposed to leaves) being processed more efficiently in the left hemifield/right hemisphere. However, those same trunks and branches would represent relatively high SFs and would be processed better in the right hemifield/left hemisphere in a scene with mainly low SFs (e.g., a landscape).

In addition, relative processing of SFs could allow hemispheric specialization for different features to be invariant over a range of distances from the viewed object. For example, the right hemisphere's preferential involvement in emotional recognition of faces, driven by relatively lower SFs, and the left hemisphere's preference for face identity information, driven by relatively higher SFs (Vuilleumier et al., 2003), could be maintained for both near and far faces. A hemispheric filtering mechanism based solely on absolute SF would not allow such flexible scaling because both emotion and identity information could be low or high SF, depending on viewing distance.

Using static gratings containing multiple SF components, Christman et al. (1991) found that a SF of 2 cpd was differentially processed in the two hemispheres, depending on whether it was relatively higher or relatively lower than other simultaneously presented SFs. However, this study relied on RTs to presentation of gratings, and not a direct measure of dynamic perceptual experience, to investigate hemispheric asymmetry. Moreover, the task (subjects indicated whether the 2 cpd component was present or absent) may have been influenced by non-perceptual factors (e.g., response criterion bias). Thus, the role of relative SF processing in hemispheric differences in perceptual selection of SFs from the environment is unknown.

In the present study, we investigated the role of relative SF processing in hemispheric asymmetries in visual perception. To do this, we expanded the range of SFs that were competing with each other for conscious awareness in binocular rivalry, such that a given grating was the relatively higher SF in some trials but the relatively lower SF in other trials. We found that the hemispheric preferences for a given SF differ as a function of whether it is the relatively low versus high SF in a rivalry pair. Our results demonstrate a novel influence of relative processing on visual perception, in which flexible selection of SFs in the environment results in preferential and asymmetric processing in the two hemispheres.

# MATERIALS AND METHODS

## Participants

Fifteen right-handed participants (aged 19–38, 10 women) completed this study. All participants provided written informed consent in accordance with the Declaration of Helsinki, and all experimental protocols were approved by the Committee for the Protection of Human Subjects at the University of California, Berkeley. Each participant completed a 1-h session composed of two blocks. We collected data from 20 participants but excluded five participants' data sets from analysis. Of the excluded participants, two were missing substantial portions of data due to incorrect response key mapping, one responded incorrectly on more than 25% of the catch trials (see Procedure), and two experienced no perceptual alternations of rivalrous stimuli on a majority (>75%) of trials (perhaps due to strong eye dominance; Dieter et al., 2017). While our study is focused on initial perceptual selection and not on perceptual alternations in binocular rivalry per se, perceptual alternations are a defining characteristic of binocular rivalry, and it was unclear that these two subjects experienced binocular rivalry in a typical manner.

# Visual Stimuli

Binocular rivalry stimuli were generated on a Macintosh PowerPC using MATLAB and Psychophysics Toolbox (Kleiner et al., 2007) and were displayed on a gamma-corrected NEC MultiSync FE992 CRT monitor with a refresh rate of 60 Hz at a viewing distance of 100 cm. Participants viewed all stimuli through a mirror stereoscope with their heads stabilized by a chin rest. Stimuli were monochromatic circular patches of sine wave grating 1.8◦ in diameter that were surrounded by a black annulus with a diameter of 2.6◦ and a thickness of 0.2◦ (**Figure 1**). Binocular presentation of this annulus allowed it to serve as a vergence cue to stabilize eye position. The stimuli were presented on the horizontal meridian, centered at 3.5◦ eccentricity either to the left or right of a black central fixation cross. Because the fixation crosses were in the same location on the screen in both hemifield conditions (**Figure 1**, top), participants' eye position, relative to the head, was the same in both conditions. All gratings were presented at 100% contrast and had the same mean luminance as the neutral gray background (59 cd/m<sup>2</sup> ).

In all trials except for the catch trials (see Procedure), the two sinusoidal gratings had different SFs, corresponding to the values depicted in one of the columns of **Figure 2A**. Specifically, each pair of SFs was one of the following: 0.75/1.5 cpd, 1.5/3 cpd, or 3/6 cpd. We refer to the relatively lower SF in each pair as the LSF and the relatively higher SF as HSF. The two gratings in a pair were orthogonal, with ±45◦ orientations relative to vertical. The SF and orientation of the grating presented to each eye were fully counterbalanced and randomly selected across trials.

## Procedure

Before starting the experiment, each participant adjusted the stereoscope by rotating its mirrors until the two eyes' images (**Figure 1**, bottom, with the orthogonal gratings replaced by identical figures in both eyes for this adjustment phase) were fused, allowing the participant to perceive a single cross and annulus with binocular viewing. All participants completed five practice trials in each hemifield condition before starting the experiment to ensure that they were using the correct response keys and that the stereoscope was properly aligned.

In each trial, the static gratings, fixation cross, and annuli (**Figure 1**) were presented continuously for 10 s, with a 1500-ms blank interval (consisting of only the fixation cross and annuli) between trials. A brief (250 ms) pure tone auditory cue was presented immediately before the onset of the grating stimuli to signal the beginning of each trial. Throughout each trial, participants used one of two keys to indicate their percept: either a grating tilted to the left or a grating tilted to the right. We asked participants to report tilt, a feature orthogonal to the dimension of interest (SF), to reduce the likelihood of response bias. Participants were instructed to continuously press a key with their right hand for as long as the corresponding percept was dominant and to not press any key for ambiguous percepts. The experiment was separated into two blocked conditions: left hemifield and right hemifield, the order of which was counterbalanced across participants. Each participant completed 96 trials and 8 catch trials per hemifield condition.

Catch trials, in which the gratings presented to the two eyes were identical in every way (e.g., both 1.5 cpd and ±45◦ ), were randomly interleaved with the normal rivalry trials throughout the experiment to determine whether participants were accurately reporting their percept and using the correct response key mapping. In these catch trials, gratings were presented statically for the entire duration of each trial. As in the experimental trials, participants were asked to continuously report what they saw throughout the trial, and responses were measured over the entire trial duration. Participants who responded incorrectly (i.e., made at least one key press corresponding to the tilt that was orthogonal to that of the presented gratings) in more than 25% of the catch trials were excluded.

# RESULTS

Immediately after presentation of a pair of stimuli in binocular rivalry, participants often experience an ambiguous percept (consisting of a patchwork or mixture of the two images), followed by a perceptual alternation between two distinct images.

We defined the initial response on each trial as the first key press by the subject, as this indicates the subject's first percept that clearly corresponded to one of the two orthogonal gratings. On average, this initial response occurred 1.6 s after the start of the trial. Only 3% of the total trials across participants contained any response in the first 300 ms, suggesting that automatic responses occurred very infrequently, if at all.

For each subject, we measured the proportion of initial responses corresponding to either the LSF (relatively lower) or HSF (relatively higher) grating for each of the three SF pairs (**Figure 2A**). Initial responses were recorded for both visual hemifields, so every subject contributed three pairs of scores for the LVF and three pairs for the RVF. To evaluate hemispheric differences in SF processing, we subtracted the proportion of initial responses in the RVF (left hemisphere, or LH) from that in the LVF (right hemisphere, or RH) for each SF (**Figure 2B**). Because the two plotted difference values for a given SF pair are always identical in magnitude but opposite in sign (due to the complementary nature of responses, which always corresponded to either the LSF or HSF stimulus), we only analyzed the LSF responses.

A one-way ANOVA indicated that there was no significant main effect of SF pair on the magnitude of the hemispheric difference score [F(2,14) = 1.13, p = 0.34]. The assumption of equality of variances was met for this ANOVA and for all other ANOVAs reported in this paper (Levene's test, p > 0.10 in all cases). We therefore averaged the three hemispheric difference scores for each subject (**Figure 2B**) for the LSF stimuli (top row of gratings in **Figure 2A**) to estimate the overall hemispheric bias across all three stimulus pairs. A positive score corresponds to a RH bias for the relatively lower SF in a pair (and a LH bias for the HSF), a negative score corresponds to a LH bias for the LSF, and a score of 0 corresponds to no difference between the hemispheres in processing the two SFs in a pair. We found that this overall score was significantly greater than zero across subjects (two-sided one-sample Wilcoxon signedrank test; p < 0.02), indicating that hemispheric differences in initial perceptual selection of SFs depend on relative frequency processing.

For each of the two SFs that appeared in two different grating pairs (1.5 and 3 cpd, shown in blue and green, respectively, in **Figure 2A**), we also compared the hemispheric difference score (**Figure 2B**) for the two SFs with which it had been paired (for 1.5 cpd, paired with either 0.75 or 3 cpd; for 3 cpd, paired with either 1.5 or 6 cpd). Specifically, for each SF, we compared the within-subject hemispheric difference score between trials in which it was the relatively lower SF in a pair and the trials in which the same grating was the relatively higher SF in a pair. This procedure enabled us to directly quantify the contribution of relative SF processing to hemispheric asymmetry in perceptual selection. We found a significant main effect of whether a given SF was the relatively higher or lower member of a rivalry pair on hemispheric difference scores across the two analyzed SFs (1.5 and 3 cpd) [two-way ANOVA, F(1,14) = 5.59, p < 0.05]. In addition, there was not a significant interaction between this factor and absolute SF (1.5 vs. 3 cpd) [F(1,14) = 0.65, p = 0.80], indicating that the magnitude of the relative SF processing effect did not differ significantly between the two analyzed SFs.

Finally, we also analyzed the latency and duration of the initial response in each trial (**Figure 3**). For each measure, we again computed a hemispheric difference score (i.e., RH–LH latency, RH–LH duration). We analyzed LSF and HSF separately here because these measures, unlike proportion of initial responses, are not complementary. One-way ANOVAs indicated

that there were no significant main effects of SF pair on hemispheric differences in latency of initial responses for either LSF [F(2,14) = 1.56, p = 0.23] or HSF [F(2,14) = 0.64, p = 0.54] gratings or on the hemispheric differences in duration of initial responses for either LSF [F(2,14) = 1.35, p = 0.28] or HSF [F(2,14) = 0.19, p = 0.82] gratings. We therefore averaged the three hemispheric difference values for each subject for the LSF stimuli and for the HSF stimuli to estimate the overall hemispheric biases across all three grating pairs.

This average hemispheric difference score (RH–LH) was not significantly different from 0 for latencies of LSF (Wilcoxon signed-rank test; p = 0.39) or HSF (p = 0.93) perceptual reports (**Figure 3A**). The same average hemispheric difference score (again, RH–LH) was not significantly different from 0 for duration of LSF initial responses (p = 0.93), but durations of HSF responses were significantly greater in the LH than the RH (p < 0.02; **Figure 3B**), a pattern consistent with results reported in Piazza and Silver (2014).

The trial duration we used here (only 10 s, compared to 30 s in our previous paper, Piazza and Silver, 2014) was designed to investigate the properties of initial responses. Because binocular rivalry typically involves a perceptual alternation, the second response strongly depends on the first response. With sufficiently long trial duration, the influence of the first percept on subsequent responses diminishes, but in our study, the relatively short trials preclude meaningful analysis of responses after the initial response.

# DISCUSSION

We have found that differences in perception of basic visual information between the two cerebral hemispheres are influenced by the set of SFs that is present in the environment at any given time. Our findings extend previous reports of hemispheric asymmetries in processing of SFs, as measured with RT (Christman et al., 1991), fMRI (Peyrin et al., 2004), EEG (Martínez et al., 2001), and perceptual selection (Piazza and Silver, 2014), by showing that the hemispheric asymmetry in perceptual selection of a given SF depends on the other SFs that are simultaneously present.

To the best of our knowledge, the origin and adaptive value of hemispheric asymmetries in SF processing remain somewhat mysterious. There has been some speculation that these asymmetries have developed in conjunction with literacy, in particular because text contains relatively high SFs, languagerelated areas (e.g., visual word form area) are generally lateralized to the left hemisphere, and literate and non-literate individuals differ in the degree of lateralization (Lecours et al., 1988). Future work investigating perceptual selection of broadband naturalistic stimuli may help to elucidate the evolutionary origin of the role of relative SF processing in hemispheric asymmetries.

Our findings are generally related to previous work on contextual modulation in that perception of a stimulus is influenced by other stimuli present in the visual scene. However, unlike contextual effects that arise from interactions between stimuli at different spatial locations (reviewed in Albright and Stoner, 2002; Wagemans et al., 2012), the novel modulation of perceptual selection that we report here is based on a comparison between two SFs, presented in the same retinal location but to different eyes, that compete for visual awareness. This dependence of perceptual experience on a comparison, or integration, of information between two distinct images that are presented simultaneously to different eyes is similar to established effects of interocular Gestalt grouping on binocular rivalry (Díaz-Caneja, 1928; Kovács et al., 1996; see Bressler et al., 2013, for a review).

Because the hemispheric preference for a given SF in our study depends on the SF with which it is paired, our results cannot be explained solely by bottom-up filtering based on a simple absolute SF threshold applied to each grating and thus likely involve some higher-order integration of the rivaling gratings. However, there may be additional mechanisms based on absolute SF that also influence perceptual selection. For example, we have found in the present study that in general, responses to LSFs are more prevalent and faster than responses to HSFs (data not shown), replicating our previous finding that low SFs tend to dominate

high SFs (Piazza and Silver, 2014). Moreover, low SF stimuli are more strongly suppressed than high SF stimuli in continuous flash suppression (Yang and Blake, 2012). It should be noted that even these effects might be influenced by relative processing that is based on continuous recalibration in response to the current set of SFs in the visual scene.

As we did not measure eye position, it is possible that subjects made some eye movements toward the peripheral gratings. However, our overall pattern of results is very unlikely to be explained by eye movements. First, we find that LSF initial percepts were generally more prevalent than HSF percepts (Wilcoxon signed-rank test, comparing likelihood of initially responding to 0.75 or 1.5 cpd versus 3 or 6 cpd, p < 0.001), and they also have shorter latencies (same comparison, p < 0.05). These results are consistent with measurements of initial perception in binocular rivalry that were reported in Piazza and Silver (2014), whereas previous work (Baddeley and Tatler, 2006) demonstrated that eye movements tend to be drawn toward HSF content.

Second, previously reported attentional biases toward the LVF (Siman-Tov et al., 2007) could not explain how each SF's hemifield bias changes across different SF pairs, as our primary hemispheric difference measure is symmetric across the hemifields for a given SF pair (**Figure 2B**). Third, the combination of attentional biases for the LVF and HSFs presumably result in an increased probability of higher SF initial percepts for LVF stimuli, which is exactly the opposite of the pattern of results that we observed. Finally, if participants were frequently making eye movements to the peripheral gratings, thereby placing them at foveal locations, this would be likely to reduce the measured differences between the LVF and RVF in SF processing.

Our results suggest that the visual system classifies SFs that are present in the current visual environment and preferentially routes them for processing into the left or right hemisphere. Future work might explore the neural mechanisms of this routing (e.g., which brain areas have responses that reflect relative SF processing, the role of top-down feedback from within or beyond the visual system) and how temporal context (i.e., recent experience with scenes containing low vs. high SFs) may influence hemispheric preferences for a given SF. Another important future direction for research is to reconcile the well-established symmetries in visual field representations in the two hemispheres (i.e., the many cortical areas that contain contralateral representations of the visual field; Silver and Kastner, 2009) with the hemispheric asymmetry in relative processing of SFs that we report here.

# AUTHOR CONTRIBUTIONS

EP designed the study, collected and analyzed the data, and wrote the paper. MS designed the study and wrote the paper.

# FUNDING

This research was supported by the Department of Defense (through a National Defense Science and Engineering Graduate Fellowship awarded to EP) and NEI Core grant EY003176.

# ACKNOWLEDGMENTS

The authors thank Karen Wong and Christopher Purington for assistance with data collection.

# REFERENCES

fpsyg-08-00559 April 18, 2017 Time: 14:3 # 7


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Piazza and Silver. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Emergence of Visual Awareness: Temporal Dynamics in Relation to Task and Mask Type

Markus Kiefer<sup>1</sup> \* and Thomas Kammer<sup>2</sup>

<sup>1</sup> Section for Cognitive Electrophysiology, Department of Psychiatry, Ulm University, Ulm, Germany, <sup>2</sup> Section for Neurostimulation, Department of Psychiatry, Ulm University, Ulm, Germany

One aspect of consciousness phenomena, the temporal emergence of visual awareness, has been subject of a controversial debate. How can visual awareness, that is the experiential quality of visual stimuli, be characterized best? Is there a sharp discontinuous or dichotomous transition between unaware and fully aware states, or does awareness emerge gradually encompassing intermediate states? Previous studies yielded conflicting results and supported both dichotomous and gradual views. It is well conceivable that these conflicting results are more than noise, but reflect the dynamic nature of the temporal emergence of visual awareness. Using a psychophysical approach, the present research tested whether the emergence of visual awareness is context-dependent with a temporal two-alternative forced choice task. During backward masking of word targets, it was assessed whether the relative temporal sequence of stimulus thresholds is modulated by the task (stimulus presence, letter case, lexical decision, and semantic category) and by mask type. Four masks with different similarity to the target features were created. Psychophysical functions were then fitted to the accuracy data in the different task conditions as a function of the stimulus mask SOA in order to determine the inflection point (conscious threshold of each feature) and slope of the psychophysical function (transition from unaware to aware within each feature). Depending on feature-mask similarity, thresholds in the different tasks were highly dispersed suggesting a graded transition from unawareness to awareness or had less differentiated thresholds indicating that clusters of features probed by the tasks quite simultaneously contribute to the percept. The latter observation, although not compatible with the notion of a sharp all-or-none transition between unaware and aware states, suggests a less gradual or more discontinuous emergence of awareness. Analyses of slopes of the fitted psychophysical functions also indicated that the emergence of awareness of single features is variable and might be influenced by the continuity of the feature dimensions. The present work thus suggests that the emergence of awareness is neither purely gradual nor dichotomous, but highly dynamic depending on the task and mask type.

#### Edited by:

Guido Hesselmann, Charité – Universitätsmedizin Berlin, Germany

#### Reviewed by:

Bruno Breitmeyer, University of Houston, USA Timo Stein, University of Trento, Italy

\*Correspondence: Markus Kiefer markus.kiefer@uni-ulm.de

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 26 September 2016 Accepted: 20 February 2017 Published: 03 March 2017

#### Citation:

Kiefer M and Kammer T (2017) The Emergence of Visual Awareness: Temporal Dynamics in Relation to Task and Mask Type. Front. Psychol. 8:315. doi: 10.3389/fpsyg.2017.00315

Keywords: visual masking, awareness, consciousness, psychophysics, vision, graded, dichotomous

# INTRODUCTION

fpsyg-08-00315 March 1, 2017 Time: 15:12 # 2

Elucidating the cognitive and neural mechanisms underlying phenomenal consciousness (Block, 1995), that is, the experiential qualities of sensations, remains one of the greatest and most exciting scientific endeavors in the 21st century. One particular challenge in the scientific explanation of phenomenal consciousness is the privacy of subjective experiences (Nagel, 1974), which renders an objective assessment of the phenomenon in question difficult. However, an adequate characterization of consciousness phenomena is an important prerequisite for determining the underlying neuro-cognitive mechanisms (Dehaene and Naccache, 2001; Kiefer et al., 2011).

One aspect of consciousness phenomena, the temporal emergence of visual awareness, has been subject of a controversial debate (for a review see, Windey and Cleeremans, 2015). A first class of proposals assumes that visual awareness is a phenomenon gradually developing over time ranging from unawareness over a coarse glimpse to full awareness of the visual stimulus (Dennett and Kinsbourne, 1992; Overgaard et al., 2006; Seth et al., 2008). A second opposing class of proposals argues that consciousness is an all-or-none phenomenon (Sergent and Dehaene, 2004; Quiroga et al., 2008; Sekar et al., 2013; Asplund et al., 2014). These proposals state that the transition between unawareness and full awareness of a stimulus proceeds in a binary fashion. A third class of intermediate proposals suggests independently accessible levels of representation such as physical energy, simple visual features, letters, word forms and meaning (Kouider et al., 2011; Windey et al., 2013; Windey and Cleeremans, 2015). States of full awareness include access to all levels of representations, whereas in complete unawareness there is no access to any level. Most critically, this class of proposals also assumes states of partial awareness, in which observers only experience the informational content of some restricted levels. According to different variants of this partial awareness hypothesis, access to the different levels of representation either exclusively occurs in an all-or-none fashion (Kouider et al., 2011) or might depend on the feature type: Conscious access to low-level visual features (such as energy, geometrical elements, lines and color) is assumed to occur in a gradual fashion, whereas access to higher-level features (e.g., word form or meaning) is supposed to be allor-none (Windey et al., 2013; Windey and Cleeremans, 2015). Higher-level features of visual stimuli referring to lexical (word form) or semantic representations (meaning) transcendent the information provided by the visual sensory channel and can be characterized as multimodal or even amodal in some aspects (Kiefer and Pulvermüller, 2012). In keeping the terminology with previous research and for simplicity reasons, we use the term "visual awareness" throughout this paper also in the context of these higher-level features, in order to express that their representations are accessed from visual stimuli.

Most previous studies addressing the temporal emergence of visual awareness used subjective awareness ratings of the clarity of the percept as primary variable of interest (e.g., Sergent and Dehaene, 2004; Overgaard et al., 2006; Windey et al., 2013). Such subjective measures based on introspection were preferred over objective measurements of discrimination performance based on signal detection theory, because the latter measures can also capture the influence of unconscious processing and cannot be taken as an exclusive index of visual awareness (Sandberg et al., 2011). Furthermore, as objective awareness measures based on signal detection theory typically average detection performance over a series of trials, they do not consider the entire performance distribution. However, the validity of subjective ratings of introspective experience in single trials has also been controversially discussed: Subjective ratings might be affected by response biases and cannot be necessarily taken as direct reflections of subjective visual experience (Snodgrass and Shevrin, 2006; Asplund et al., 2014; Schmidt, 2015).

In addition to these measurement problems, previous studies yielded heterogeneous results (for a discussion see, Bachmann, 2013) supporting all-or-none (Sergent and Dehaene, 2004; Quiroga et al., 2008; Sekar et al., 2013; Asplund et al., 2014), graded (Overgaard et al., 2006; Sandberg et al., 2010), or partialawareness proposals (Windey et al., 2013). Given that different experimental blinding methods (visual masking vs. attentional blink), different stimuli or stimulus features (low level vs. high level) or masks (random pattern vs. random letters) were used in the previous studies, it is well conceivable that these conflicting results are more than noise, but reflect the dynamic and contextdependent nature of the temporal emergence of visual awareness (see also, Overgaard and Mogensen, 2016). We propose that, depending on the specific stimulation context and blinding method, the percept could emerge either in an all-or-none or in a more graded fashion. Despite some differences (for a discussion see, Breitmeyer et al., 2015), many current models converge on the assumption that visual awareness requires recurrent processing of the stimulus within multiple brain systems, thereby consolidating its representation (Enns and Di Lollo, 2000; Dehaene and Naccache, 2001; Lamme, 2003; Kiefer et al., 2011). Consolidation through recurrent processing can be characterized as reaching an attractor state within neural networks (Herzog et al., 2016). Depending on the specific context, the attractor state might be reached from previous intermediate states of unconscious processing through relatively sharp or smooth transitions resulting in a more dichotomous or gradual emergence of awareness. Furthermore, again depending on the context, the attractor state might encompass different neural systems coding specific features of the stimulus. This would lead to awareness of just a few or all stimulus features as suggested by the partial awareness hypothesis (Kouider et al., 2011).

In line with such a dynamic and context-dependent view of the temporal emergence of visual awareness, findings with masking and attentional blink paradigms yielded quite heterogeneous results depending on the specific stimuli used. For instance, attentional blink paradigms with word, color or face stimuli produced a data pattern consistent with an all-ornone emergence of a visual percept (Sergent and Dehaene, 2004; Asplund et al., 2014), whereas attentional blink paradigms with letters were associated with a more gradual emergence of a visual percept (Nieuwenhuis and de Kleijn, 2011). Masking experiments with complex pattern masks, in contrast, frequently yielded results consistent with a gradual emergence of consciousness (Sergent and Dehaene, 2004; Sandberg et al., 2010).

In order to address the issue whether visual awareness emerges dynamically in a context-dependent fashion, in the present study, we determined identification and discrimination thresholds for stimulus features of varying complexity in different masking contexts using a temporal two-alternative forced choice task (temporal 2-AFC). This temporal 2-AFC task has the advantage to provide an objective psychophysical measurement of identification or discrimination performance as an index of visual awareness with a comparable set of word stimuli while minimizing biases from unconscious processing. In the task, participants were presented with two stimulusmask sequences separated by a delay of 900 ms. The critical task-relevant stimulus feature (energy or stimulus presence/letter case/lexicality/semantics) randomly appeared either in the first or the second interval. After the second sequence, participants were prompted to indicate in which interval the designated stimulus feature was presented. Above-threshold performance in this temporal 2-AFC task most likely exclusively reflects awareness of the critical feature, because response biases from unconscious processing are minimized for several reasons: Semantic priming elicited by unconsciously perceived masked stimuli have been shown to decay rapidly after about 100 ms (Brown and Hagoort, 1993; Greenwald et al., 1996; Kiefer and Spitzer, 2000; Kiefer and Brendel, 2006), while response tendencies initiated by masked stimuli are even inhibited after this time interval (Eimer and Schlaghecken, 2003). As the response in this task is delayed after the second presentation interval, it is unlikely that unconscious semantic priming or visuo-motor processes were able to bias the response (see also, Milner and Dijkerman, 2001). Furthermore, the task requires the comparison of the percepts in intervals one and two. We are not aware of any evidence that such a complex comparison can be performed on the basis of unconscious visual processes (Ansorge et al., 2014).

Visibility of the word stimuli were manipulated by gradually varying the target stimulus mask onset asynchrony (stimulus mask SOA) according to a staircase algorithm. Psychophysical functions were then fitted to the accuracy data of stimulus detection (presence of an stimulus or energy) or discrimination (letter case/lexicality/semantics) performance in the different task conditions as a function of the stimulus mask SOA in order to determine the conscious detection or discrimination threshold (point of inflection of the logistic function, estimated accuracy of 75%). A higher threshold indicates that the stimulus has to be presented longer in isolation before mask onset so that a given task-relevant feature can be consciously identified.

In addition to thresholds, slopes of the fitted psychophysical functions in the different conditions can also be determined. A steep slope of the function is taken to index that visual awareness of a given feature emerges in a more all-or-none fashion, whereas a shallow slope is assumed to indicate a gradual transition from unconscious to conscious at the feature level (Koch and Preuschoff, 2007; Sandberg et al., 2011). Hence, when analyzing thresholds and slopes of the psychophysical function we can determine the relative timing of access to consciousness across stimulus features (thresholds) as well as the abruptness vs. smoothness of the transition from the unconscious to the conscious within stimulus features (slopes).

Our experimental approach using stimulus mask SOA as variable to infer the temporal emergence of awareness within the temporal 2AFC task is based on the following rationale: The higher the threshold in terms of stimulus mask SOA, the longer the stimulus requires processing within the visual system to achieve consolidation before the mask interferes with its processing. Based on these considerations, the stimulus mask SOA provides information about the approximate time point, at which masked stimulus features are sufficiently consolidated to be available for conscious access as indicated by above-threshold performance in the temporal 2AFC task (estimated SOA at 75% correct performance). Whether the observed time course of features thresholds not only reflects the time course of conscious access, but also the time course of phenomenal experience critically depends on the plausible assumption that visual awareness (subjective perceptual experience) and access consciousness (above-threshold discrimination performance of stimuli) exhibit an at least correlated time course. Furthermore, our psychophysical approach informs us about the temporal emergence of features only for briefly presented masked stimuli, and is mute with regard to the relative timing of stimulus features under unmasked conditions or for longer stimulus duration. However, these limitations apply to all experimental approaches to consciousness; whether the blinding technique of choice is a masking or attentional blink paradigm, or whether psychophysical measures such as thresholds and slopes or subjective awareness ratings are used as index of awareness.

Although psychophysical functions are necessarily fitted on performance data of a series of trials, the threshold or slope parameters of this function characterize the entire accuracy distribution across single trials with varying stimulus mask SOAs and are not simple average performance measures. Hence, the criticism of objective awareness measures based on averages across trials, as raised in the context of signal detection theory mentioned above, does not apply to our psychophysical approach.

The competing proposals of the temporal emergence of visual awareness make different predictions regarding the relative temporal ordering of thresholds for the different taskrelevant stimulus features and slopes of the psychophysical functions. According to all-or-none proposals, thresholds of the different features should be very similar, and psychophysical functions should generally have a steep slope (Del Cul et al., 2009). Proposals assuming a gradual emergence of consciousness (Dennett and Kinsbourne, 1992; Overgaard et al., 2006; Seth et al., 2008) as well as the partial awareness hypothesis (Kouider et al., 2011; Windey et al., 2013; Windey and Cleeremans, 2015), in contrast, predict that threshold SOAs of the features should depend on their level of complexity with energy (stimulus presence decision) having the lowest threshold followed by letter form (capital letter decision), word form (lexical decision), and meaning (semantic decision).

Although all variants of the partial awareness hypothesis suggest a sequential contribution of stimulus features to the conscious percept as a function of their complexity level, there is a controversy how awareness of one particular feature emerges. According to one variant, the transition from unawareness to awareness for each feature occurs in an all-or-none fashion (Kouider et al., 2011). This implies that threshold SOAs for the features should show the temporal gradient described above, but the slope of the psychometric function should be invariantly steep. According to the other variant (Windey et al., 2013), transition from unawareness to awareness for the different features depends on their complexity level. Hence, this proposal predicts both a temporal gradient of threshold SOAs and a variation of slopes of the psychometric function: Awareness for low-level features such as color should emerge gradually resulting in a shallower slope of the psychometric function. Awareness for higher-level features such as semantics should emerge in an all-or-none fashion resulting in steep slopes.

# Predictions with Regard to Threshold SOAs in Relation to Mask Type and Task

Target-mask threshold SOAs indicate the time needed for visual consolidation so that a given stimulus feature is available for conscious access. In order to test the contextual dynamics of visual awareness, we created four different masks, which systematically varied with regard to their similarity with the word targets (random pattern mask, false font mask, random letter mask, and word mask). Masks are more efficient to suppress stimulus visibility, when mask and target share common features (Breitmeyer and Ögmen, 2006 ˘ ). Likewise, comparable similarity effects between targets and distractors have been observed within the context of the attentional blink paradigm (Maki et al., 2003; Dux and Coltheart, 2005). We therefore predicted that higher-level masks (random letters, word) should yield higher thresholds SOAs compared with low-level masks (random pattern and false fonts). Most critically, in line with the notion of a context-dependent dynamic emergence of visual awareness, as outlined above, we expected that the relative ordering of threshold SOAs in the different task conditions (stimulus presence, letter case, lexicality, and semantic category) is modulated by the type of mask: As the random pattern mask does not bear any similarity with any task-relevant word feature, all features should quite simultaneously contribute to the conscious percept. Threshold SOAs in the different tasks should be therefore similar or even identical indicating that the conscious percept emerges in a more discontinuous, or even in an all-or-none fashion. In contrast, for higher-level letter and word masks, which specifically interfere with the lexical or semantic levels of word representation, access to the corresponding features of the percept should be delayed compared with lowlevel features. We therefore expected to observe a greater differentiation of threshold SOAs in the different tasks with these higher-level masks. As a differentiation of thresholds indicates that features can be discriminated more sequentially at distinct stimulus mask SOAs, such a pattern of results would be consistent with a more gradual emergence of visual awareness.

# Predictions with Regard to the Slopes of the Psychophysical Functions in Relation to Mask Type and Task

The slopes of the psychophysical function index the transition from unawareness to awareness within each task probing a specific feature type (Koch and Preuschoff, 2007; Windey et al., 2013). All-or-none proposals (e.g., Del Cul et al., 2009) predict generally steep slopes (and comparable SOA thresholds for all tasks). The analysis of slopes is suited to distinguish between different variants of the partial awareness hypotheses, which all predict a temporal gradient of threshold SOAs as outlined above. If conscious access to individual features were generally dichotomous as assumed in the framework by Kouider et al. (2011), slopes of the psychophysical function in the different tasks should be invariantly steep. In contrast, the partial awareness framework proposed by Windey et al. (2013) and Windey and Cleeremans (2015) predicts shallower slopes, i.e., more gradual emergence of awareness, for low-level visual features (energy, letter form) compared with higher-level word form or semantic features, which should exhibit a more discontinuous emergence of awareness. However, a further scenario is also conceivable: The complexity of processing and thus the time needed for consolidation may also influence the transition from unawareness to awareness for individual features, resulting in shallower slopes for features with longer threshold SOAs. As described above, random letter and word masks may specifically interfere with and thus delay the processing of higher-level lexical and semantic features compared with the more neutral pattern and false font masks. As a consequence, for higherlevel features the transition from unawareness to awareness would be more gradual under random letter and word masks resulting in shallower slopes in the lexical and semantic tasks.

# MATERIALS AND METHODS

# Subjects

Eighty-four subjects (mean age 22.3, 47 female) were recruited for the study. They were native German-speaking volunteers without any history of neurological or psychiatric disorders. They participated after giving written informed consent and they were compensated for participation either by money or by course credits. The study has been approved by the institutional review board of Ulm University. Sample size was determined according to earlier psychophysical studies. The effects of the four different masks were investigated in a between-subject design (see below). Each subject was randomly assigned to one of the four groups. Data collection was stopped when the data of at least 16 subjects within each group fulfilled the inclusion criteria for analysis (see below).

Visual acuity was tested using FrACT (Version 3.7.1b, central Landoldt-C, 4AFC, 30 trials, observer distance 2.5 m, Bach, 1996). All subjects had a binocular visuals of 0.85 at minimum, and no subject was excluded due to vision impairment.

## Stimuli and Apparatus

fpsyg-08-00315 March 1, 2017 Time: 15:12 # 5

Five word lists comprising 50 pairs of stimuli were generated from the CELEX lexical database (Baayen et al., 1995). Only words with six characters were used. For the different tasks (see below) different word lists were used (see Supplementary Table S1), which were matched for word frequency and word length.

Stimuli were generated using Psychopy (v1.78.01, cf. Peirce, 2007) and presented on a CRT screen (21<sup>0</sup> , iiyama, Hoofddorp, The Netherlands) at a frame rate of 150 Hz. Target words were flashed for one frame using the font Courier new with a height of 0.38◦ in white (25 cd/m<sup>2</sup> ) on a gray background (5 cd/m<sup>2</sup> ). According to the German spelling norms, first letters were always written in capital, with the exception in the capital task (see below). The mask with a duration of 30 frames (200 ms) followed the target with an SOA varying from 1 to 50 frames, i.e., 6.7–340 ms (**Figure 1**). Four different types of masks were applied: (a) a pattern mask consisting of 4 × 28 squares, either white (50 cd/m<sup>2</sup> ) or gray (5 cd/m<sup>2</sup> , space averaged luminance 27.5 cd/m<sup>2</sup> ) distributed at random, with a size of 3.5◦ by 0.5◦ covering the target words, (b) a random string of eight symbols from a false font created using elements of Courier new, (c) a random string of eight consonants, lower and upper cases mixed at random, (d) an abstract word, semantically unrelated to any

FIGURE 1 | Schematic drawing of the tasks. Each of the four tasks consists of two target-mask sequences with target pairs (targets 1 and 2) representing the respective task features: word and blank screen, word in upper and lower cases, word and pseudoword, natural object and artifact. Duration of target presentation was one frame on the CRT screen. Stimulus onset asynchrony (SOA) between target and mask was identical in both target-mask sequences. It was adaptively varied in order to measure discrimination thresholds for the given task between 6.7 ms (no blank frame between target and mask) and 340 ms (50 blank frames between target and mask). Mask duration was 200 ms as depicted in the semantic condition, interval between the two target-mask sequences was 900 ms. Subjects had to indicate by button press in which of the two intervals the feature of the given task was seen (response). The four masks depicted as inset in interval 1 for the absent/present task (pattern, false font, random strings, and abstract words) were not varied within subjects but were administered to four different groups of subjects. The mask in interval 2 was always different to the mask in interval 1. The hash displayed at the begin of each interval and the question mark displayed following mask 2 are not depicted. Notice that text sizes and mask sizes are enlarged in relation to the screen for sake of clarity. The fifth task (visibility) was identical with the absent/present task, using only the shortest SOA of 6.7 ms.

of the target words, with a length of eight characters chosen at random from a list of 20 words, written with a capital at the beginning followed by lower case letters (**Figure 1**). Symbol string masks (b, c, and d) had the same height as target words (0.38◦ ) and were two letters longer. Luminance of the three symbol string masks was 50 cd/m<sup>2</sup> . The string masks covered about 30% of the total space, resulting in a space averaged luminance of 18.5 cd/m<sup>2</sup> .

Subjects sat in front of a CRT screen with a distance of 1.5 m in a room with dimmed ambient light. During the experiment they responded with left and right index fingers, respectively, on a keyboard.

Since timing of the target – mask sequence was critical in the experiment, precision of sequences was tested externally using a photodiode at the CRT screen during development of the task. Test runs with variable SOAs demonstrated precise timing without frame drops and with stable durations of target and mask.

#### Design of the Experiment

Participants performed five different two-interval forced choice tasks with 50 trials each, in which discrimination thresholds for a certain task feature were measured by adaptively varying the target-mask SOA. A trial consisted of the presentation of two target-mask sequences, finished by a temporal two-alternative forced choice decision. The sequence started with a hash (#) for 500 ms followed by a blank screen for 500 ms. Then the target word was flashed for one frame followed by the mask for 200 ms. Target and mask were separated by a blank period with a variable SOA of 6.7–340 ms. After a pause (blank screen for 900 ms following the presentation of the mask), the second sequence with identical timing parameters started with a hash. Thus, the second sequence started 2100 ms + SOA after the first sequence. Immediately after the end of the second sequence, a question mark indicated the subject to respond. (**Figure 1**, for sake of clarity the hash as well as the question mark are not depicted). As responding in this task is delayed and unconscious priming has been shown to decay rapidly after about 100 ms (Greenwald et al., 1996; Kiefer and Spitzer, 2000; Kiefer and Brendel, 2006), above-threshold performance in this temporal 2-AFC task most likely exclusively reflects awareness of the critical feature, because biases from unconscious processing are minimized (for a more detailed discussion of this issue, see the introductory section).

The order of the first four tasks was counterbalanced over subjects, followed by the 5th task, the visibility task, always at the end of the experiment. The five tasks were: (i) Absent/present task. In only one of the two target-mask sequences a word was flashed, while in the other sequence only the mask was shown. (ii) Capital task. In one sequence the target word was written in capitals, in the other sequence the target word was written in lower cases, including the first letter. (iii) Lexical decision task. A word or a pseudoword was presented as target in the sequences. (iv) Semantic task. One target word referred to a natural object, whereas the other target word named a manmade artifact. (v) Visibility task. Similar to the absent/present task, in only one sequence a target word was presented. In difference to the former task the target-mask SOA was fixed to 6.7 ms, and no discrimination threshold but a detection ratio was measured. This task served to assess, whether stimuli were entirely unconscious at minimal stimulus mask SOA.

Report categories (e.g., "In which interval was the word written in capital letters?" vs. "In which interval was the word written in lower case letters?") were balanced over subjects. The target-mask interval, in which the critical stimulus was present (first or second interval), was varied randomly with the restriction that the critical stimulus appeared equally often in the first and second interval, respectively. Prior to each task, an instruction text on the screen described the task (e.g., "In one interval a word written in lower case letters will be presented, in the other interval the word will be written in capital letters.") and the report category. Each task started with five training trials with a fixed SOA of 173 ms. Then 50 trials with adapting SOAs were presented. Two simple 2-down 1-up staircases were randomly intermixed, 25 trials each, to achieve a good sampling across a broad SOA range. The first staircase started at an SOA of 173 ms, step size was 33 ms. The second staircase started at an SOA of 73 ms, step size was 20 ms.

The effects of the four different masks were investigated in a between-subject design in order to keep the length of the entire experimental session (about 1 h) manageable for the participants. Furthermore, the number of available word stimuli for the semantic condition was too small to allow for presentation within a within-subject design without stimulus repetition. Each subject was randomly assigned to one of the four groups, and in each of the five tasks the same mask was used.

## Data Analysis

In each individual and task, responses from the two staircases were collapsed, and a psychometric function (**Figure 2**) was fitted to the accuracy distribution as a function of target-mask SOA using psignifit (v2.5.6, cf. Wichmann and Hill, 2001). The logistic function

$$\mathbf{F}(\mathbf{x}) = \boldsymbol{\gamma} + \frac{1 - \boldsymbol{\gamma} - \boldsymbol{\gamma}}{1 + \mathbf{e}^{-\beta(\mathbf{x} - \alpha)}}$$

with α as threshold, β as slope parameter, γ as probability rate, and λ as lapsus rate, was applied, with the free parameters α and β while fixing γ = 0.5 and λ = 0. Please notice that an increase in β yields a steeper psychometric function. Detection thresholds (absent/present task) or discrimination thresholds (the other three tasks) were defined as SOA with 75% correct responses (point of inflection of the logistic function, α).

Thresholds and slopes were analyzed using a mixed-design analysis of variance (ANOVA, Statistica V12, StatSoft, Hamburg, Germany). Greenhouse-Geyser corrections were applied in case of violation of sphericity (Mauchley's test). We report Greenhouse-Geyser's ε together with the uncorrected degrees of freedom. Post hoc analyses were performed using Newman–Keuls tests.

Seventeen subjects (20%) had to be excluded. In nine subjects, accuracy in the visibility task, in which masks were presented at the shortest possible target-mask SOA, was above chance performance [correct identification of 32/50 trials (64%) or more, above chance performance according to binomial distribution, p = 0.03]. We excluded these subjects from analyses of the

condition, bottom: participant in the word mask condition. The four different tasks are depicted in different symbols and different colors. The size of the symbols codes the number of presentations per SOA which differs due to the staircase procedures. In the examples the range of number of presentations is n = 1 (lower graph, red square at 17 ms) to n = 17 (lower graph, blue triangle at 73 ms). The largest SOA of 173 ms is not included into the graph for sake of clarity (performance 1.0 in each task). Slope values in the order of the data labels: upper 1.33, 0.0499, 0.0729, 0.139; lower 1.33, 0.0862, 0.227, 0.110.

main experiment, because it cannot be guaranteed that states of complete unawareness were reached, even at the shortest SOA. Performance above chance level at the shortest SOA does not allow a reliable threshold estimation based on fitting of a sigmoid function, because the lower left part of the function, performance at random, is not reached. In the remaining eight rejected subjects, thresholds could not be determined because the psychometric function of one or several tasks could not be adequately fitted due to a non-monotonous variation of performance as a function of target-mask SOA. The number of subjects included was: 17 (pattern mask), 17 (false font mask), 16 (random string mask), 17 (word mask).

#### RESULTS

#### Analysis of Threshold

Threshold data from 67 subjects were subjected to an omnibus mixed-design ANOVA with the group factor MASK (four groups: pattern, false font, random string, and word) and the within factor TASK (four levels: absent/present, capital, lexical, and semantic). Both factors revealed significant differences: group factor MASK [F(3,63) = 5.0, p = 0.0036, η 2 <sup>p</sup> = 0.19], within factor TASK [F(3,189) = 145.4, ε = 0.90, p < 0.001, η 2 <sup>p</sup> = 0.70]. Most importantly, the interaction MASK × TASK was also significant [F(9,189) = 2.97, ε = 0.90, p = 0.0026, η 2 <sup>p</sup> = 0.12] (**Figure 3** and Supplementary Table S2).

Regarding factor TASK, post hoc tests showed that lowest thresholds were obtained with the absent/present task, followed by capital and semantic tasks, whereas the lexical decision task resulted in highest thresholds (all comparisons p < 0.005). These post hoc tests thus revealed a gradient of awareness thresholds as a function of the task-relevant stimulus feature. Post hoc tests for the factor MASK indicated that thresholds were similar for mask 1 (pattern) and mask 2 (false font, p > 0.25). Thresholds for both masks were significantly lower compared to mask 3 (random string, pattern p = 0.019, false font p = 0.015) and mask 4 (word, pattern p = 0.033, false font p = 0.018). Thresholds for masks 3 and 4 did not differ from each other (p > 0.25).

In order to assess the complex interaction between the influences of MASK and TASK on awareness thresholds in more detail, subsidiary ANOVAs were calculated (**Table 1**). Firstly, between group analyses are reported comparing the impact of the four different masks on one of the four different tasks (**Table 1A** and **Figure 3**). In the absent/present task different masks did not significantly affect thresholds [between group comparison, F(3,63) = 2.33, p = 0.083, η 2 <sup>p</sup> = 0.10]. This shows that all masks were equally effective in interfering with the ability to consciously report the presence of a stimulus. For the remaining three tasks, capital, lexical, and semantic, the different masks significantly affected thresholds. In the capital task, mask 3 (random string) yielded significantly higher thresholds compared to mask 1 (pattern, p = 0.0097) and to mask 2 (false font, p = 0.014). Threshold with mask 3 (random string) was higher than that obtained with mask 4 [word; although this difference did not reach conventional significance levels (p = 0.066)]. Thresholds with mask 4 (word) did not significantly differ from those of mask 1 (pattern, p = 0.25) and mask 2 (false font, p > 0.25).

A different pattern was observed for the two remaining tasks, lexical and semantic. In these higher-level tasks, mask 3 (random string) and in particular mask 4 (word) yielded the highest thresholds. In the lexical task, thresholds obtained with mask 3 (random string, p = 0.043) as well as with mask 4 (word, p = 0.021) were higher than thresholds with respect to mask 1 (pattern). Furthermore, the thresholds with mask 4 (word) were significantly higher than those of mask 2 (false font, p = 0.044). For mask 3 (random string), the difference to mask 2 (false fonts) just failed to reach conventional significance levels (p = 0.055). Finally, in the semantic task only thresholds obtained with mask 4 (word) were significantly higher compared to the thresholds obtained with mask 1 (pattern, p = 0.009) and mask 2 (false font, p = 0.016). Thresholds with mask 3 (random string) did not significantly differ from those of mask 1 (pattern, p = 0.085) and mask 2 (false font, p > 0.25) and those of the mask 4 (word, p = 0.20).

To summarize, the different tasks interfered differently with the different masks. In the absent/present task, thresholds did not differ across masks. Whereas in the capital task highest thresholds were obtained applying mask 3 (random string) followed by mask 4 (word), in the lexical and semantic task the highest thresholds were seen with mask 4 (word) and mask 3 (random string), although only thresholds with mask 4 consistently differed from both those of mask 1 (pattern) and mask 2 (false font). Thresholds with mask 1 (pattern) and mask 2 (false font) were always below mask 3 (random string) and mask 4 (word).

Secondly and most importantly to the purpose of the present study, we determined the threshold pattern of the different tasks within each masking condition. To this end, for each masking condition, a repeated-measure ANOVA with the within factor TASK was performed. For all masks, the factor TASK was significant (**Table 1B** and **Figure 3**). However, congruent with the interaction task × mask obtained in the omnibus ANOVA, the tasks elicited a differential pattern of thresholds depending on the masking condition. While the absent/present task yielded significantly lower thresholds compared to the other three tasks in all masks (all p < 0.001), the pattern of thresholds in the other tasks differed across masks. Thresholds in the capital and semantic tasks were comparable for mask 1 (pattern. p = 0.11), mask 2 (false font, p = 0.072) and mask 3 (random string, p > 0.25), but significantly differed from each other only with mask 4 (word, p < 0.001). Thresholds of the semantic task were comparable to those of the lexical task for mask 1 (pattern, p = 0.078). For the other masks, thresholds were consistently and significantly higher for the lexical than the semantic task (mask 2: p = 0.004, mask 3: p = 0.027, mask 4: p = 0.002). With all four masks, the capital task differed from the lexical task (mask 1: p = 0.004, mask 2: p < 0.001, mask 3: p = 0.028, mask 4: p < 0.001). To summarize, while the absent/present task consistently exhibited the lowest threshold across all masking conditions, the threshold pattern of the other tasks varied as a function of the mask. For mask 1 the thresholds for the capital, lexical and semantic tasks were least differentiated with only a significant difference between the capital and lexical tasks. For

#### TABLE 1 | Subsidiary analysis of variance (ANOVAs).



(B) Within a given masking group: effect of the within-subject factor task.


mask 4, in contrast, the threshold pattern was most differentiated with significant differences between all three tasks. The threshold pattern for masks 2 and 3 showed an intermediate form of differentiation with diverging thresholds for the lexical task on the one hand and those of the capital and semantic tasks on the other hand.

# Analysis of Slopes of the Psychometric Functions

Slopes of the fitted psychometric functions were analyzed (similar to threshold measures) by an omnibus mixeddesign ANOVA with the group factor MASK (four groups: pattern, false font, random string, and word) and the within factor TASK (four levels: absent/present, capital, lexical, and semantic). Only the factor TASK yielded significant effects [F(3,189) = 23.3, ε = 0.85, p < 0.001, η 2 <sup>p</sup> = 0.27, **Figure 4** and Supplementary Table S3], whereas the group factor MASK [F(3,63) = 0.073, p > 0.5, η 2 <sup>p</sup> = 0.004] as well as the interaction MASK × TASK [F(9,189) = 0.92, p > 0.5, η 2 <sup>p</sup> = 0.04] were not statistically significant. Post hoc tests showed that slopes for the absent/present task were steeper compared to the other three tasks (p < 0.001). The other task conditions did not differ from each other.

#### Analysis of Visibility Task

Mean performance accuracy in the visibility task (absent/present at constant SOA of 6.7 ms) for all subjects included into the analysis (n = 67, see Materials and Methods) was 51.5% ± 0.62%, range from 38 to 62%. An ANOVA comparing the performance accuracy over the four mask groups revealed no systematic difference [F(3,63) = 0.5, p > 0.25, η 2 <sup>p</sup> = 0.02]. This suggests that masks were similarly effective at the smallest SOA.

# DISCUSSION

The present study assessed the temporal emergence of visual awareness by determining the awareness thresholds of stimulus features of varying complexity using an objective psychophysical approach. In order to obtain a measure of the temporal dynamics of awareness, stimulus visibility was varied by manipulating target-mask SOA according to a staircase

algorithm. Feature-specific discrimination thresholds were determined by fitting psychophysical functions to the accuracy distribution across trials as a function of target-mask SOA. Our approach has the advantage over detection measures based on signal detection theory that the fitted psychophysical function reflects the entire performance distribution and is not an average measure. Furthermore, our temporal 2-AFC choice task minimizes responses based on unconscious processing. Hence, in line with general psychophysical approaches to threshold measurements (Kingdom and Prins, 2016), our paradigm is suited to yield valid information about the temporal emergence of stimulus features contributing to a percept. We specifically tested whether the dispersion of feature discrimination thresholds depends on the context by varying the similarity between mask and stimulus. Such an interaction between feature type and mask would indicate a context-dependent dynamic emergence of visual awareness.

One might argue that psychophysical measures based on performance in masking experiments, e.g., thresholds as a function of the SOA as in our study, are not informative with regard to the emergence of visual awareness per se, but simply reflect resilience of different visual processes to mask interference or the ability of visual processes to handle two stimuli (target and mask) in parallel. We do not think that these alternative interpretations contradict our interpretation, because they interpret the data at a different theoretical level. Less interference by the mask should be related to a higher discrimination accuracy at shorter SOAs (i.e., a lower threshold), which in turn should index that phenomenal awareness of the stimulus feature is reached after shorter processing, i.e., temporally earlier. As we already have laid out in the section "Introduction," in order to be able to make inferences from behavior to the content of phenomenal consciousness, we must make the plausible (but difficult to validate) assumption that phenomenal consciousness (subjective experience) and access consciousness (here: abovethreshold discrimination performance) are at least correlated in time.

Furthermore, one could deny the assumption of a temporal correlation of threshold SOAs and the time course of the emergence of subjective experience. One could instead alternatively postulate that the time course of emergence into awareness is completely uncoupled from earlier processing stages we characterized by measuring threshold SOAs. This alternative assumption, however, seems implausible, since processing within neural networks typically occurs in a cascading fashion: Information from earlier processing stages is immediately transmitted to further stages so that temporal characteristics from earlier processing stages are still present or even more exaggerated at later stages (Humphreys et al., 1988; Kammer et al., 1999).

# Context-Dependency of Feature Thresholds: Emergence of Awareness across Feature Types

In line with our predictions, the type of the mask significantly influenced the relative sequence of feature thresholds SOAs. Although stimulus presence decisions (awareness of stimulus energy) had consistently the lowest threshold SOAs across masks, the differentiation of thresholds for the other features was strongly influenced by the specific masking context. For pattern masks, which do not show any similarity with any target

feature, thresholds for the capital, lexical and semantic tasks were comparable with only a significant difference between the capital and lexical tasks. For the word mask, which has a high similarity with the target words at all levels of representation (energy, letter, lexical and semantic level), the temporal differentiation of feature thresholds was largest, with significant differences between all four tasks. Finally, the sequence of thresholds for the false font and random letters masks, respectively, which bear similarity with the target words only at the energy and letter level, showed an intermediate form of differentiation with diverging thresholds only for the lexical task on the one hand and those of the capital and semantic tasks on the other hand.

When considering the sequence of feature threshold SOAs, the least differentiated dispersion of thresholds under the pattern mask, although not compatible with the notion of a sharp dichotomous transition between unaware and aware states, suggests a less graded and more discontinuous emergence of awareness (Sergent and Dehaene, 2004; Sekar et al., 2013; Asplund et al., 2014): The absence-presence task had the lowest thresholds suggesting that stimulus presence ('energy') is rapidly consciously accessible. However, the more complex letter, lexical or semantic features exhibited significantly higher thresholds and thus contributed later, but almost simultaneously to the percept. In contrast, the high temporal differentiation of thresholds under the word masks suggest that awareness of letter, semantic and lexical word features emerges sequentially, consistent with both a gradual emergence of consciousness (Dennett and Kinsbourne, 1992; Overgaard et al., 2006; Seth et al., 2008) and the partial awareness hypothesis (Kouider et al., 2011; Windey et al., 2013; Windey and Cleeremans, 2015). The observed interaction between feature type and mask type shows that the temporal emergence of visual awareness is highly flexible (see also, Overgaard and Mogensen, 2016) and can be more discontinuous or gradual depending on the context (here mask type). Hence, these results support our proposal that the transition between unconscious and conscious visual perception can be characterized as reaching an attractor state within distributed brain areas (Herzog et al., 2016). Depending on the context, the attractor state might involve different visual and non-visual areas coding specific types of stimulus features at either comparable (discontinuous transition) or dispersed (graded transition) time points of visual processing.

In contrast to our and others' (Kouider et al., 2011) predictions, lexical word features (lexical decisions) exhibited consistently across mask types (except for the pattern mask) higher thresholds than semantic word features (semantic decisions), although semantics is typically considered to be a higher representational level compared to lexical representations (McClelland and Rogers, 2003). Two mutually not excluding explanations of this unexpected finding are possible: (i) As pseudowords were created from real words by exchanging a few letters, a greater perceptual clarity is required to be able to successfully differentiate between words and pseudowords, in particular with similar masks. For semantic living/non-living decisions and the other tasks, the decision categories are better differentiated so that less perceptual clarity is needed. (ii) Awareness of word meaning might emerge earlier than awareness of lexicality (see also Anzulewicz et al., 2015) because semantic processing of the living/non-living dimension depends on an extended network of brain areas in fronto-parietal and occipital areas compared with lexical processing (Kiefer and Pulvermüller, 2012), which involves relative restricted areas in temporal cortex (Cohen and Dehaene, 2004). It is possible that activity within the larger semantic brain network accumulates faster within reverberating processing loops to reach the ignition level necessary for awareness (Noy et al., 2015).

# Context-Dependency of Slopes of the Psychophysical Functions: Emergence of Awareness within Features

In addition to feature threshold SOAs, we also analyzed the slopes of the fitted psychophysical function within each task: A shallow slope is taken to index a gradual transition from unawareness to awareness for a feature probed by a given task, whereas a steep slope indicates a discontinuous or even a sharp dichotomous transition (Koch and Preuschoff, 2007; Del Cul et al., 2009; Windey et al., 2013). Within the context of the partial awareness framework, one variant assumed an all-or-none emergence of awareness for single features, which should result in generally steep slopes (Kouider et al., 2011). According a second variant of the partial awareness framework, awareness of higher-level semantic features is assumed to be dichotomous resulting in relatively steeper slopes in contrast to the gradual emergence of awareness for lower-level perceptual color features leading to shallower slopes (Windey et al., 2013; Windey and Cleeremans, 2015). We further assumed that slopes for higher-level word and semantic features would be shallower for the random letter and word masks compared with pattern and false font masks, because random letter and word masks should specifically delay the processing of lexical and semantic word features.

In contrast to our or others' expectations, we found the steepest slopes for the absent/present task independent of masking suggesting a rather discontinuous emergence of awareness. Slopes for the higher-level letter, lexical and semantic discrimination tasks were shallower than for the absent-present task and did not differ from each other. This suggests that for features more complex than the detection of stimulus presence ('energy') awareness seems to evolve more gradually.

When interpreting the slope parameter of a psychophysical function the following issues must be considered: Slopes cannot be classified as being "steep" or "shallow" without any reference, e.g., another psychophysical function with a different slope. However, in our view this holds true for any investigation in perceptual performance. We believe that any psychophysiological process evolves in time due to the functional properties of neurons within their networks. From that perspective, any perceptual process can be described with a transition function, which normally is sigmoidal. This formally contradicts the concept of a true step function, which might be the underlying concept of an "all-or-none" mechanism in perceptual awareness. However, a sigmoid function, where the transition between performance at chance level and performance without error

happens within a few milliseconds, as for the absent/present task in the present study, looks like a step function. Such a very brief transition phase, although formally different from a true step function, could indicate a discontinuous emergence of awareness.

We are also aware that the slope value is related to the resolution of the abszissa and therefore, the first is influenced by the latter. Simulations with idealized data showed that using the present step size of 6.7 ms for the stimulus-mask SOA the steepest slope obtainable was 1.33. Decreasing the step size of the SOA to 2 ms (which is difficult to achieve with standard CRT monitors) would increase the steepest obtainable slope to 4.0. With respect to our results (steepest slopes in absent/present condition, mean slope value 0.759, Supplementary Table S3) we would therefore not expect a substantial change in the relation of the slope pattern in the present study. Of course, the magnitude of the slope can be characterized with more precision, when the step size of the SOA variation are smaller.

The pattern of slopes (steeper slope for energy than for higher-level features) was incompatible with our expectations and with earlier findings (Windey et al., 2013). We speculate that slopes of psychophysical functions and thus the transition from unawareness to awareness for a given feature are highly flexible and depend among other factors on the dimensional continuity of the features probed in the task (see also, Windey and Cleeremans, 2015): Dimensional continuity between blue and red hues, but also between living and non-living concepts is more continuous compared with stimulus absence vs. presence or with integer number magnitude, which are both intrinsically discontinuous. It is conceivable that awareness of a feature on a continuous dimension emerges rather gradually compared with features on discontinuous dimensions, which give rise to sharp transitions between unaware and aware states. As outlined in the introduction, the attractor state related to visual awareness might be reached from previous intermediate states of unconscious processing through relatively sharp (discontinuous feature dimension) or smooth (continuous feature dimension) transitions resulting in a more dichotomous or gradual emergence of awareness. Of course, this explanation is only tentative and has to be tested in future studies. Also contrary to our expectations, the slopes for lexical and semantic features were not modulated by the masks, i.e., they were not shallower for random letter and word masks, which are assumed to more strongly delay processing of these features compared with pattern and false font masks. It must remain open, whether the slopes for each feature type are relatively invariant and generally insensitive for influences of the different masks. Alternatively, it is possible that modulation of the slopes by the masks can be found under stimulation conditions, in which the influence of the mask is putatively stronger, for instance when the target stimuli are presented at lower contrasts.

#### REFERENCES

Ansorge, U., Kunde, W., and Kiefer, M. (2014). Unconscious vision and executive control: how unconscious processing and conscious action

# CONCLUSION

By using a psychophysical approach, the present study shows that the temporal transition between unaware and aware states is modulated by the context: Depending on the mask, word features (except energy) contribute almost simultaneously to the percept suggesting a more discontinuous transition from unawareness to awareness or had quite dispersed threshold SOAs indicating a highly temporally graded transition. A high similarity between the probed feature and the mask appears to be an important factor to induce a graded emergence of awareness across stimulus features. Analyses of slopes of the fitted psychophysical functions also indicated that the temporal emergence of awareness of single features is variable and might be influenced by the continuity of the feature dimensions. The present work thus suggests that the emergence of awareness is neither purely gradual nor dichotomous, but highly dynamic and context-dependent at both the stimulus and feature level. Future studies could combine psychophysical measurements of thresholds and slopes with subjective ratings of awareness in the search for convergent evidence regarding the temporal dynamics of visual awareness.

# ETHICS STATEMENT

The procedures of this study have been proved by the Ethics Committee of Ulm University. Participants received written information about the experimental procedure and signed a written informed consent.

# AUTHOR CONTRIBUTIONS

MK and TK developed the study concept and were involved in data collection. TK performed the data analyses in cooperation with MK. MK and TK drafted the manuscript, provided critical revisions and approved the final version for submission.

# ACKNOWLEDGMENT

This research was supported by a grant of the German Research Foundation (DFG) to MK (Ki 804/3-2).

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg. 2017.00315/full#supplementary-material

control interact. Conscious. Cogn. 27, 268–287. doi: 10.1016/j.concog.2014. 05.009

Anzulewicz, A., Asanowicz, D., Windey, B., Paulewicz, B., Wierzchon, M., and Cleeremans, A. (2015). Does level of processing affect the transition from

unconscious to conscious perception? Conscious. Cogn. 36, 1–11. doi: 10.1016/ j.concog.2015.05.004


unconscious perceptual effects. Cognition 101, 43–79. doi: 10.1016/j.cognition. 2005.06.006


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Kiefer and Kammer. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# In and Out of Consciousness: How Does Conscious Processing (D)evolve Over Time?

Jaan Aru1, 2 \* and Talis Bachmann<sup>1</sup>

*<sup>1</sup> Department of Penal Law, School of Law, University of Tartu, Tartu, Estonia, <sup>2</sup> Chair of Data Science, Institute of Computer Science, University of Tartu, Tartu, Estonia*

Keywords: consciousness, NCC, microgenesis, time course of consciousness, transitions of consciousness

Mental information processing includes both unconscious and conscious modes and there are transitions between these two. The content of subjective experience can emerge from preconscious content, but an opposite process of loss of conscious content or its decay from subjective experience is an inevitable reality as well. Both ways of transition on the border between unconscious and conscious processing are ubiquitous. But how does this transition unfold over time? While there is quite some literature on whether conscious perception is all-or-none or graded (Sergent and Dehaene, 2004; Overgaard et al., 2006), we ask the complementary question: how does conscious perception change and evolve over time? While in phenomenological approaches in the philosophy of mind all-embracing temporal perspective has been acknowledged as crucial in understanding consciousness (e.g., anticipation, present, and retention in Husserl, 1928), in experimental paradigms only narrow temporal slices have been typically examined.

#### Edited by:

*Morten Overgaard, Aarhus University, Denmark*

#### Reviewed by:

*Giorgio Marchetti, Mind, Consciousness, and Language Research Net, Italy*

#### \*Correspondence:

*Jaan Aru jaan.aru@gmail.com*

#### Specialty section:

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology*

Received: *25 October 2016* Accepted: *18 January 2017* Published: *02 February 2017*

#### Citation:

*Aru J and Bachmann T (2017) In and Out of Consciousness: How Does Conscious Processing (D)evolve Over Time? Front. Psychol. 8:128. doi: 10.3389/fpsyg.2017.00128*

We will first focus on the knowledge about the transitions gained from studying brief visual stimuli. According to the microgenetic tradition mental content does not emerge instantaneously, in an all-or-none manner (review: Bachmann, 2000). Instead, conscious content arises as a gradual process of formation where the initial transition (there was no content and now there is some content) grows over to a time consuming process where subjective phenomenal content of the same intentional object matures by acquiring systematically more qualities to the preceding version of the percept. This intentional object might be a visual object, scene, memory representation etc. These microgenetically developing attributes or characteristics include subjective clarity, subjective contrast, subjective fragmentariness/exhaustiveness, coarseness/detail, subjective stability, etc. (Bachmann, 2000, 2012). In other words, conscious experience of the content pertaining to the same intentional object changes considerably over time (See also Hegdé, 2008; Breitmeyer, 2014; Pitts et al., 2014).

On the other hand (especially when brief objects typical to most of the experiments are presented) a certain experience with its phenomenal subjective content sooner or later disappears from consciousness by an analogous, but reversed gradual process—a kind of "anti-genesis" (Bachmann, 2000). **Figure 1** illustrates the notion of microgenesis with its formative and disformative stages. Note that this figure is an abstraction based on the empiricial research described in Bachmann (2000). However, we hope that provocatively drawing this time course inspires researchers to investigate how it exactly looks like.

This simple conceptualization prompts surprisingly many old and new questions.


FIGURE 1 | The function of conscious experience evolving over time. (A) Microgenesis of perception with its formative and disformative stages. The image is based on empirical research reviewed in Bachmann (2000). Note the proposed asymmetric inertia of formation and disformation. (B) It is conceivable that the functions over time are different for two types of conscious experience: immediate iconic perception (blue line) and slower memory-based experience (yellow line; see point 7 below). How is this shape influenced by top-down factors like attention, prior knowledge or working memory? Do these factors lead to a quicker build up and higher clarity of conscious content (green line; see point 6 below)? The units on the Y-axis are arbitrary units to be worked out.

order estimation, speeded discrimination, and other experimental tasks are smaller than experimental estimates of the duration of immediate memory (Bachmann, 2000).


Now, an intriguing theoretical question appears: if one and the same stimulus-event is related to both, perceptual and immediate-memory microgenetic processes with concomitant two sets of NCC, should we then regard these NCC as different aspects of one NCC or a principally different, two, NCCs (Bachmann, 2015)?


transitions in and out of consciousness. Can one benefit from the knowledge gathered while studying brief visual stimuli to understand processes that unfold with other types of stimuli?

To understand this last question, let us list some examples about the transition into consciousness to illustrate the heterogeneity of this kind of transformation: remembering an item or idea as cued by external instruction or question; remembering an item or idea as ignited by intrinsic associative cues; having an insight; experience of an external sensory stimulus after its initial pre-conscious processing; experience of an already presented stimulus after focusing attention on it; becoming consciously aware of an intention (agency) to act after preconscious preprocessing of the action decision; becoming consciously aware of a different aspect (feature, attribute, property, quality) of a stimulus or scene after the preceding consciously aware experience of some other aspect(s); noticing the change in a change-blindness display; noticing the target in an inattentional blindness experiment; becoming consciously aware of the Gestalt content in Mooney face or Dalmatian dog types of image after an initially "meaningless" experience; becoming aware of the words within the sine-wave speech recording; reversal of binocularrivalry dominance in becoming aware of the suppressed stimulus; reappearance of the sensory afterimage. Is in all of these cases the emergence of conscious content gradually evolving over time?

There are also many examples for the transition out of consciousness: fading of the conscious percept; fading of iconic memory; loss of thought or imagery content; loss of explicitly experienced WM content; loss of conscious awareness of a stimulus after refocusing attention elsewhere; loss of a certain aspect (feature, attribute, quality) of conscious perception of a discontinued stimulation while other aspect(s) sustain; loss of conscious perception of a binocular-rivalry stimulus when it becomes suppressed; fading of perceptual content (e.g., color, spatial contrast modulation, luminance contrast step gradient) due to sensory adaptation; fading of afterimage. Is the fading of conscious content gradually evolving over time in all of these cases?

Gathering such a list leads yet again to interesting questions. For example, which transitions are reversible and which ones not? Loss of formed Gestalt content back to meaningless array of elements seems difficult; re-establishment of changeblindness/inattentional blindness seems impossible; after hearing the words within the sine-wave speech it is impossible to go back to hearing noise. What could this small set of non-reversibility tell us about the neural mechanisms of consciousness? The list

### REFERENCES


of these phenomena seems to suggest that conscious experience is heavily influenced by prior knowledge—once insightful knowledge about a particular stimulus is established, it is hard or even impossible to remove it.

More importantly, the variety of examples leads to the question whether there are general mechanisms and regularities underlying all of these phenomena. Do all these other types of transitions share some of the key features with the transitions happening in visual perception (**Figure 1**)? In visual perception it is relatively straightforward to "slice up" perception with techniques like visual masking (Bachmann, 1994; Bachmann and Francis, 2013), but even then studying the time course of visual perception is a time consuming and a difficult endeavor (Bachmann, 2000). Is it possible or even meaningful to try to do it with other types of transitions? How would one proceed with "slicing up" memory retrieval, Gestalt perception or insight formation? We do not have definitive experimental approaches, but we consider these questions to be important to put forth and to explore.

The present manuscript had a few aims: (1) we wanted to emphasize that conscious content evolves and changes over time, (2) we noted that the exact time course of how conscious content evolves over time is yet unknown and tentatively drew a time course to provoke more research in this direction, (3) we wanted to demonstrate that thinking about the time course of conscious processing prompts many interesting and intriguing questions, (4) finally we asked how general are such microgenetic regularities—do all kinds of transitions in and out of consciousness have gradual formation and disformation (**Figure 1**)? On the way we also seem to have stumbled on a few novel concepts applicable in studying dynamics of conscious experience—the formation/disformation (a)symmetry, reversibility, and the possibility of having two different NCCs for the same perceived object. We hope that some of these ideas and concepts are beneficial for unraveling the neural mechanisms of consciousness.

### AUTHOR CONTRIBUTIONS

TB conceived the initial ideas, JA expanded them, both JA and TB discussed the ideas and contributed to writing the manuscript

### ACKNOWLEDGMENTS

This work is partly supported by institutional Grant IUT20-40 from Estonian Ministry of Education and Research.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Aru and Bachmann. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# A Sensorimotor Signature of the Transition to Conscious Social Perception: Co-regulation of Active and Passive Touch

Hiroki Kojima<sup>1</sup> \*, Tom Froese2,3, Mizuki Oka<sup>4</sup> , Hiroyuki Iizuka<sup>5</sup> and Takashi Ikegami<sup>1</sup>

<sup>1</sup> Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan, <sup>2</sup> Institute for Applied Mathematics and Systems Research (IIMAS), National Autonomous University of Mexico (UNAM), Mexico City, Mexico, <sup>3</sup> Center for Complexity Sciences (C3), National Autonomous University of Mexico (UNAM), Mexico City, Mexico, <sup>4</sup> Graduate School of Systems and Information Engineering, University of Tsukuba, Ibaraki, Japan, <sup>5</sup> Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Japan

It is not yet well understood how we become conscious of the presence of other people as being other subjects in their own right. Developmental and phenomenological approaches are converging on a relational hypothesis: my perception of a "you" is primarily constituted by another subject's attention being directed toward "me." This is particularly the case when my body is being physically explored in an intentional manner. We set out to characterize the sensorimotor signature of the transition to being aware of the other by re-analyzing time series of embodied interactions between pairs of adults (recorded during a "perceptual crossing" experiment). Measures of turn-taking and movement synchrony were used to quantify social coordination, and transfer entropy was used to quantify direction of influence. We found that the transition leading to one's conscious perception of the other's presence was indeed characterized by a significant increase in one's passive reception of the other's tactile stimulations. Unexpectedly, one's clear experience of such passive touch was consistently followed by a switch to active touching of the other, while the other correspondingly became more passive, which suggests that this intersubjective experience was reciprocally co-regulated by both participants.

#### Edited by:

Morten Overgaard, Aarhus University, Denmark

#### Reviewed by:

Glenn Carruthers, Charles Sturt University, Australia Kim Mouridsen, Skejby Sygehus, Denmark

\*Correspondence: Hiroki Kojima kojima@sacral.c.u-tokyo.ac.jp

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 31 October 2016 Accepted: 25 September 2017 Published: 13 October 2017

#### Citation:

Kojima H, Froese T, Oka M, Iizuka H and Ikegami T (2017) A Sensorimotor Signature of the Transition to Conscious Social Perception: Co-regulation of Active and Passive Touch. Front. Psychol. 8:1778. doi: 10.3389/fpsyg.2017.01778 Keywords: embodied cognition, social interaction, intersubjectivity, agency detection, direct perception

# INTRODUCTION

There is growing acceptance that humans develop social awareness much earlier than had long been assumed, including suggestions of a capacity for false belief understanding even in the case of preverbal infants (Baillargeon et al., 2010). Moreover, there is compelling evidence for the ability to recognize and respond to another's presence appropriately even in the first months after birth (Reddy, 2008). In order to account for these findings it is necessary to expand the explanatory framework that has traditionally been employed by developmental psychology and social cognition research (Hutto et al., 2011). The root developmental form of social understanding is starting to be conceived as more interactive and perceptual than just detached and cognitive. For example, before infants can begin to theorize about another person's minds from a third-person perspective,

or to imagine what it is like to be them from a first-person perspective, they arguably can already experience another's presence from the second-person perspective, that is, in the context of mutual engagement (Reddy and Morris, 2004). On this expanded view, the classic mechanisms of social cognition build on and are preceded by embodied forms of social understanding, such as those realized by neural resonance in motor areas (Gallese, 2007) and by the dynamics and experience of social coordination itself (Froese and Gallagher, 2012). However, although the "higher-level" cognitive capacities develop on the basis of the "lower-level" embodied capacities, they do not replace them. Development is characterized by a diversification of social capacities, and we learn to deploy a combination of both embodied and cognitive skills in a context-sensitive manner (Fiebich et al., 2016).

This reappraisal of the development of social awareness is consistent with an ongoing reevaluation of social understanding in adults. It is already widely accepted that our capacities for theorizing about and simulating other minds are not mutually exclusive but complementary (Frith and Frith, 2012), for example in terms of dual processing (Bohl and van den Bos, 2012). And, as has been argued by phenomenological, embodied, and enactive approaches, it makes sense that the mechanisms of reflective social cognition are in turn complemented by interaction dynamics and perceptual experience (Wiltshire et al., 2015). In contrast, it had long been assumed that social understanding must be mainly cognitive because other minds are fundamentally unobservable (Premack and Woodruff, 1978), a claim which follows from the traditional theory that we only perceive physical properties of the world. However, this position is problematic because it underestimates the scope of perceptual experience and is therefore forced to overburden the role of intellectual activity (Gallagher, 2008). When we attend to how we relate to other persons in our everyday life, we realize that the ability to simply perceive others as subjects guided by their own minds remains our primary means of social understanding and that uncertainties can often be resolved in interaction (Gallagher, 2012).

This phenomenological insight is particularly evident when adopting a second-person perspective, that is, when we are engaged in mutually responsive interaction in which another person appears as an immediate "you" rather than as a detached he or she. As long as there is nothing strange about the situation, it is possible to perceive a large extent of the other's psychological states in the ways in which they act in the world and especially in how they interact with us (Ratcliffe, 2007). In acknowledgment of ecological theories of perception, this capacity to perceive the other's mind without the need for explicit theorizing or imagining is often referred to as the "direct perception" of other minds (Froese and Leavens, 2014). As the direct social perception thesis has become more widely accepted in cognitive science, the debate has shifted from clarifying the phenomenology to elucidating its underlying mechanisms (Michael and De Bruin, 2015). Many researchers still appeal to mechanisms of inference and/or simulation that are restricted to an individual, but there is also a growing recognition that embodied interaction itself plays a role in such social perception.

# Toward a Relational Hypothesis

We can motivate this interactive approach to the phenomenon of direct social perception from both phenomenological and developmental perspectives, which converge on basically the same relational hypothesis: the conditions underlying social awareness cannot be reduced to an isolated individual; it is rather the other's intentional and attentional engagement with one's self that helps to constitute one's awareness of the other's presence.

Reddy (2008) has argued for this hypothesis at length from a developmental perspective. She points out that if the other's mindedness is directly expressed in their embodied interaction with the world, and especially in their engagement with one's self, then all an infant needs is a predisposition to attend to and pick up the other's embodied manifestations of mentality. And the infant is surrounded with such manifestations from the moment of birth, in particular the unavoidable manipulations of its own body when it is picked up and carried, fed, cleaned, bathed, and so forth. Accordingly, she contends that infants' first experience of others is constituted by being the object of their caretakers' attention (Reddy, 2003, 2005).

The relational hypothesis has also been defended by the phenomenological tradition of philosophy, especially by Merleau-Ponty (1960/1964a,b). He argued that infants' awareness of other minds develops from within a dyadic state of consciousness, and that our most basic awareness of other minds even as adults is constituted by passive touch. He specifically focused on touch, rather than other perceptual modalities, because the experience of touching one's own body exemplifies how we can perceive embodiment in both is objective and subjective aspects: while attending to how my right hand is touching my left hand I can experience myself as the active subject of that action, but I can also switch my attention to the left hand as it is being touched and thereby experience myself as the passive object of that action. In this way we have direct experiential insight into how it feels to be the target of someone's intentional touching (in this case by ourselves).<sup>1</sup>

It is notable that Merleau-Ponty appeals to an element of passivity in his account of intersubjective experience, despite his usual emphasis that perceiving is an active process. Similar to how one needs to adopt a passively receptive attitude to feel one's left hand as being intentionally touched by the active right hand, a moment of passivity in one's interaction with another person helps to make the other's activity appear as intentionally originating from the other. In other words, by temporarily withholding one's self from being the active center of reference of one's experience of the interaction one allows the other's contribution to become present as originating from another autonomous center of reference.

<sup>1</sup>Other perceptual modalities do not offer this possibility, e.g., of directly seeing myself looking at my own body, which is why they are argued to be secondary. However, it must be acknowledged that we can nevertheless perceive the presence of other subjects visually and even do so while they are not attending to us. We return to the problem of generalizing the capacity of social perception in the section "discussion." For the moment we simply note that it is still the case that when one is aware of being looked at one tends to experience a particularly strong presence of another subject.

Miyahara (2015) has identified an unresolved issue in Merleau-Ponty's account of passive touch as the foundation of intersubjectivity. He notes that we can come into tactile contact with many kinds of objects, and merely undergoing a tactile sensation is not sufficient by itself to cause a transition to a conscious experience of another subject's presence. For example, if an apple happens to fall on my head while I walk under an apple tree, I will passively undergo the tactile sensation of its impact. But that experience is not a case of passive touch: experiencing a mere tactile impact is not the same as a feeling of being touched by someone. Miyahara (2015) suggests that a necessary condition for the latter type of experience is a more continuous form of actual or potential contact. For example, if my arm is continuously stroked by a branch of the tree while it is blown by the wind, I may have the creepy sensation that the tree is touching me.<sup>2</sup>

Nevertheless, we suggest that an illusion of passive touch will normally quickly break down, for example if I occasionally interrupt my passive mode by actively responding to the stimulation. This will reveal that the movements of the branch are not contingent on my actions and are therefore themselves not susceptible of passive touch. Accordingly, we propose that the reciprocity of social interaction, for example during coordinated handshakes, is another essential factor in the constitution of awareness of other minds. Reciprocity also implies that the participants of the interaction share their social awareness, such that one's awareness of the other is at the same time matched by the other's awareness of one's self, i.e., there is a common sense that we are aware of each other.

Given these considerations we propose a relational hypothesis of social perception: one's awareness of another mind emerges in a context of co-regulated interaction and is preceded by a passive period of being the autonomous other's object of attention. We set out to empirically evaluate this hypothesis in the tactile modality because, as we have discussed, this has been argued to be the primary modality for direct social perception in both developmental and phenomenological terms. Given that the hypothesis is sufficiently general so as to be applicable to infants and adults, we base our analysis on a paradigm involving pairs of adult participants because this enables us to evaluate subjective reports of the quality of their experience of each other.

#### The Perceptual Crossing Paradigm

A popular approach to studying the tactile sensorimotor interaction dynamics associated with recognition of the presence of another person was pioneered by Lenay and colleagues (Lenay et al., 2006; Auvray et al., 2009; Auvray and Rohde, 2012). This so-called "perceptual crossing" paradigm was designed to test whether adults are sensitive to the responsiveness of another's tactile stimulations under the controlled conditions of a minimalistic virtual reality environment, an approach which in turn was inspired by the double TV monitor paradigm that Murray and Trevarthen (1985) designed to demonstrate that infants are sensitive to social contingency during visual interaction with their mothers. The general aim of these paradigms is to allow the controlled study of social interaction dynamics in real time.

In the perceptual crossing paradigm two players in separate rooms are asked to locate each other in a virtual environment, a line that is invisible to them, by using a simple haptic interface. No other form of interaction is possible. They can move their avatars using a computer mouse and they receive a tactile stimulation to their hand whenever their avatar overlaps with a virtual object. There are three objects that can be encountered by a player: (1) the other player's avatar, (2) the other avatar's "shadow" – an object that simply copies the other avatar's movements at a fixed distance, and (3) a static object (see section "Materials and Methods" for more details). All three objects have exactly the same size, which means that overlapping with them gives rise to the same "on" tactile sensation. Nevertheless, the objects afford different forms of interaction: only two out of the three can move, and out of these two only the other's avatar can respond to being touched because only this situation of perceptual crossing means that both participants receive tactile feedback at the same time. In other words, the shadow object serves as a special kind of distractor because even though it moves identically to the other participant's avatar, only the latter is potentially responsive. Participants were asked to click a button on the computer mouse to record the moment whenever they judged that they were currently interacting with the other's avatar. They knew that in addition to another's avatar they could also encounter two other objects, but they were not informed that one of those objects copied exactly the movements of the other's avatar. No feedback was provided about click correctness.

The original study by Auvray and Rohde (2012) and several replications found that participants were able to successfully locate each other in the virtual space, with most clicks being on target. However, surprisingly, participants seemed unable to consciously distinguish their partner's avatar from the moving distractor object: the probability of clicking after making contact with the partner was not significantly different from the probability of clicking after making contact with the other's shadow. Instead, the correct social judgments could be explained by the relative stability of mutual interaction. Since both participants actively look for each other, they tend to continue interacting when they happen to make mutual contact, while tending to move away from overly stable (likely caused by a static object) and overly unstable (likely caused by a non-responsive object) situations. The solution to the task, i.e., a participant's sensitivity to social contingency, was thus interactively realized at the collective level of description.

Froese et al. (2014a) implemented a team-based variation of the perceptual crossing paradigm where the pairs of players were asked to cooperate. They found that most teams were successful; on average players were significantly more likely to click in response to a stimulation from the other's avatar than any other object. In contrast to previous studies this difference remained significant even at the level of individual

<sup>2</sup>This example shows that the direct social perception thesis is not committed to the infallibility of experience. Even if I have the experience that someone is touching me, this may actually turn out not to be the case. On the other hand, it is also not the case that fallibility entails that perceptual experience never directly involves its object of perception. The details of this philosophical issue go beyond the scope of this paper (see Beaton, 2013).

conditional probabilities. They also found that trials in which both players correctly identified each other led to significantly higher perceptual clarity ratings than when only one player clicked correctly or a player clicked wrongly. Such situations of clear perceptual awareness and joint success tended to be preceded by elevated levels of turn-taking (TT) interaction, which was interpreted as confirming the hypothesis by Froese and Di Paolo (2011) that co-regulation of interaction gives rise to a distinctively social kind of perceptual experience. Clicks in jointly successful trials were most likely to occur within seconds of each other, which is consistent with the possibility that a dyadic awareness of each other was shared between participants.

This dyadic nature of the interaction was further supported by a re-analysis of the sensorimotor time series by Zapata-Fonseca et al. (2016). They revealed that the clustering statistics (Allan factor) of discrete movement events was characterized by fractal scaling, which highlights that the social interaction process operates in a distributed fashion across multiple timescales. Moreover, the complexity matching, defined as the interpersonal similarity between these scaling laws of the clustering statistics, was significantly more pronounced in real pairs of participants as compared to surrogate dyads. This confirms the multi-scale distributed character of real-time social coordination, and extends previous complexity matching results from dyadic verbal conversations (Abney et al., 2014) to embodied interaction dynamics. The finding will not come as a surprise to developmental psychologists who have long recognized the communicative and dialogical potential of even basic embodied interaction (Trevarthen, 1979; Tronick, 1989; Stern, 1998; Reddy, 2008).

A diachronic re-analysis of that original study by Froese et al. (2014b) confirmed the expected relevance of the perceptual crossing paradigm for understanding the development of social perceptual awareness. It seems that the unfamiliar experimental setup forced the adults to implicitly re-learn the embodied skill of social perception, which may provide researchers with an opportunity to study a recapitulation of its original development in adults. In particular, a trial-by-trial qualitative analysis of participants' free-text subjective descriptions turned out to be roughly consistent with Reddy's (2003) proposal about the qualitative stages involved in being the other's object of attention. Initially, participants were more likely to report a self-centered awareness of the other's presence via being the object of their attention, such as a feeling of being explored by the other, which later on became complemented by descriptions of more dyadic and complex forms of attention, such as exchanging specific patterns of stimulation and spontaneously adopting roles of leader and follower. This trial-by-trial process of implicit learning was accompanied by a general increase in the frequency of clear perceptual awareness scale (PAS) scores and joint clicking success.

The diachronic analysis of phenomenological descriptions by Froese, Iizuka, and Ikegami was consistent with the primacy of passive touch in that descriptions of the other's attention to the self tended to precede more complex forms of social engagement. However, their qualitative analysis cannot tell us whether the moment preceding recognition of the other was objectively characterized by a relative increase in the passive reception of tactile stimulation caused by the other's movements. Neither does it allow us to verify if a longer duration of passive touch as such entails a clearer awareness of the other's presence, or whether coordinated sensorimotor interaction is more important. In this study we addressed these challenges based on a more refined analysis of the sensorimotor data generated by that experiment.

# MATERIALS AND METHODS

The dataset we re-analyzed was originally reported by Froese et al. (2014a). We therefore only briefly describe the participants and the experimental setup as far as it is necessary to understand the data. Then we introduce the techniques of analysis we applied.

# Participants

Participants were healthy volunteers recruited from acquaintances at the University of Tokyo and at the University of Osaka (N = 34). There were 25 Japanese nationals, the rest were from various countries. Six were female. The mean age was 29 years. Teams of participants were created as volunteers became available.

The study protocol was approved by the local ethics research committee of the Graduate School of Information Science and Technology, University of Osaka, and by the local ethics research committee of the Graduate School of Arts and Sciences, University of Tokyo, and has been performed in accordance with the ethical standards laid down in the Declaration of Helsinki. All of the participants gave their written informed consent before taking part in the study.

# Experimental Setup

In Froese et al.'s (2014a) version of the perceptual crossing paradigm, two adults are placed in distinct locations such that they cannot perceive each other; their sight is blocked and they wear noise-cancelling headphones (**Figure 1**). Their only manner of making contact is via a simple interface consisting of a trackball that records horizontal movements and a hand-held vibration motor that is either on or off. The trackball is operated with the dominant hand while the motor is held in the other hand. Their movements control the motions of an avatar located in an invisible 1D virtual environment (**Figure 2**). The motor continuously vibrates whenever their avatar overlaps with another object in the virtual space. Position and sensor data were recorded every 10 ms (100 Hz).

The task given to pairs of players was to form a team and to help each other to find each other in the virtual space. They are to click once using the trackball (and only once per trial) in order to signal to the experimenters when they become aware of interacting with the other player; the other player is not aware of the click. No feedback is provided during the experiment. Each pair can interact in a sequence of 15 trials, each with a duration of 60 s. After each trial the experience of the players is evaluated in several ways if they happened to click in that trial. In particular, they were asked to rate the clarity of their experience of the other's presence at the moment of their click on the basis of a

trackball that controls the linear displacement of their virtual avatar, and a hand-held haptic feedback device that vibrates at a constant frequency for as long as a player's avatar overlaps with another virtual object and remains off otherwise. Three small lights on each desk signal the start, halftime (30 s), and completion of each 60-s trial. Figure originally published in Froese et al. (2014a).

social version of the PAS, based on the PAS that was proposed by Ramsøy and Overgaard (2004). In this scale 1 means having had no experience, 2 means having had an ambiguous experience, 3 means having had an almost clear experience, and 4 means having had a clear experience (**Table 1**).

## Analysis of Sensorimotor Trajectories

First, we replicated and extended previous work on measuring the amount of interpersonal coordination that precedes clicks.

The degree of TT was calculated following the method proposed by Froese et al. (2014a). At each time step we classified the state of each player's behavior in binary terms as either moving (1) or non-moving (0) by evaluating his or her trackball movement (we will refer to these binary movement time series as B1 and B2 for participants Pa and Pb, respectively). Movement was considered to have taken place whenever the change in avatar position dx from one time step to the next was bigger than an 8th of the avatar's length (i.e., 4/8 = 0.5 so that if dx > 0.5, 1, else 0).



Since avatar positions tend to fluctuate during a player's "turn" we chose to set a lower limit to the duration of movement pauses so as not to accidentally end up with a turn being divided into microturns. Thus, we only set movements to 0 if there was no motion over at least 50 consecutive time steps (500 ms), otherwise they remain set to 1.

In order to determine the differences between players' activity we applied the logical "Not-And" operator to their movement time series, which resulted in a time series of activity differences D (i.e., D = B1 Not-And B2). Then, we assigned to each participant their active contribution of this exchange by applying the logical "And" operator and summing the result [i.e., C1 = sum(B1 And D); C2 = sum(B2 And D)]. The overall TT performance for a given time period was then calculated by multiplying the player's active contributions. This multiplication means that one-sided situations, in which one player is continuously active while the other is continuously passive, get low TT scores. Finally, we normalized the outcome such that the TT score TT = (4 <sup>∗</sup> C1 <sup>∗</sup> C2)/T 2 , where T is the number of time steps. The range of TT is therefore [0, 1], with 0 representing a complete absence of TT interactions and 1 representing a perfect exchange of periods of activity and passivity between the subjects. We analyzed the TT in the 10 s preceding a click (or correspondingly less when the click occurred within the first 10 s of a trial).

The measure of TT interaction that was proposed by Froese et al. (2014a) can tell us whether players were exchanging periods of activity and passivity in an orderly manner, but it does not say much about the similarity of the patterns of activity that were being exchanged. Given that interpersonal synchrony is widely considered to reflect psychological connectedness, we applied measures of movement synchrony, namely cross-correlation (CC) and windowed cross-lagged regression (WCLR). CC is a common measure but can be confounded by auto-correlation, which may lead to inflated measures of interpersonal synchrony, a problem which is avoided by WCLR (Altmann, 2011). High levels of synchrony can mean that both players are moving similarly at the same time or with a lag. We analyzed periods of 10 s with a time lag in the range of [−5, 5] seconds, which means that clicks occurring during the first 15 s of a trial were excluded from CC and WCLR analysis. In total, 28 clicks had to be excluded. We also used WCLR to calculate the windowed time delay yielding the largest CC value, which gives an indication of the most relevant timescales in which synchrony can be measured.

Second, we looked more specifically at the influence of players' movements on each other's tactile sensations preceding a click using a measure known as local transfer entropy (TE). This analysis of the whole sensorimotor loop is a significant methodological advance because previous time series analyses of perceptual crossing have only focused on movement by itself, thus leaving the interdependency between movement and sensation underlying meaningful perception (Noë, 2004; Mossio and Taraborelli, 2008) unexamined. TE was originally proposed by Schreiber (2000), and this measure can capture the directional influence from one time series to the other time series. This TE can be also formulated in temporally local form (Lizier et al., 2008), which allows us to calculate TE for specific segments of a time series.

Transfer entropy uses a joint probability distribution from two time series, and to calculate this distribution function empirically from the time series, we need to discretize the time series. The simplest way is to convert the movement time series to a binary sequence is in terms of the directional change of movement, i.e., we label the point in time as a state 1 when it changes, otherwise we turn it into a state 0. The sensory time series was also converted to a binary sequence, i.e., when the haptic feedback turns on or off we label the point in time as a state 1, otherwise it is set to 0 to mark the absence of a change in sensor state. In order to determine the most important timescale of the time series, first we calculated TE by utilizing the whole trial while adjusting the down-sampling rate. We found there was a peak of TE around 50 ms from the movement data to the sensory input data. We therefore took 50 ms as the characteristic timescale and used it for the further analysis of the local TE.

Given that we are interested in determining the sensorimotor signature of social awareness, we related these objective measures with the subjective PAS ratings of the clarity of the other's presence. In particular, we excluded clicks that were not reported to have been associated with an experience (i.e., PAS 1 or no PAS report), and restricted the data to ambiguous, almost clear, and clear experiences (i.e., PAS 2, 3, and 4, respectively). In addition, we did not further discriminate between the clicks that are associated with these conscious reports in terms of their objective correctness since we were interested in studying the general conditions of the transition to a social experience rather than to a veridical social experience per se. The final dataset consisted of 101, 122, and 143 clicks associated with reports of a PAS score of 2, 3, and 4, respectively. Out of these 366 clicks 321 correctly identified the other's avatar.

### Statistical Analysis

In order to analyze the relationship between the PAS and the movement coordination measures (TT, CC, and WCLR), first we averaged those movement measures with the same PAS, and applied one-way ANOVA followed by post hoc Bonferroni test.

In the analysis of local TE, we used Welch's t-test to examine whether the average TE were different before and after a click.

# RESULTS

Players' activity during a typical trial is shown in **Figure 3**. Note the extended period of interpersonal interaction in the first half of the trial, followed by nearly instantaneous clicks by both players.

This is followed by disengagement, then a short interaction with their respective static objects, and finally re-engagement just before the end of the trial.

# Qualitative Analysis of Movement Coordination

We can use the example trial shown in **Figure 3** to illustrate the CC and WCLR measures (**Figure 4**). It can be seen that CC greatly overestimates the amount of movement synchrony, while WCLR picks out only a few temporal regions. For example, there is a bright blue patch from x = 15 to 20 s for time lags of around −3 s. This tells us that during the preceding 10 s periods, starting from 5 to 15 s and ending during the period of 10–20 s of that trial, player Pb leads Pa with a delay of around 3 s. If we check this result against what is happening during that time in **Figure 3**, we can see that indeed player Pb (blue line) leads Pa (red line) beginning around 8 s by inducing the latter to also start oscillating. In the 10 s preceding the clicks, **Figure 3** shows that the direction of influence has become reversed, with Pa staring to oscillate from around 25 s and then pausing, while Pb continues to oscillate until pausing around 30 s. If we compare this with **Figure 4**, we see some bright blue bands following the clicks for time lags of around 2 s, which suggests that in the seconds preceding the clicks Pa's behavior leads Pb's behavior.

Nevertheless, it can also be observed that the WCLR method may be confounded when the players happen to move similarly but without interacting directly. For example, it turns out that the highest values in **Figure 4** are given for the period from around x = 42 to 50 s for lag times of around 0.5 s, even though **Figure 3** reveals that in the corresponding period starting from 32 s onward the players had already separated and just happened to move in roughly similar ways, with Pa slightly leading Pb but without direction interaction. However, we do not arbitrarily want to exclude such cases of behavioral coordination because they may still tell us something meaningful about the quality of the interaction. After all, one possible reason why these two players continued to move similarly even after spatially disengaging is that they had already become entrained during the first half of the trial.

We note that TT interaction and synchrony can both give high values for a trial when the players exchange periods of activity and passivity whereby that activity is similar in form, too. But they can also be mutually dissociated in other cases. As illustrated in **Figure 5**, players can exchange periods of activity and passivity whereby that activity itself does not have much resemblance (high TT and low WCLR), and players can greatly overlap in their activity but still share a lot of similarity in their movements (low TT and high WCLR). Here we applied the measures to the 10 s preceding a click, and we calculated the WCLR value to be the maximum value from a range of window time lags [−5 s, 5 s].

# Quantitative Analysis of Movement Coordination

Turn–taking interaction and movement synchrony could spontaneously emerge from the interaction dynamics without necessitating any explicit intention to coordinate behaviors or awareness that this is in fact occurring (Froese et al., 2012). In other words, conscious experience of social interaction cannot be reduced to objective measures of coordination; both subjective and objective aspects must be taken into account in an integrated manner.

Here, we compared each movement coordination measures with different PAS ratings (CC: 0.27 ± 0.01, 0.30 ± 0.01, 0.31 ± 0.01; WCLR: 0.084 ± 0.005, 0.095 ± 0.005, 0.11 ± 0.01; and TT: 0.15 ± 0.01, 0.18 ± 0.02, 0.23 ± 0.01, with PAS 2, 3, and 4, respectively), and we found that there was difference among different PAS ratings for all the objective measures (ANOVA, F(2,337) = 4.969, p < 0.01, F(2,337) = 3.792, p = 0.024, and F(2,364) = 8.178, p < 0.001 for CC, WCLR, and TT, respectively). Especially we found that all movement coordination measures accompanying with PAS 4 were higher than with PAS 2 (t(225) = 3.160, p < 0.0.01, t(225) = 2.728, p = 0.021, t(242) = 4.055, p < 0.001 for CC, WCLR, and TT, respectively, with Bonferroni correction) (**Figure 6**).

This is an indication that these measures are characterizing some part of the sensorimotor interaction signature of a clear experience of the other's presence. This seems to suggest that elevated levels of TT and movement synchrony are a common feature of the transition to social awareness during interactions.

### Timescales of Movement Coordination

In order to learn more about the timescales in which synchrony of movements is most pronounced, we used WCLR to calculate the delay giving the highest CC value for each trial. Since here we were not interested in which of the two players was leading the interaction, we took the absolute value of the lag times. We related these values with players' PAS ratings to determine whether some timescales are more relevant for explaining a clearer experience of the other's presence (**Figure 7**). If Miyahara (2015) is correct in suggesting that a continuous possibility of passive touch is essential for perceiving the presence of the other, clearer awareness should presumably be correlated with a longer timescale of interaction.

Averaged lag time with each PAS rating were evaluated as 2.7 ± 0.1, 2.5 ± 0.1, and 3.0 ± 0.1 s. We found that the average lag time was significantly different among different PAS ratings [F(2,337) = 4.395, p = 0.013], and especially we found that the lag time with PAS 4 was significantly longer than that with PAS 3 [t(240) = 2.915, p = 0.012, with Bonferroni correction]. Those findings suggest that phenomenologically more salient forms of movement synchrony are based on a longer timescale of interaction.

# Analysis of Direction of Influence

We used local TE to quantify directions of influence before and after a click at different timescales (a period of 10, 5, and 1 s before a click and 1, 5, and 10 s after a click). The periods of 1 and 10 s were chosen to coincide with the two cognitive scales of Varela's (1999) three scales of duration of the temporal horizon: (1) basic or elementary neural events (the "1/10" scale); (2) relaxation time for large-scale neural integration of cognitive or perceptual acts (the "1" scale); and (3) descriptive-narrative assessments of

FIGURE 3 | Example of time course of an illustrative trial (E1T1). Thick red and blue lines show the change in position of the avatars of players Pa and Pb, and red and blue dotted lines trace their "shadow" objects. Horizontal lines at y = –150, 150 correspond to the position of the player-specific static objects. Red and blue arrows indicate the time of clicking by Pa and Pb, which occurred practically instantaneously (within 0.05 s). After the trial players Pa and Pb reported that their experience of the other's presence at the moment of the click consisted in a "vague impression" (PAS 2) and a "clear experience" (PAS 4), respectively.

the situation (the "10" scale). The 5 s scale was chosen as an intermediate scale that is consistent with the lag times used for the synchrony analyses. It is also of interest as an expression of cognitive events taking place at the "1" scale: spontaneous speech in many languages is organized such that utterances last 2–3 s and short intentional movements (such as self-initiated arm motion) are embedded within windows of this duration (Varela, 1999). We return to this point in the discussion.

We denote S1 as the "self's" sensor time series and S2 as the "other's" sensor time series. Self and other are determined relative to the player who made the click. Who clicks first was not considered here. When one player touches the other, both sensors get activated at the same time (i.e., S1 = S2). They are only different (i.e., S1 6= S2) when either player touches the static objects or the shadows. Yet even though this means that **Figures 8**, **9** are expected to return similar values for situations of perceptual crossing, we separate M1/2 → S1 (**Figure 8**) and M1/2 → S2 (**Figure 9**) for the sake of clarifying active/passive touch differences.

In support of the hypothesis of passive touch we found that the transition to perception of the other's presence was characterized by passively received tactile stimulation (**Figure 8**). In general, there tends to be more influence from the other's movements (M2) on the self's sensations (S1) compared with the influence of the self's movements (M1). However, we did not find a continuous period of passive touch, a result that is consistent with our finding that significant lag times are <3 s. A statistically significant heightened influence of M2 on S1 was only observed in the second immediately before the click and only for high PAS ratings [t(214.0) = 2.83, p < 0.01, t(279.9) = 3.54, p < 0.001 for PAS 3 and 4, correspondingly).

This pattern is reversed after the click: moments of awareness rated as PAS 3 and 4 are followed by heightened influence of the self's movements on the self's sensations (i.e., comparatively

FIGURE 5 | Illustrative comparison between turn–taking (TT) and WCLR measures. For clarity we only plot the change in position of the avatars of players Pa and Pb (blue and red lines). Arrows mark the moment of player Pa's click. Taking turns by exchanging periods of movement and passivity does not entail a similarity between players' movement patterns (left, E7T6), and a similarity between players' movement patterns does not entail a TT interaction (right, E17T15). The values for TT and WCLR for the 10 s before Pa's click (shaded regions) are 0.49 and 0.015 for the example on the left and 0.0 and 0.18 for the example on the right, respectively. The lag times that yielded these maximum WCLR values were –4.4 (left) and –2.8 s (right).

<sup>∗</sup>p < 0.05, ∗∗p < 0.01, ∗∗∗p < 0.001 (with Bonferroni correction).

more TE from M1 to S1 compared to M2 to S1). Moreover, this difference in influence only becomes significant after a few seconds and remains so until at least 10 s [t(274.8) = 5.16, p < 10−<sup>6</sup> , t(263.2) = 4.47, p < 10−<sup>4</sup> , for 5 and 10 s with PAS 4, respectively).

# Analysis of Switch from Passive to Active Touch

This post-click reversal in the flow of influences was unexpected. We considered two plausible explanations. On the one hand, this period of self-generated activity could be an example of reciprocity, in which the self now tries to make its presence clear to the other by providing them with an opportunity for undergoing passive stimulation in return, which would incidentally also stimulate the self's own sensor due to the situation of perceptual crossing. However, an alternative possibility, which is more in line with the illustrative trial shown **Figure 3**, is that a click marks the end of a bout of close interaction followed by a period of temporary disengagement, in which the movements of the other participant play only a diminished role for the self's sensations. We therefore redid the analysis shown in **Figure 8**, but this time focusing on the TE from the self to the other's sensations. The aim is to verify if the self's movements comparatively increase their influence on the other's sensations or not (**Figure 9**).

We found that in the periods leading up the self's click the other's movements dominated the other's own sensations (M2 → S2), which is to be expected if the self is mostly passive during the transition to social awareness (M1) and the other is actively moving (M2). After a click the situation becomes more complex. Following moments of clear awareness (PAS 4) the other's sensations are more influenced by the self's movements (M1 → S2), and significantly so in the longer post-click periods [t(276.8) = 3.93, p < 0.001, t(263.6) = 3.26, p < 0.01, for 5 and 10 s with PAS 4, respectively). This is consistent with the idea that the self returns the feeling of passive touch to the other, which from the self's perspective involves a transition from passive to active touch, but this possibility is more typical for clear awareness. After less clear experiences (PAS 2 and 3) there tends to be a stronger influence from the other's movements to other's own sensations, a trend especially notable for the immediate post-click period (1 and 5 s) and for the least clear experience (PAS 2). This is consistent with the idea that the self disengages from its interaction with the other after making a click, thereby leaving the other alone to generate their own sensations, but this decoupling is more typical for when the other's presence was not experienced sufficiently clearly.

Both of these situations can be confirmed in **Figure 3**, where the player who first disengages after the clicks (Pa) was also the one who gave a PAS score of only 2, while the other player who apparently would have continued interacting gave a PAS score of 4. These differences in the self's awareness-dependent post-click behavior, namely the transition from passive to active touch compared with relative disengagement, deserve attention in future studies.

# DISCUSSION

We confirmed Froese and Di Paolo's (2011) hypothesis that real-time co-regulation of interaction, as measured by TT and movement synchrony, is higher when there is clearer awareness of the other's presence as measured by PAS ratings. However, although movement synchrony was significant, the relationship between PAS ratings and the movement synchrony was not particularly conspicuous, which suggests that other factors are in play as well. We therefore looked at the role of timescales. We found that the duration of an interaction makes a difference for how it is experienced, because the time lags involved in the most pronounced periods of synchrony were different among different awareness ratings.

We then looked at how participants influenced each other's sensations via their movements. The results of the TE analysis are consistent with the importance of short timescales. Passive stimulation first becomes notable around 5 s before a click, but only becomes significant in the second before the click. To be fair, this relative reduction in self-generated stimulation may be partially due to the fact that participants often paused their movements in order click. However, even so this pausing does not explain a corresponding increase in influence from the other's actions to one's sensations (for instance, the other player could pause as well or disengage altogether). More importantly, it does not explain why the extent to which this passively received stimulation exceeded self-generated stimulation was positively correlated with the reported clarity of the other's presence. Also, in the original perceptual crossing experiment by Auvray et al. (2009) they reported that the most common cue preceding clicks was "changes in the stimulation without moving" and this is consistent with our result.

We can better make sense of this positive correlation by considering that if one feels a sensation while moving, it is not clear whether this sensation was caused by one's own or by the other's movement. However, if one feels a new sensation without moving, then one cannot have caused the sensation. Moreover, the structure of the other's autonomously generated movements should be more apparent. For example, if I feel a repeated, irregular and yet contingent, i.e., not random, series of sensations it is most likely generated by the other (Lenay et al., 2011). We therefore suggest that participants remained passive during their transition to social perception because this permitted the presence of the other to be most clearly experienced in terms of the other's autonomous movements being applied to one's self.

FIGURE 9 | Average local TE to the other's sensations before and after clicks. We analyzed periods of 10, 5, and 1 s before a click and 1, 5, and 10 s after a click. Yellow and green bars indicate how much the movements made by the self (M1), i.e., the player who made the click, and by the other player (M2) contributed to the other's tactile stimulation (S2), respectively. Error bars represent standard error; significance was calculated using the Welch's t-test. <sup>∗</sup>p < 0.05, ∗∗p < 0.01, ∗∗∗p < 0.001.

However, in contrast to our expectations, we did not find that a continuous feeling of passive touch is essential, at least not if we interpret continuity to imply periods of more than just a few seconds. The average lag time characteristic of the most pronounced periods of synchrony does not go beyond Varela's (1999) "1 s" timescale, according to which an integrated cognitive–perceptual act tends to have a maximum duration of 2–3 s. Similarly, passive touch was only found to be statistically significant within one second before a click. Nevertheless, Miyahara's (2015) speculation that an essential difference between the intersubjective phenomenon of passive touch and other passive tactile encounters resides in the temporal dimension may still be correct. Specifically, he claims: "To experience the other body as really involving the other agent, we must experience it as an actual or potential constituent of passive experience in an ongoing, continuous manner" (p. 31).

And indeed the final moment of clear passive touch was embedded within a longer coordination, in which either player could always potentially be the receiver of passive touch and which may already have included several actual instances of passive touch in the form of TT. Presumably this preceding period of co-regulation ensures the success of interactively coordinated symmetry breaking into the complementary roles required for passive touch. Agentbased models confirm that TT and movement synchrony can be analyzed as spontaneous cooperative and co-creative processes, and they indicate that changes in the extent of mutual predictability might play a role in the coordinated timing of role switching (Ikegami and Iizuka, 2007). Future studies could try to analyze changes in relative predictability using human data.

One limitation of the current study is that it was only applied to adult interactions. Nevertheless, we suggest that our methods could be applied to sensorimotor data from developmental studies with the aim of analyzing preverbal infants' initial transitions to social awareness. The results are at least consistent with the theory that infants' first form of primitive social awareness is based on their being the object of other's attention. They also support the claim that passive touch continues to be operational in adulthood within a larger repertoire of social skills. For example, we cannot tickle ourselves but depend on others to tickle us, so there may still be something special about passive touch even in adulthood. Nevertheless, the relative importance of passive touch for adult social perception is debatable. Even if infants' social perception is primarily constituted in this relational manner in the tactile modality, as adults we can perceive others as subjects in their own right without being touched by them and without even being the object of their attention. How passive touch could develop into this more generalized capacity is an interesting open problem.

We speculate that once infants develop their visual awareness they tend to experience themselves as being the object of the other's attention in both the tactile and the visual modality, and that this repeated multimodal association enables them to perceive the other's presence even when being their object of attention in the visual modality alone. The same may apply to other modalities, such as audition. Presumably, the subsequent development of the capacity for joint attention, which makes infants aware that the other's attention can also be directed at other objects than the self eventually enables them to perceive the presence of others by only perceiving their engagement with the rest of the world in general. A recent extension of the perceptual crossing paradigm designed to investigate the interaction dynamics underlying mutual awareness of a shared object (Deschamps et al., 2016) could perhaps be adapted to evaluate these ideas in an experimental manner.

# CONCLUSION

Developmental and phenomenological approaches to embodied cognition have converged on the relational hypothesis that being the other's object of attention, as exemplified by the phenomenon of passive touch, is the most basic form of awareness of other minds. We evaluated this hypothesis quantitatively by reanalyzing the sensorimotor time series of a perceptual crossing experiment, in which pairs of players were tasked to cooperatively coordinate their movements so as to locate each other in a minimal virtual environment using haptic interfaces, and to identify and subjectively evaluate the moment they became aware of the other's presence.

We found that the transition to clear awareness of the other's presence coincides with pronounced moments of passive touch. To our knowledge this is the first time that the relational hypothesis has been supported with quantitative results. In addition, our time series analysis enabled us to extend this hypothesis into new directions. We proposed that elevated levels of sensorimotor coordination enhance one's awareness of being the other's object of attention. On our view, a salient moment of passive touch depends on spontaneous symmetry breaking of interpersonal movement coordination and therefore emerges out of a more extended co-regulation of behaviors. This makes reciprocity another essential factor in our account. We suggest that developmental and phenomenological versions of the relational hypothesis have underestimated the role of mutuality, as exemplified by the surprising finding that the clearest social experience tends to be associated with players switching roles in a transition from passive to active touch such that one individual's awareness of the other transforms into a dyadic awareness of each other.

# AUTHOR CONTRIBUTIONS

HK did the main analysis of the experiment using TE and CCs. TF helped to write the manuscript, especially the philosophical background of the paper. MO initiated and contributed to the analysis of the experiment and the discussion of the results. HI contributed to the analysis of the paper, and developed the TT measures. TI helped in writing the manuscript, especially the main findings of the paper, and contributed to the analysis.

# FUNDING

fpsyg-08-01778 October 11, 2017 Time: 18:24 # 13

TI and HI were supported by a KAKENHI (Scientific Research on Innovative Areas) project on "Cognitive Interaction Design (CID)" (Nos. 15H01612 and 15H01611, respectively). TF was supported by UNAM-DGAPA-PAPIIT project "Explorando los Alcances de la Auto-organización Social: Desde la Cultura hasta la Célula" (IA104717). This research was supported by Global Station for Big Data

#### REFERENCES


and Cybersecurity, a project of the Global Institution for Collaborative Research and Education at Hokkaido University.

# ACKNOWLEDGMENT

We would like to thank Kazuhiro Ueda for his valuable comments on our statistical analysis.



Varela, F. J. (1999). Present-time consciousness. J. Conscious. Stud. 6, 111–140.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Kojima, Froese, Oka, Iizuka and Ikegami. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Oscillatory Correlates of Visual Consciousness

Stefano Gallotto1,2, Alexander T. Sack1,2† , Teresa Schuhmann1,2† and Tom A. de Graaf1,2 \*

<sup>1</sup> Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, Netherlands, <sup>2</sup> Maastricht Brain Imaging Centre, Maastricht, Netherlands

Conscious experiences are linked to activity in our brain: the neural correlates of consciousness (NCC). Empirical research on these NCCs covers a wide range of brain activity signals, measures, and methodologies. In this paper, we focus on spontaneous brain oscillations; rhythmic fluctuations of neuronal (population) activity which can be characterized by a range of parameters, such as frequency, amplitude (power), and phase. We provide an overview of oscillatory measures that appear to correlate with conscious perception. We also discuss how increasingly sophisticated techniques allow us to study the causal role of oscillatory activity in conscious perception (i.e., 'entrainment'). This review of oscillatory correlates of consciousness suggests that, for example, activity in the alpha-band (7–13 Hz) may index, or even causally support, conscious perception. But such results also showcase an increasingly acknowledged difficulty in NCC research; the challenge of separating neural activity necessary for conscious experience to arise (prerequisites) from neural activity underlying the conscious experience itself (substrates) or its results (consequences).

#### Edited by:

Guido Hesselmann, Charité Universitätsmedizin Berlin, Germany

#### Reviewed by:

Laura Dugué, Université Paris Descartes, France Julian Keil, Charité Universitätsmedizin Berlin, Germany

\*Correspondence: Tom A. de Graaf

tom.degraaf@maastrichtuniversity.nl †These authors are the co-authors.

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 02 February 2017 Accepted: 23 June 2017 Published: 07 July 2017

#### Citation:

Gallotto S, Sack AT, Schuhmann T and de Graaf TA (2017) Oscillatory Correlates of Visual Consciousness. Front. Psychol. 8:1147. doi: 10.3389/fpsyg.2017.01147

Frontiers in Psychology | www.frontiersin.org July 2017 | Volume 8 | Article 1147 |

Keywords: consciousness, awareness, vision, correlates, oscillations

# INTRODUCTION

In the last few decades, progress in technology and signal analysis have resulted in new neuroimaging and electrophysiology techniques, greatly enhancing the range and resolution of brain research applications. As such, our understanding of the brain has proceeded at a staggering pace. Naturally, these techniques have been tried on the oldest problem of all: the nature of consciousness.

'Consciousness' can be defined in many ways (for our own taxonomy, see de Graaf et al., 2012; de Graaf and Sack, 2014). Generally, it is useful to separate minimally two concepts of consciousness. 'State consciousness' determines consciousness on a global level, for example distinguishing the extent of consciousness in coma, wakefulness, or anesthesia (e.g., Koch, 2004; Laureys and Tononi, 2010). 'Content consciousness' refers to moment-by-moment experiences of a conscious being, such as the experience of seeing blue, hearing a trumpet, or the famous 'what-it-is-like' to momentarily be a bat (Nagel, 1974). In this article, we focus on content consciousness, specifically in the visual modality.

The neural correlates of consciousness (NCCs) have been defined as the minimal set of neuronal mechanisms that are jointly sufficient for a conscious experience (Crick and Koch, 1990b).

To study NCCs, one generally tries to induce minimally two different conscious experiences using 'consciousness paradigms' [e.g., illusions, multistable and ON–OFF paradigms (Kim and Blake, 2005; de Graaf and Sack, 2014)], to then measure and compare brain activity in both

(with neuroimaging techniques). This basic approach has been referred to as 'contrastive analysis' (Baars, 1989; Aru et al., 2012). It can help reveal endogenous neural mechanisms underlying conscious perception, particularly if the physical stimuli remain identical in both conscious states. For example, when a low-intensity visual stimulus is repeatedly presented at perception threshold, the participant consciously perceives it on some but not on all trials. Thus, under identical stimulation conditions, this creates two types of trials: trials with conscious perception (ON) and trials without conscious perception (OFF) (de Graaf and Sack, 2014).

Different neuroimaging techniques can compare brain activity in both types of trials, such as functional magnetic resonance imaging (fMRI), magneto-/electroencephalography (M/EEG), electrocorticography (ECoG), or positron emission tomography (PET). Each has distinct advantages and applications, but here we focus on M/EEG, which can detect rhythmic fluctuations of brain activity, i.e., oscillations, with high temporal resolution. This is valuable as there is increasing evidence that oscillatory signatures may index conscious perception (e.g., Hanslmayr et al., 2007; Romei et al., 2008b; Busch et al., 2009; Lange et al., 2013). We here review such evidence, organized by frequency-band. For some of the oscillatory correlates of consciousness, recent studies investigated their causal contribution to conscious perception. By using brain stimulation techniques or rhythmic sensory stimulation, fascinating new 'entrainment' approaches allow the experimenter to control oscillatory activity to evaluate its causal role in conscious perception. From this overview, we address the question; what are the oscillatory correlates of consciousness?

In addressing this question, this review has three goals. Firstly, it is meant to be instructive. We provide a basic overview of oscillations and how to measure them, paradigms used to identify, isolate, and study consciousness, and results: oscillatory measures reported to correlate with (visual) consciousness using such approaches. Secondly, we draw attention to the recent applications of entrainment to study the causal role of these oscillatory measures. Thirdly, we use the reviewed findings to illuminate an old problem: how to determine the functional role of such mechanisms? We have previously discussed how NCC, of any type or form, can factually be three sorts of processes: neural prerequisites, neural substrates, and neural consequences of a conscious experience (de Graaf et al., 2012; de Graaf and Sack, 2014). Interpreting oscillatory correlates of consciousness in this framework may provide new insights, and should be kept in mind when designing and interpreting future studies.

# HOW TO STUDY OSCILLATORY CORRELATES OF CONSCIOUSNESS?

#### Consciousness Paradigms

Generally speaking, consciousness paradigms share the ability to induce at least two different conscious percepts of a physically identical stimulus (Logothetis, 1998; Blake and Logothetis, 2002). We previously grouped them into three different categories: illusions, multistable paradigms, and ON–OFF paradigms (de Graaf and Sack, 2014, 2015), illustrated in **Figure 1**.

Illusions are conscious percepts that are created endogenously, in absence of sensory information from the physical environment usually causing the conscious percept now observed (i.e., in other situations or in other observers). A famous example is the Kanizsa triangle (Kanizsa, 1976): one perceives triangle-contours, even though lines delineating the sides of the triangle – which usually cause the conscious triangle percept – are missing from the image. Illusions can be useful to study consciousness, since brain activity correlated to their perception reflects 'constructive' processes of conscious vision (Goebel et al., 1998). A different approach involves afterimages: a percept remains present in visual experience even though the stimulus that evoked it has been removed (Zaidi et al., 2012). Hallucinations, lastly, do not involve any input and might be classified as illusions as well. They are typically present in pathologies as schizophrenia, in which the patient can experience different percepts (e.g., auditory, olfactory) in the total absence of external stimulation. But in fact, many of us may perceive hallucinatory illusions if we are deprived of sensory inputs altogether (Vosburg et al., 1960).

There are other examples of illusions in the absence of sensory stimuli, from less controlled and more complex [e.g., phantom pain (Blakeslee and Ramachandran, 1998), or illusory percepts in a scotoma] to fully controlled (e.g., magnetic pulse-induced 'phosphenes'; illusory visual experiences induced without visual stimulation (de Graaf et al., 2014). Goebel et al. (1998) provided a compelling demonstration of how to use illusions to study conscious perception. They used static visual stimuli to induce illusory contours that appeared to move (the illusory motion quartet), and mapped correlating brain activity with fMRI. By separating the features of the visual inputs (static) from the features of the illusory percept (motion), the activity observed in the human motion areas could only be attributed to the endogenous construction of conscious motion perception.

Multistable paradigms notably include the well-researched paradigms of binocular rivalry (Fox, 1991; Blake, 2001), and ambiguous figures, such as the famous Necker cube or Rubin's vase/faces (see **Figure 1**). In the binocular rivalry paradigm, two different images are presented, each to one eye, at corresponding retinal locations. They need to be sufficiently different from each other, so that binocular fusion is impossible. As a result, the conscious percept of the observer keeps changing, even though stimulation never changes. Similarly, an observer will always experience only one conscious percept at a time when presented with a constant ambiguous figure, such as Rubin's vase/faces (**Figure 1B**), where the observer either experiences the vase, or the face, but never both simultaneously. In binocular rivalry, and here, comparing brain activity during both possible percepts can be very useful to find NCC's, because there is a change in consciousness unaccompanied by a change in external inputs. Any change in brain activity, occurring together with the change in consciousness, can be interpreted as underlying conscious processing of whichever percept is now reported. These correlates of conscious percepts are then not confounded by several unconscious perceptual processes that normally result from changes in inputs.

However, in those multistable paradigms, a condition is defined by the participant's report of their conscious percept. Participants mainly signal their experience by button presses. Practically, this creates problems in neuroimaging, since brain activity correlated to such percept switches (Lumer et al., 1998; Tong and Engel, 2001) is contaminated with task performance (Knapen et al., 2011). With M/EEG, the variability in response times creates additional difficulty (Strüber et al., 2000; Strüber and Herrmann, 2002), so it is promising that new and temporally accurate measures of percept switch timing are being explored (e.g., ocular reflexes; Frässle et al., 2014).

The high temporal resolution of M/EEG makes these techniques particularly well-suited for a third class of consciousness paradigms: ON–OFF paradigms. ON–OFF tasks have two conscious states: 'stimulus perceived' (ON) and 'stimulus not perceived' (OFF), i.e., conscious vision present vs. not present. The implementations of this basic principle come in many forms, such as visual masking (Breitmeyer and Ogmen, 2006) or transcranial magnetic stimulation (TMS) (Taylor et al., 2010). Generally, brain activity is simply contrasted between the ON and the OFF condition. Hemodynamic imaging allows us to study consciousness using 'weak ON–OFF tasks' in which small stimulus parameter adjustments cause stimuli to be always perceived or never perceived – enabling the implementation of experimental blocks of ON and OFF trials (e.g., word masking in Dehaene et al., 2001). But M/EEG can employ 'strong ON–OFF tasks' in which the exact same stimuli are used in all trials. In this case, brain activity highlighted by contrastive analysis is strictly related to endogenous processes differentiating stimulus perceived (ON) from not perceived (OFF) conditions, since the input does not change at all. With strong ON–OFF tasks we can therefore isolate and compare even more precisely the activity related to the two conditions. As per our earlier example; the simplest form of this is visual stimuli presented at perception threshold, causing detection (ON) on half of all trials, and failure to detect (OFF) on the other half of trials.

Thus, illusions, multistable, and ON–OFF paradigms, are all suitable for brain imaging experiments employing contrastive analysis. Yet they also share a conceptual difficulty which should be noted. In the example of a stimulus detection task, ON trials can engage a neural mechanism 'N,' which is not activated in OFF trials. 'N' is therefore an empirical correlate of consciousness. But which level of processing is 'N' involved in? A conscious percept finally arises from a cascade of processing, much of which is unconscious and which can likely be segmented into many steps and stages depending on context and framework. Thus, the exact role of 'N' can usually not be determined from a single experiment. We return to this issue in Section "Functional Roles of Oscillatory NCCs".

#### Oscillations

To continue with the example of a detection task, once ON and OFF trials have been post hoc labeled based on participant responses, oscillatory activity can be contrasted between both conditions. Though, we will discuss primarily oscillatory activity as measured with non-invasive neuroimaging methods such as M/EEG, much will apply to oscillatory signals measured more invasively in humans (e.g., ECoG) or oscillatory signals from smaller populations (e.g., local field potentials). So what is 'oscillatory activity'? A single oscillating signal can be characterized by three parameters: frequency, amplitude, and phase.

The rhythmic fluctuations in M/EEG signals primarily reflect rhythmic synchronous firing of populations of pyramidal neurons [i.e., excitatory and inhibitory postsynaptic action potentials (Creutzfeldt et al., 1966; Proudfoot et al., 2014)]. The strength of the signal, which translates to the amplitude (directly related to 'power') of an oscillation, depends on the

absolute number of firing pyramidal cells, how often they fire, and to what extent they fire synchronously. This synchronization is mainly guided by interneurons which, discharging together, generate perisomatic inhibitory postsynaptic potentials (Bartos et al., 2007). The rhythmic nature of individual neuronal firing bursts results in population-level activity that follows a sinusoidal pattern, with alternating high and low levels of activity. At any point in time, where (or rather when) the signal finds itself on this repeating sinusoidal activity cycle is defined as its phase. How often the activity cycle goes up and down in a certain unit of time (generally seconds) is defined as the signal's frequency, see **Figure 2** for a visualization.

When looking at an oscillating signal across repeated trials, one could analyze how similar the phase is across trials with reference to a particular time-locked event; called phase-locking (Tallon-Baudry et al., 1996). When looking at brain systems, one could also evaluate phase-locking between two nodes of a brain network, evaluating how consistent the phase relationship is between two oscillating signals when a particular event occurs. Or, more generally, and independently of certain time-locking occurrences, how consistent the phase relationship is between two ongoing signals from two brain regions, in which case one is quantifying phase coherence (Srinivasan et al., 1999). Such measures likely reflect functional connectivity between regions, and while there is a whole range of more advanced analyses one might consider in such contexts, for instance to evaluate directed connectivity (which region drives activity in the other?), these are beyond the scope of this review (see Bastos and Schoffelen, 2015 for a recent review of advanced analyses).

Data obtained with M/EEG measurements reflect a combination of noise and signals, which can be analyzed in different ways. As shown in **Figure 3**, one can extract the contribution of oscillatory signals in different frequencies (**Figure 3B**) to the original (preprocessed) data (**Figure 3A**), or visualize how these contributions change over time (**Figure 3C**). Oscillatory brain activity itself can fluctuate over time, and different networks in the brain are characterized by different frequencies (Keitel and Gross, 2016). For instance, occipital and parietal brain areas are mostly characterized by alpha activity, and sensory areas by alpha as well as beta activities (Pfurtscheller et al., 1996; Hari and Salmelin, 1997; Hillebrand et al., 2012). Furthermore, it has been suggested that activity within a given brain network may reflect a unitary sampling rhythm that is different between distinct networks (Canolty and Knight, 2010). For example, while small local networks usually operate in higher frequencies, larger distributed networks may employ slower fluctuations (Draguhn and Buzsaki, 2004). In line with this idea, theta/alpha-band oscillations (4–13 Hz) have been related to long-range communication, but beta/gamma-band oscillations (20–100 Hz) to short-range signaling (von Stein et al., 2000).

Moreover, certain brain systems may inherently prefer different frequency-bands, referred to as their 'normal frequencies' (Niedermeyer, 1999). In fact, different brain systems show particularly strong responses in different frequency-bands, measured with EEG, in response to single magnetic pulses (TMS), with occipital cortex presenting a stronger response to alpha-band oscillations, parietal cortex to beta-band oscillations and frontal regions to fast beta and gamma oscillations (Rosanova et al., 2009). But it has also been suggested that brain networks might be flexible enough to employ different frequencies depending on sensory modality, task demands or parameters (VanRullen, 2016). In sum, the engagement of oscillatory mechanisms in distinct frequency bands across regions, tasks, and brain states, remains a topic of intense investigation.

This completes our introductions into consciousness paradigms, oscillation signals, and (analysis of) oscillatory brain mechanisms. In what follows, we review current evidence for

oscillatory mechanisms that correlate with conscious (visual) perception.

# OSCILLATORY CORRELATES OF CONSCIOUSNESS

We have seen that there is a range of paradigms to study oscillatory NCCs, and a range of oscillatory parameters to evaluate. In this section, we will review key findings on oscillatory correlates of consciousness, grouped by frequency-band. Lower frequencies (delta, theta) do not have a dedicated section because we are not aware of much evidence supporting their role in conscious vision per se.

# Gamma Frequency

Experimental evidence for a relationship between gamma-band (high-frequency: ∼30–100 Hz) activity and conscious perception was highly influential, helping to reinvigorate the scientific study of consciousness when Crick and Koch (1990a) summarized it in the '40 Hz hypothesis.' This hypothesis proposed that distributed neuronal activity is 'bound' through synchronization of oscillations, and that such synchronized activity specifically in the gamma-band is a neural correlate of conscious perception. Engel and Singer (2001) noted that binding by synchrony is implicated in several major processes related to conscious perception; arousal (Munk et al., 1996), segmentation (Engel et al., 1991), selection (Fries et al., 1997), and working memory (Tallon-Baudry et al., 1998). In cats, global features of visual stimuli (i.e., coherency of motion) produced gamma synchronization in the visual cortex (40–60 Hz) (Gray et al., 1989). Moreover, gamma-band synchrony directly indexed which of two incompatible images was perceived by a cat in a binocular rivalry implementation (Fries et al., 1997). In macaques, local field potential (LFP) fluctuations in the gamma range were recently correlated to phenomenal perception, higher up in the visual hierarchy in lateral prefrontal cortex (Panagiotaropoulos et al., 2012).

In humans, M/EEG studies showed that synchronization between large populations of neurons in anterior and posterior brain areas correlates to conscious vision (Srinivasan et al., 1999), occurring at global rather than local level as, for instance, it happens during the encoding of an external stimulus from a sensory area (Ward, 2003). Words that are consciously perceived, as compared to words not perceived, lead to a transient distributed gamma synchronization response, phaselocked both across and within hemispheres (Melloni et al., 2007). Furthermore, long-distance gamma synchronization appears only when perceptual objects are perceived as coherent conscious percepts (i.e., faces) as opposed to meaningless shapes (Rodriguez and George, 1999; Doesburg et al., 2005).

Gamma-band activity and consciousness have been investigated extensively, but not exclusively, in the visual domain. Gamma synchronization also correlates with conscious perception of non-visual stimuli. For example in the auditory system oscillatory activity near 40 Hz is not only related to the sensory but also to the cognitive processing of auditory clicks stimuli (Joliot et al., 1994). Furthermore, it has been related also to multimodal perception (Senkowski et al., 2008). For example, a recent study used a paradigm consisting of visual and auditory stimuli and showed that gamma power correlates with audiovisual perception (Balz et al., 2016). It can also have a functional role in the binding of distributed neural activities in olfactory consciousness (Mori et al., 2013). Lastly, tactile stimulation of one hand increases gamma-band coherence in the contralateral primary somatosensory cortex only when the stimulus is consciously perceived (Meador et al., 2002) as well as when tactile stimuli are associated to visual stimuli, showing

contralateral enhancement of gamma-band activity in occipital cortex (Lange et al., 2011).

In spite of these examples, gamma oscillatory activity as a 'signature' of consciousness continues to be debated. Gamma synchronization may also be induced by processes such as attention (Fries et al., 1997), which should be separated from conscious perception whenever possible (Koch and Tsuchiya, 2007). In this context, Wyart and Tallon-Baudry (2008) have suggested that attention is more associated with high gamma frequency, whereas conscious perception is more associated with mid-range gamma synchronization. Yet, there is also evidence that gamma-band activity does not solely appear when consciousness is present but can also persist or even increase during anesthesia (Vanderwolf, 2000; Imas et al., 2005; Murphy et al., 2011) or seizures (Pockett and Holmes, 2009), brain states that are clearly not characterized by consciousness. It may therefore be that although gamma-band activity is present in many different conscious states, it is not exclusive to them (Hermes et al., 2015) and not sufficient to allow consciousness (Luo et al., 2009). Of course different measures of gamma-band activity have been considered in the past, from local gamma power to distributed gamma coherence, and moreover in and across different brain systems, so the picture remains incomplete.

# Beta Frequency

One example of an ON–OFF paradigm is the 'attentional blink' paradigm, in which a rapid stream of visual stimuli is presented at fixation (rapid serial visual presentation, RSVP, task). Participants are given two targets (i.e., specific letters) to watch out for, and press a button whenever they see either one of them. The attentional blink phenomenon is the observation that participants are more likely to miss a target, if it follows a preceding target in a particular temporal window [target 1 to target 2 onset asynchrony (stimulus onset asynchrony, SOA) of around 200–500 ms; Shapiro et al., 1997]. The 'weak' version of this paradigm uses two different SOAs, one leads to stimulus perception (ON) and one does not (OFF). When only one SOA is used, for which target 2 is sometimes detected and other times not, this paradigm becomes a 'strong' ON–OFF paradigm. Gross et al. (2004) measured MEG during such an implementation. They found increased power in the low beta-band during the entire stream of stimuli when targets were detected (ON) compared to when they were not (OFF). Furthermore, they found stronger beta synchronization in a network dominated by right inferior parietal and left prefrontal regions, in ON trials.

The enhancement of beta synchronization might reflect a general state of increased sensitivity to behaviorally relevant stimuli, which could explain better target detection performance. Gaillard et al. (2009) presented masked words at threshold contrast, in an intracranial EEG study. In detected versus non-detected trials, there was stronger beta synchronization between long-distance regions, especially during the late phase of the conscious access, whereas this coherence was suppressed when the same stimulus does not become conscious. Interestingly, in both studies the synchronized activity appears not only in posterior regions, but spreads in a broader network that involves also frontal areas.

The relationship between beta oscillations and visual consciousness is not yet fully clear. For instance, one recent study with invasive recordings in the macaque, showed that the power of beta oscillations in lateral prefrontal cortex is not modulated by conscious versus unconscious stimulus processing (Panagiotaropoulos et al., 2013). But here again, we should keep in mind that local oscillatory synchronization, i.e., local oscillatory power, may reflect at least partially non-overlapping brain processes as compared to measures of phase coherence. Synchronization across brain regions is not the same as synchronization within brain regions.

# ALPHA-BAND ACTIVITY

Alpha oscillations have been extensively researched in relation to conscious (visual) perception. The alpha rhythm (7–13 Hz) is strongly linked to posterior areas of the brain, and has been associated to input regulation (Lorincz et al., 2009) as well as attention (Worden et al., 2000; Sauseng et al., 2005; Kelly et al., 2006; Thut et al., 2006; Marshall et al., 2015). When our brain is not engaged in a particular task, oscillations with alpha rhythm are more prominent and easy to detect, leading to the notion that alpha is an 'idling' rhythm, the activity of the brain at rest (Pfurtscheller et al., 1996). For instance, simply closing the eyes strongly enhances alpha power (Berger, 1929). At the same time, a large body of research has led to several sophisticated theories on exactly which role alpha activity plays in attention, perception, and awareness. Below, we discuss in turn several parameters of alpha activity and how they have been studied using versions of consciousness paradigms.

# Alpha Power

Since ongoing alpha power does not stay at a constant level but fluctuates over time (Lopes da Silva, 1991), alpha power fluctuations have been studied in relation to fluctuations in visual target detection. For instance, across participants, Hanslmayr et al. (2005) showed that lower performance in a visual perception task, in which participants discriminated different letters, correlated to higher parieto-occipital alpha amplitudes. Also within participants, the higher pre-stimulus alpha power activity, the less likely it is that a stimulus is detected. This probability of detection can be predicted by the amount of pre-stimulus alpha power trial-by-trial (Ergenoglu et al., 2004), particularly from alpha signals originating in the parieto-occipital sulcus (van Dijk et al., 2008). One interpretation of these results suggests that alpha power indexes a state of excitability (Klimesch et al., 2007). Indeed, Lange et al. (2013) cleverly used so-called 'double flash illusion' and 'fusion effect' paradigms to distinguish whether reduced alpha power increases the accuracy of visual processing (correctly reporting the occurrence of either one or two stimuli) or rather increases visual excitability (reporting two stimuli irrespectively of the correct answer). Their findings supported the latter hypothesis.

It has been suggested that a more direct measure of visual cortex excitability can be derived from phosphene perception. Phosphenes are fleeting conscious visual experiences, elicited experimentally through direct stimulation of visual cortex (Marg and Rudiak, 1994). For instance, TMS can be used to non-invasively excite neurons in occipital cortex, which in many participants results in phosphene perception if the stimulation intensity is sufficient. Different levels of excitability can be assessed directly by evaluating the stimulation intensity required to elicit phosphenes (phosphene threshold), or the proportion of trials that result in phosphene perception at some fixed level of stimulation intensity. A lower phosphene threshold, or higher proportion of phosphene perception at fixed TMS intensity, indicates higher visual excitability. Measuring alpha power with EEG, and visual excitability with TMS, alpha power has been related to excitability (with higher alpha power indicating lower excitability) across (Romei et al., 2008b) and within (at trial-by-trial level) participants (Romei et al., 2008a).

In sum, converging evidence suggests that the power of alpha oscillations around stimulus (or TMS pulse) onset co-determines whether that stimulus reaches conscious perception.

# Alpha Phase

Inherently, oscillatory phase fluctuates more quickly than power. In the case of alpha-band oscillations, several studies have correlated visual detection performance to the phase of naturally occurring alpha oscillations at the moment of target presentation. Busch et al. (2009) showed that the threshold to detect light flashes covaries over time with alpha phase, suggesting that alpha phase might shape our perception by determining whether or not a visual stimulus is selected for awareness. Similarly, Mathewson et al. (2009) revealed that metacontrast-masked visual targets are more likely to be detected if targets are presented at the peak, as opposed to the trough, of ongoing alpha oscillations measured with EEG. Interestingly, they found that alpha phase predicted detection performance only when alpha amplitude was high. Thus, oscillatory phase and amplitude, though different measures, may be challenging to evaluate separately.

It is possible that, as we saw above for alpha power, also alpha phase directly reflects visual excitability. Once again, TMS-elicited phosphene perception has been used as a probe for occipital excitability. And indeed, phosphene perception, and thus visual excitability, depends on the phase of ongoing alpha oscillations (Dugué et al., 2011). At the same time, it has been suggested that alpha oscillations represent the time frames of perception (VanRullen, 2016): short visual 'snapshots' of the world are represented by single cycles of the alpha oscillation. This hypothesis has been supported by studies showing that two visual stimuli presented in a short period can be detected as one, or two, depending on the precise frequency of alpha oscillations in individual observers. The shorter the cycle is [higher individual alpha frequency (IAF)], the higher the temporal resolution of perception will be, and thus the more likely it will be that an observer can correctly detect the presentation of two separate stimuli over time (Samaha and Postle, 2015), independently of the amplitude of alpha-band activity (Milton and Pleydell-Pearce, 2016).

There is further evidence for a functional role of alpha phase in the context of conscious vision, stemming from a different category of studies to which we now turn.

# IS OSCILLATORY ACTIVITY CAUSALLY INVOLVED IN VISUAL CONSCIOUSNESS? ENTRAINMENT APPROACHES

The studies discussed so far have utilized a correlational approach, generally contrasting passively measured brain activity in trials in which a stimulus was perceived with trials in which a stimulus was not perceived. Such studies have clearly shown that oscillatory activity, namely power, phase, and coherence in distinct frequency bands, can be related to conscious vision. They do not clarify, however, whether such electrophysiological processes play a causal role in perception and awareness, or are epiphenomenal consequences of other brain mechanisms that underlie conscious perception. To evaluate the causal role of oscillations, one should find a way to manipulate oscillatory parameters externally, bringing neuronal oscillations under experimental control. This general approach is called 'entrainment' (Thut et al., 2011a; Herrmann et al., 2016), and can be achieved in different ways. These include rhythmic sensory stimulation and brain stimulation. Here, we briefly review evidence that these approaches can indeed affect behavioral performance and neuronal oscillations, followed by an overview of which oscillations appear to be causally relevant for conscious vision.

# Entraining Behavior

A participant presented with a stream of auditory stimuli, in a constant rhythm, can predict when an upcoming stimulus will appear. This phenomenon may be related to 'sensory entrainment'; the alignment of a sensory system to the rhythm of sensory stimulation (Sameiro-Barbosa and Geiser, 2016). To test whether the sensory system of the participant is aligned with an external stimulation, researcher measure task performances as, for example, reaction time or detection accuracy. When the synchronization to the rhythm of presentation occurs, the response to an upcoming external stimulus is typically faster (i.e., lower reaction time), compared to when the entrainment is not present.

Rimmele et al. (2011) used auditory stimuli in order to test whether spatial and temporal expectations may change task performance. They used four different conditions (temporal expectation, spatial expectation, temporal and spatial expectation, and no expectation) and showed enhanced target detection and faster reaction time only in the condition of stimuli presented with temporal regularity. Furthermore, entrainment may lead to a more accurate performance. Facilitated performance has been shown in discriminating the intensity of a tone (Jones et al., 2002), as well as its duration (McAuley and Jones, 2003). In the visual domain, when gabor patches were presented within a stream of stimuli with fixed

SOA, they were discriminated better compared to when SOAs in the stream were jittered (Rohenkohl et al., 2012). The same results have been shown by Marchant and Driver (2013) who, using auditory (tones) and visual (red annuli) stimuli, showed faster reaction times and improved visual sensitivity when they were presented in a isochronous (with temporal regularity) condition compared to when they were presented randomly.

# Entraining Oscillations

An important question is whether this temporal alignment (i.e., synchronization) occurs not only at behavioral level, but also between intrinsic neural activity and the rhythm of the external stimulation. Oscillatory brain activity can be entrained by stimuli of different nature (e.g., visual, auditory, tactile) which may lead to synchronization of neural activity in visual (Mathewson et al., 2012; de Graaf et al., 2013), auditory (Luo and Poeppel, 2007; Besle et al., 2011; Nozaradan et al., 2016), or somatosensory (Langdon et al., 2011; Ross et al., 2013; Ruzzoli and Soto-Faraco, 2014) brain areas. Oscillations in different frequency bands can be synchronized to external stimuli depending on the rhythm of stimulation (Lakatos et al., 2008, 2013). Rhythmic auditory stimulation, for example, can modulate neural activity in high frequency bands as beta and gamma (Snyder and Large, 2005; Fujioka et al., 2012) and even more robustly in low frequencies as delta and theta (Kayser et al., 2009; Howard and Poeppel, 2012; Ding et al., 2014).

One shortcoming of sensory entrainment is that it can be difficult to localize the brain mechanisms underlying its effects. After all, the rhythmic sensory stimuli are processed throughout a sensory system, making it difficult to evaluate the causal role of oscillations in a specific brain area of interest. Fortunately, it is also possible to entrain neuronal oscillations locally, by directly stimulating a brain region with a particular frequency. Non-invasive brain stimulation (NIBS) has been applied to study the causal contribution of brain areas to a wide variety of processes (including conscious vision, see for review: de Graaf and Sack, 2014), and recently also brain oscillations.

Transcranial magnetic stimulation and transcranial alternating current stimulation (tACS) are NIBS techniques used to entrain neuronal oscillations (Antal et al., 2008; Thut et al., 2011b). Single TMS pulses have been shown to affect oscillatory mechanisms in distinct frequency bands depending on the site of stimulation (Rosanova et al., 2009). When multiple TMS pulses are applied in a certain frequency (e.g., 10 Hz), this likely causes a resetting of the phase of oscillatory neural activity followed by amplification of local oscillatory power in that same frequency range (Thut et al., 2011b).

Transcranial alternating current stimulation uses a low-intensity alternating current (i.e., it changes direction periodically) which can affect the membrane potential. Thereby it can interact with cortical excitability, allowing the modulation of spontaneous brain activity in specific frequencies (Antal et al., 2008; Chaieb et al., 2011; Wach et al., 2013). Zaehle et al. (2010) showed that when tACS is applied at IAF, its effects last beyond the stimulation, resulting in enhanced alpha power as measured by EEG after versus before tACS. Neuling et al. (2013) suggest that the after-effect can last up to 30 min, but emerges only when tACS amplitude is greater than the endogenous IAF power. Also using an online paradigm (i.e., the stimulation is applied while EEG records neural activity), Helfrich et al. (2014) could show that oscillatory entrainment at 10 Hz in parieto-occipital areas increases alpha power. However, it appears relevant that tACS is continuous. Strüber et al. (2015) used a short intermittent protocol composed of 1.5 s of resting EEG and 1 s of tACS stimulation, showing that such short stimulation bursts did not cause entrainment.

Despite the substantial number of studies reporting entrainment, the mechanisms underlying the effect of tACS is still not completely clear. For example, the effects may depend on brain state during stimulation, such as having eyes open or closed (Ruhnau et al., 2016). Furthermore, Vossen et al. (2015) replicated with EEG that alpha frequency tACS increased power in the alpha band (for repeated 8 s but not 3 s bursts of tACS). However, these after-effects of tACS were observed independently of whether sequential bursts of tACS were in phase or not. Also, EEG alpha oscillations immediately following tACS bursts did not phase-align with the preceding tACS burst. Lastly, the peak frequency in the alpha band after tACS did not correspond well with the exact tACS frequency, rather reflecting IAF. These results led the authors to propose a different hypothesis regarding the after-effects of tACS stimulation; reflecting synaptic plasticity rather than entrainment.

# Causal Role of Oscillations for Conscious Vision

In sum, both behavior (i.e., task performance) and neuronal oscillations can be affected by rhythmic sensory stimulation or rhythmic brain stimulation. Have these techniques been applied to oscillatory correlates of consciousness? If human brain oscillations can be controlled through entrainment approaches, oscillatory power and phase in specific frequencies become independent variables, allowing us to probe their causal role in conscious vision.

Using visual stimuli, Mathewson et al. (2010) found that detection performance depends on the latency of target presentation relative to a preceding rhythmic visual cue train. In fact, visual perception performance can oscillate across multiple alpha cycles following an alpha cue train (Mathewson et al., 2012; de Graaf et al., 2013). It seems likely that phase-reset/-locked neuronal alpha oscillations underlie such patterns of visual performance, as even a single sound can induce visual excitability fluctuations with alpha frequency (Romei et al., 2012).

In a pioneering study, Romei et al. (2010) showed that a burst of TMS pulses applied at 10 Hz directly affected whether or not a subsequent visual target was perceived. TMS pulses applied at different frequencies (5 or 20 Hz) had no such effect. Presenting visual targets at different latencies from a rhythmic alpha TMS burst also modulated target perception, suggesting that not only alpha power, but also alpha phase is causally relevant (Jaegle and Ro, 2014). Chanes et al. (2013) used TMS to entrain high-beta (30 Hz) or gamma (50 Hz) frequencies. They showed that neural activity was entrained only when these two specific

frequencies were used, but not when the stimulation did not have a specific rhythm (used as control conditions). Depending on the frequency of stimulation, specific behavioral aspects of task performance were altered, such as perceptual sensitivity and response criterion.

The causal role of oscillatory activity in conscious vision has also been studied with tACS. Helfrich et al. (2014) suggest that tACS-entrained alpha phase is relevant for visual perception. Kanai et al. (2008) reported that with ambient light, it is possible to induce phosphenes with occipital tACS at beta frequency. In contrast, in darkness, phosphenes were more likely perceived with tACS at alpha frequency. In a recent study, a 'square' of two sets of diagonal light stimuli were presented in alternation (a 'motion quartet'). In this bistable apparent motion stimulus, two lights could be perceived as moving back and forth horizontally, or vertically. TACS was applied at 40 Hz over both occipital cortices. The stimulation led to a relative decrease in horizontal motion perception, but only if the two hemispheres were stimulated with a 180 degrees phase difference (i.e., antiphase) and not with 0 degrees phase difference (in-phase) (Strüber et al., 2014).

In sum, entrainment approaches allowing researchers to control the power or phase of oscillations at a particular frequency have indeed been applied to conscious vision paradigms. But at the same time, comparing these studies with the overview of oscillatory correlates makes clear that (1) many oscillatory correlates of consciousness remain to be tested causally using entrainment techniques, and (2) the reviewed entrainment studies have focused predominantly on local power and phase, while conscious perception might depend (also) on more complicated oscillatory mechanisms, such as widespread coherence. Therefore, it seems useful to quickly review some of the exciting recent developments in entrainment methodology, which may open up causal studies of oscillatory mechanisms of consciousness even further.

### Advanced Entrainment Approaches

Polanía et al. (2012) successfully manipulated oscillatory coherence between frontal and parietal cortex in a memory task, using tACS. Experimentally synchronizing oscillations in the theta (6 Hz) band (applying tACS over both regions with 0 degree phase difference) improved working memory performance, while experimentally desynchronizing oscillations (tACS over both regions with 180 degree phase difference) impaired performance. In another application, Alekseichuk et al. (2016) recently modulated cross-frequency coupling, showing that gamma bursts coinciding with theta-peaks improved working memory performance, while this effect was absent if gamma bursts coincided with theta-troughs. This experimental manipulation was achieved with tACS stimulation, with short bursts of gamma-signals superimposed on an ongoing thetasignal.

In principle, such sophisticated tACS entrainment approaches require only an appropriate electrode montage, and equipment that allows external control of electrical stimulators. A complex electrical waveform such as required for cross-frequency coupling modulation can 'simply' be programmed and fed into the stimulation devices. Also the presentation of stimuli in single or multiple modalities can be time-locked to one or multiple tACS waveforms, to for example consistently present certain inputs at certain phases. We recently discussed hardware and freely available software solutions to enable such experiments (ten Oever et al., 2016). While the examples discussed directly above did not relate to conscious vision, most of the oscillatory correlates of consciousness reviewed here could be causally studied with entrainment, using such available tools.

# FUNCTIONAL ROLES OF OSCILLATORY NCCs

Oscillatory mechanisms that covary with conscious experience are, by definition, NCC. Such empirical findings can be called 'empirical NCCs' (de Graaf and Sack, 2015). But it has been repeatedly noted that finding empirical NCCs is not the end goal. An empirical NCC can still fulfill different functional roles, which should be understood in order to move forward to understanding how the brain actually establishes conscious experiences (Miller, 2001, 2007; Noë and Thompson, 2004; Bachmann, 2009; Hohwy, 2009; Melloni and Singer, 2010; Aru et al., 2012; Kanai and Tsuchiya, 2012; Sergent and Naccache, 2012). In the context of oscillatory correlates of consciousness, this is exactly why entrainment approaches are so valuable; they allow us to go beyond correlation.

# Prerequisites, Substrates, Consequences

Several authors proposed different frameworks with possible roles that neural correlates, including oscillatory correlates, may play. What they appear to have in common, at least on a conceptual level, is that among the wealth of empirical neural correlates, only some reflect conscious experience itself.

Sergent and Naccache (2012) discuss the Global Workspace model (see also Baars, 1989; Dehaene and Naccache, 2001; Dehaene et al., 2006), postulating that many brain networks are continuously active, processing incoming information unconsciously. When top-down attention comes into play and leads to non-transient coherent activity throughout the brain, information can become conscious. In a first step, at low-level areas in the visual hierarchy (e.g., primary visual cortex) for about 200 ms after presentation of the stimulus, visual information is not yet conscious ('upstream processing'). As the information spreads to higher-order areas (i.e., frontal lobes), in a second step we can reach 'ignition' of the global workspace. Ignition means that we will have conscious experience on the one hand, and several 'downstream' processes that result from conscious experience and its underlying neural signature on the other hand. These can be hard to distinguish.

In another framework, Ruhnau et al. (2014) suggest that the parameters of power and phase are useful to describe local excitability and consequent stimulus detection, but not sufficient to thoroughly explain conscious experience. In fact, they propose that other networks in the brain (connected to high-order areas, i.e., parietal and prefrontal) need to be pre-activated to open

a so-called "window to consciousness" (Win2Con) and allow conscious perception. Local cortical excitability seems to be a "prerequisite" for conscious perception but does not reflect its neural process. General brain connectivity (from local to global level) seems to be required for visual consciousness, leading to conscious experience only when integration of relevant areas is achieved.

We and others (Aru et al., 2012; de Graaf et al., 2012; de Graaf and Sack, 2014) suggest that to define (and refine) correlates of consciousness it is useful to distinguish three core roles of an empirical NCC: neural substrates, neural prerequisites, and neuralconsequences of a conscious experience. 'Substrates' are the 'actual' NCC of interest, in the sense that the neural substrates of experience are directly causing, or are identical with, the phenomenal conscious experience. 'Prerequisites' are the neural events and mechanisms that are needed for neural substrates (and thus for a conscious experience) to arise. Consequences are in a sense less interesting, because they merely occur as a side-product of the neural prerequisites/substrates, however, meaningful in a cognitive/behavioral sense they may be. All the same, only a correct understanding, or even allocation, of empirical NCCs in light of these three different 'roles' can lead to a complete model of brain-experience relationships. Looking at the other examples of theoretical frameworks, it is easy to draw parallels. So we will continue to use our own terminology to refer to, for instance, 'prerequisites' rather than 'upstream processes,' even if similar conclusions could arise.

# Oscillatory Prerequisites, Substrates, and Consequences?

It might be useful to evaluate how this taxonomy maps onto oscillatory NCCs of conscious experience reviewed so far. This will not be exhaustive, to avoid repetition, but rather an exercise and illustration of the core concepts. For instance, it immediately becomes clear that many of the previously discussed empirical findings may fall in the 'prerequisites' category (Ergenoglu et al., 2004; Hanslmayr et al., 2007; Romei et al., 2008a; van Dijk et al., 2008; Busch et al., 2009; Lange et al., 2014). After all, any neuronal mechanism that occurs prior to a conscious experience can by definition not be a neural substrate or neural consequence of a conscious experience (de Graaf et al., 2012). In other words, beta, gamma, but most notably alpha power, phase, and coherence that occur before or at the moment of stimulus presentation, are either not required for conscious experience, or are prerequisites for it. They are empirical neural correlates, they can cause a conscious experience (later), but they cannot underlie the conscious experience itself (i.e., they are not substrates). This is because when the stimulus appears on a computer screen, there is not immediately a conscious experience of that stimulus. The visual information still needs to affect the retina, undergo rudimentary processing along several subcortical stations, reach primary visual cortex to be processed further, and only from that point onward could one reasonably start to wonder whether neural processing is or is not a substrate of a conscious experience (e.g., Silvanto et al., 2008).

On the one hand, one might argue that oscillatory phase at stimulus onset is reflective of oscillatory phase in the near-future. If one speculates that relevant visual processing occurs in primary visual cortex around 100 ms after stimulus onset (e.g., de Graaf et al., 2014), then alpha oscillations should actually be at the same phase when the information reaches the cortex, as was measured at stimulus onset. Thus, indirectly, peri-stimulus oscillatory correlates might still provide clues on neural substrates of consciousness. On the other hand, it is unclear at the moment to what extent the presentation of the stimulus itself changes 'ongoing' oscillations, for instance causing an oscillatory phasereset. Such considerations make it all the more important that some studies try to bypass certain sensory processing stages, for instance by magnetically stimulating occipital cortex directly. It is thus non-trivial that similar alpha power/phase effects on conscious experience (phosphene perception) were found in these studies (Romei et al., 2008a; Dugué et al., 2011).

In **Figure 4** a tentative model provides a hypothetical example of how prerequisites and substrates of consciousness may be related to different oscillatory correlates. We explained that a stimulus presented near sensory threshold may cause conscious experience depending on the brain state at the moment of its appearance. It might be that when power and phase of oscillatory activity fall under favorable circumstances (e.g., local alpha power in sensory – visual – areas has a momentary state below a particular threshold), they constitute (some of the) prerequisites necessary for a stimulus to become conscious. At this stage conscious experience is not yet achieved. Only when other mechanisms are engaged (e.g., long-range beta or gamma synchronization between low and high-order areas) conscious experience arises. The big challenge is to determine which of these additional processes are substrates of conscious vision.

Other empirical results presented here could potentially be reconsidered similarly. Gamma power, for example, was a long-standing candidate NCC. But there are recent findings that suggest that gamma oscillatory power is not, in the end, absolutely and always necessary nor sufficient for conscious experience (Luo et al., 2009). Formally speaking, that would mean gamma oscillatory power is not a universal prerequisite. But it could also be that it is required for some conscious experiences, such as coherent percepts that require binding of different visual features, and as such a prerequisite for specific experience and what one might call a precursor of conscious perception. Future studies should illuminate this issue, also clearly separating oscillatory gamma power from gamma-band coherence across regions.

The same could be said for beta-band oscillations. We did not cover many studies focusing on beta, but beta-band coherence still seems to be a candidate NCC. Beta-band responses to conscious perception seem to occur on a temporal scale that is consistent with conscious visual experiences, thus deserving further study. At the same time, one of the two main studies discussed that related beta-band oscillations to conscious perception actually employed the attentional blink paradigm (Gross et al., 2004), which leads us to arguably the largest confounder in NCC research: attention.

# Attention, Consciousness, Oscillations: Blurred Lines

Consciousness rarely seems to occur without attention, leading many researchers to argue that attention and consciousness are inextricably connected, if not the same process (Posner, 1994; Chun and Wolfe, 2000; O'Regan and Noe, 2001). Yet, others have argued that they are distinct phenomena, with distinct functions and neuronal mechanisms (Iwasaki, 1993; Lamme, 2003; Dehaene et al., 2006; Koch and Tsuchiya, 2007). Some recent neurophysiological evidence showed that a dissociation is not completely established (Chica and Bartolomeo, 2012), yet there are empirical demonstrations that reveal separated or even opposite effects of attention manipulations versus stimulus visibility (i.e., conscious perception) manipulations (van Boxtel et al., 2010; Watanabe et al., 2011).

The finer points of this ongoing discussion are beyond the scope of this review, but it is important to realize that many empirical correlates reviewed here can, in fact, also be interpreted as correlates of attention, rather than consciousness. Or at least, attention as a confounder can only rarely be ruled out. Many ON–OFF paradigms involve visual detection tasks, which could be said to either capture attentional efficacy, or conscious access. We saw that alpha power predicts conscious visual experience, but alpha power also indexes performance on explicit attention tasks (Thut et al., 2006). One example paradigm that demonstrates the entanglement of attention, conscious experience, and their relation to oscillations, is the attentional blink paradigm.

As we saw above, alpha oscillations play a prominent (yet not fully clear) role in attention and consciousness. In most attentional blink studies, the presentation rate for targets and distracters is approximately 10 Hz. Recent work has addressed the idea that the rhythmic stream of inputs in the attentional blink paradigm actually entrains alpha oscillations (Moratti et al., 2007). This phase-locking appears to result in visual stimulus presentation coinciding with troughs of EEG-measured parieto-occipital alpha oscillations (Hanslmayr et al., 2011). Moreover, pre-stimulus alpha phase at the onset of T1 predicts whether or not T2 will be detected (Zauner et al., 2012). These findings are in line with another study which suggests that, under strict temporal constraints, the processing of the pre-target

distracter stream enhances phase locking of the alpha oscillation, which predicts lower T2 detection (Petro and Keil, 2015). Lastly, the notion that oscillatory entrainment is somehow involved in attentional blink suppression is supported by the fact that introducing temporal discontinuities in the RSVP stream around presentation of T1/T2 reduces the attentional blink effect (Martin et al., 2011).

# CONCLUSION

To chart the exact cascade of neurocognitive events leading from visual inputs to eventual button presses, with attention and a conscious experience somewhere along the way, is still an enormous challenge. It is clear that oscillatory mechanisms are part of this process, but even when focusing on the visual modality, there is no single oscillatory mechanism that emerges as the core candidate for conscious processing. The tools to tease apart the role of various reported oscillatory correlates of consciousness are still evolving, with sophisticated developments

#### REFERENCES


in tACS entrainment procedures as a recent methodological highlight. Different parameters of oscillatory activity, frequency, power, phase, and coherence, should be evaluated with these new tools, to eventually distinguish their roles and contributions.

# AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

### ACKNOWLEDGMENTS

SG and AS are supported by the Netherlands Organization for Scientific Research (VICI grant 453-15-008 to AS), TdG is supported by the Netherlands Organization for Scientific Research (VENI grant 451-13-024). Thanks to Sanne ten Oever and two reviewers for comments on earlier drafts of the manuscript.




visual stimulation. J. Cogn. Neurosci. 24, 2321–2333. doi: 10.1162/jocn\_a\_ 00288



associated with human EEG alpha activity. Eur. J. Neurosci. 22, 2917–2926. doi: 10.1111/j.1460-9568.2005.04482.x



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer JK and handling Editor declared their shared affiliation, and the handling Editor states that the process nevertheless met the standards of a fair and objective review.

Copyright © 2017 Gallotto, Sack, Schuhmann and de Graaf. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Impact of Response Stimulus Interval on Transfer of Non-local Dependent Rules in Implicit Learning: An ERP Investigation

Jianping Huang1,2† , Hui Dai<sup>3</sup>† , Jing Ye<sup>1</sup> , Chuanlin Zhu<sup>1</sup> , Yingli Li<sup>1</sup> \* and Dianzhi Liu<sup>1</sup> \*

<sup>1</sup> Department of Psychology, Soochow University, Suzhou, China, <sup>2</sup> Department of Psychology, Tsinghua University, Beijing, China, <sup>3</sup> Student Affairs Office, Nanjing Tech University, Nanjing, China

In the literature on implicit learning, controversy exists regarding whether the knowledge obtained from implicit sequence learning consists of context-bound superficial features or context-free structural rules. To explore the nature of implicit knowledge, event related potentials (ERP) recordings of participants' performances in a non-local dependent transfer task under two response-stimulus-interval (RSI) conditions (250 and 750 ms) were obtained. In the behavioral data, a transfer effect was found in the 750 ms RSI condition but not in the 250 ms RSI condition, suggesting that a long RSI is the basis for the occurrence of non-local dependent transfer, as which might have provided enough reaction time for participants to process and capture the implicit rule. Moreover, P300 amplitude was found to be sensitive to the impact of RSI on the training process (i.e., the longer RSI elicited higher P300 amplitudes), while variations in both N200 (i.e., a significant increase) and P300 amplitudes (i.e., a significant decrease) were found to be related to the presence of a transfer effect. Our results supported the claim that implicit learning can involve abstract rule knowledge acquisition under an appropriate RSI condition, and that amplitude variation in early ERP components (i.e., N200 and P300) can be useful indexes of non-local dependent learning and transfer effects.

Keywords: implicit sequence learning, transfer, RSI, ERP, non-local dependencies

# INTRODUCTION

Implicit learning plays an important role in various forms of human cognition, such as language acquisition (Leung and Williams, 2011), music practice (Rohrmeier and Rebuschat, 2012), and the formation of perceptual-motor skills (Fu et al., 2010). As such, implicit learning is a critical component of learning and human life. How does implicit learning occur? According to Cleeremans and Jiménez (2002), representation of implicit knowledge in human brains develops from weak to strong neural association, with its representation quality as well as consciousness improved concurrently. Only knowledge with high quality representation (i.e., in terms of its stability, strength and uniqueness) can be fully conscious and orally reported. Moreover, unconscious knowledge is usually bounded to the stimuli's perceptual features and thus hard to transfer, while conscious knowledge is often related to embedded rules and thus easy to transfer (Zhang and Liu, 2014).

#### Edited by:

Morten Overgaard, Aarhus University, Denmark

#### Reviewed by:

Elisabeth Norman, University of Bergen, Norway Claudia Lunghi, Università degli Studi di Pisa, Italy

#### \*Correspondence:

Dianzhi Liu dianzhiliu@foxmail.com Yingli Li liyingli163163@163.com †These authors are joint first authors.

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 23 December 2016 Accepted: 17 November 2017 Published: 06 December 2017

#### Citation:

Huang J, Dai H, Ye J, Zhu C, Li Y and Liu D (2017) Impact of Response Stimulus Interval on Transfer of Non-local Dependent Rules in Implicit Learning: An ERP Investigation. Front. Psychol. 8:2107. doi: 10.3389/fpsyg.2017.02107

Several factors may influence knowledge representation quality in implicit learning. Destrebecqz and Cleeremans (2001, 2003) have found that variation in RSI impacts on representation quality of knowledge from implicit learning, with weak representation (i.e., mostly supported by unconscious knowledge) associated with short RSI, and strong representation (i.e., mostly supported by conscious knowledge) associated with long RSI. Similar results have been reported in several other studies (Zhang et al., 2014, 2015).

Another potential factor influencing knowledge representation quality in implicit learning is training length. Despite that Tanaka and Watanabe (2015) found that training length (i.e., number of training trials) failed to predict the occurrence of implicit transfer, other researchers have demonstrated that elongated training length can improve the representation quality of implicit rules and facilitate transfer, with full consciousness (i.e., indexed by accurate oral report) being achieved in the end (Cleeremans and Jiménez, 2002; Destrebecqz and Cleeremans, 2003; Goldstone and Sakamoto, 2003). Notably, in Tanaka and Watanabe's (2015), study the settings of training length (i.e., 4, 12, 16, and 20 trials, respectively) were probably too short for representation quality to vary among these conditions.

The implicit learning literature suggests that participants are not only able to acquire the perceptual features of implicit rules (e.g., physical properties of stimulus), but also able to obtain the deeper structural knowledge of embedded rules. In the former case, participants acquire local dependencies (i.e., knowledge of perceptual features of stimulus). In the latter case, participants acquire non-local dependencies (i.e., knowledge of abstract rules not dependent on perceptual structures). Early studies on non-local dependent transfer were focused to artificial language learning (Onnis et al., 2005; Williams, 2009), and then expanded to other areas such as implicit music learning and implicit sequence learning. Studies on implicit music learning generally show that tune structures can be implicitly learned and transferred. For example, when studying music tunes, participants were able to detect the change of embedded implicit rules (Dienes and Longuet-Higgins, 2004; Kuhn and Dienes, 2005; Dienes et al., 2012). Li et al. (2013) reported in their study that participants were capable of unconsciously obtaining the structural knowledge of rotated as well as mirror-reversed tunes. Tanaka and Watanabe (2014a,b) demonstrated that participants were able to unconsciously obtain the structural knowledge of spatial and temporal relationship of tunes and transfer it to mirror-reversed or rotated versions. It should be mentioned that these studies used music tunes as the test materials, which are concrete in nature. Therefore, the transfer effect found in those studies may be mediated by concrete knowledge of tunes (e.g., chunks of musical tunes), rather than abstract knowledge of implicit rules. To explore non-local dependent transfer (i.e., transfer of knowledge of implicit rules rather than knowledge of perceptual structures), test materials must be designed to be non-concrete in nature.

The present study was designed to use the technique of ERP to explore the impact of RSI on non-local dependent transfer of implicit sequence learning under extended training (70 trials). Event-related-potentials (ERP) is high in temporal resolution (Abla et al., 2008). It is suitable for exploring brain activation related to tasks associated with quick reactivation, such as implicit sequence learning. Some ERP studies on implicit sequence learning have shown that amplitude variations of the ERP components of N200 and P300 are sensitive to transitions in implicit sequence learning (Miyawaki et al., 2005; Ferdinand et al., 2008; Jost et al., 2011; Fu et al., 2013). There is no previous study using both an implicit transfer paradigm and the technique of ERP to investigate implicit knowledge representation patterns under different RSI conditions. Rüsseler and Rösler (2000) and Rüsseler et al. (2003) have explored knowledge representation pattern differences between implicit and explicit sequence learning by using ERP, and found that implicit sequence learners obtained a response–response connection, while explicit sequence learners obtained one of stimuli–stimuli, stimuli–response, and response– stimuli connections. Moreover, explicit learners' knowledge was represented as both perceptual and motor patterns, while implicit learners' knowledge was only represented as motor patterns. These researchers only investigated perceptual and motor implicit rules, but not higher-level abstract implicit rules. Remillard (2008, 2010) explored non-local dependent rule acquisition in a perceptual-motor sequence task and found that participants were capable of acquiring the nonlocal dependent rule. However, the non-local dependent rule used in Remillard's study is based on transfer probability and thus may not be suitable for testing abstract rule acquisition in implicit learning. For exploring the possibility of abstract rule acquisition and transfer in implicit learning, a nonlocal dependent rule with varied first-order structures (i.e., stimulus presentation or perceptual patterns), and an invariant higher-order structure (i.e., an abstract rule) may be a better choice.

Based on the above-mentioned reasons, the present study was aimed to investigate the impact of two different RSIs (i.e., 250 and 750 ms) on non-local dependent transfer in implicit sequence learning by using ERP. The literature suggests that 250 ms RSI and 750 ms RSI are two time settings sensitive to awareness changes, with mostly unconscious knowledge triggered by 250 ms RSI, and a collaboration of conscious and unconscious knowledge triggered by 750 ms RSI (Destrebecqz and Cleeremans, 2001, 2003; Haider and Frensch, 2009; Franco and Destrebecqz, 2012). It was hypothesized in the present study that, only under the condition of 750 ms RSI, participants' consciousness would continually increase during the training course, and their learning of the implicit rule would gradually develop from perceptual learning to abstract rule learning, which would eventually prompt their behavioral performance from no transfer to transfer. Moreover, Amplitude variations of the two ERP components of N200 and P300 were hypothesized to be sensitive to changes in awareness levels (i.e., consciousness vs. lack of consciousness) and transfer effects.

# MATERIALS AND METHODS

fpsyg-08-02107 December 4, 2017 Time: 17:38 # 3

### Participants

Fifty-five college students (mean age = 21.4, SD = 1.43) were randomly selected from a university. All were righthanded and with normal vision. Participants received a small amount of cash as a token of appreciation after they finished the experiment. One participant was excluded from the data because of high error rate on the test (>10%). Another four participants were excluded from the data because of high rates of noise in the ERP data. The final sample consisted of fifty participants (23 males and 27 females). Thirteen participants were randomly assigned to the 250 ms RSI experimental condition, and twelve participants were randomly assigned to the 250 ms RSI control condition. Thirteen participants were randomly assigned to the 750 ms RSI experimental condition, and 12 participants were randomly assigned to the 750 ms RSI control condition (i.e., for the rational of using two control groups, see Materials). All participants voluntarily signed a consent form, which was approved by the Research Ethics Committee of Soochow University. Full ethical review and approval were required according to the national and institutional requirements.

# Materials

This study was conducted in a sound proof room. Participants sat in front of a 17<sup>00</sup> computer screen (with a refresh rate of 75 Hz) at a distance of 90 cm. A classic sequence reaction task (Cleeremans and Jiménez, 1998) was adapted for the purpose of the present study. During the task, participants were required to press a key corresponding to the spatial location of a dark dot (with a diameter of 1 cm) presented on a computer screen as quickly and accurately as possible.

The experiment included a training phase and a transfer phase. The two experimental groups were required to finish both the training and the transfer phases, while the two control groups only the transfer phase. The experimental arrangements for both

the experimental groups and the control groups were shown in **Figure 1**.

A classic SOC rule (Reed and Johnson, 1994) was used in the sequences, in which the location of a third stimulus is determined by the locations of previous two stimuli (e.g., 34→2, 42→3, 23→1. . .. . .).

During the training phase, the spatial location arrangement for the stimuli followed a SOC1 rule: 342312143241 (i.e., numbers represent the four quadrants of the computer screen). Sample presentation sequence and corresponding key-press were shown in **Figure 2**.

During the transfer phase, the spatial location arrangement for the stimuli followed a SOC2 rule: 341243142132 (i.e., numbers represent the four horizontal locations on the computer screen). Sample presentation sequence and corresponding key-press were shown in **Figure 3**.

Notably, the use of a SOC1 rule for the training phase and a SOC2 rule for the transfer phase are critical for nonlocal dependent transfer. SOC1 and SOC2 are different in firstorder structure (i.e., perceptual features), but share the same higher-order structure (i.e., the SOC rule that the location of a third stimulus is determined by the locations of previous two stimuli). If participants' knowledge is bounded to perceptual features of SOC1 sequences, transfer to SOC2 sequences is not possible. Successful transfer to SOC2 sequences depends on the acquisition of the higher-order structure of SOC1 sequences. This arrangement is advantageous in comparison to traditional designs, such as using a mirror- reversed rule or a new rule for the transfer phase, in which perceptual transfer and abstract rule transfer can hardly be separated.

Moreover, the setting of control groups in the present study is to control for the possibility that participants are able to acquire the SOC2 rule during the transfer phase, without the training on the SOC1 sequences. If a transfer effect was observed in the experimental groups, with no significant learning effect observed in the control groups (who were tested on the transfer phase only), this transfer effect could be logically inferred as a transfer of abstract knowledge of SOC1 sequences to SOC2 sequences.

All sequences were programmed via E-prime 2.0

#### Procedure

All participants received a practice section (24 random trials) before the formal test to be acquainted with the key-pressing.

The training phase consisted of ten blocks, with ninety-six trials in each block. All the blocks (except Block 8) in the training phase were regular blocks. Each regular block contained seven repeated SOC1 sequences and one stochastically inserted random sequence (for the rationale behind inserting a random sequence, see Norman et al., 2007). Block 8 was a random block containing eight random sequences. Participants' average reaction time in Block 8 was used as a baseline for the training phase. Participants were allowed to rest 15 s between every two blocks. Their accuracy and reaction time were recorded. Logically, a learning effect (i.e. knowledge acquisition by repeated exposure to SOC1 sequences) is indexed by participants' increasingly shorted reaction time during the training phase. Therefore, implicit learning magnitude was estimated by participants'

average reaction time difference between the random block (i.e., baseline) and its proximate blocks [i.e., RT8-(RT7+RT9)/2].

The transfer phase consisted of six blocks (block 11–block 16), with ninety-six trials in each block. All the blocks (except Block 14) were regular blocks. Each regular block contained seven repeated SOC2 sequences and one stochastically inserted random sequence. Block 14 was a random block containing eight random sequences. Participants' average reaction time

in Block 14 was used as a baseline for the transfer phase. Participants were allowed to rest 15 s between every two blocks. Their accuracy and reaction time were recorded. Logically, a transfer effect (i.e., application of SOC1 knowledge to SOC2 sequences) is indexed by a quick learning (i.e., faster than the control groups, who were not trained on SOC1 sequences) on SOC2 sequences. Therefore, implicit transfer magnitude was estimated by participants' average reaction time difference between the random block and its proximate blocks [i.e., RT14-(RT13+RT15)/2].

After the test, all participants were asked the following three questions to measure their awareness levels (i.e., consciousness vs. lack of consciousness): what determines the spatial locations of the dots? Can you describe the rule underlying the spatial locations of the dots? By what time during the test did you find out this rule?

# ERP Recording and Analysis

Participants' EEG data was concurrently recorded during the test by using a 32-channel cap (Brain Product, Munich, Germany). The electrodes on the cap were positioned according to the international 10–20 system. VEOG and HEOG were recorded. The EEG signals were filtered with a bandpass of 0.05–100 Hz and sampled with a rate of 500 Hz.

The original EEG data was processed with the standard procedures provided by the software of ANALYZER 2.0. EEG data corresponding to behavioral data containing more than 10% error responses (Weiermann et al., 2010) and extreme reaction time (i.e., shorter than 100 ms or longer than 1000 ms) was excluded. After eye blink correction, other artifacts (i.e., epochs with EEG power exceeding ± 100 microvolt) were removed from the EEG data. The artifact-free data was segmented into EEG epochs (i.e., 900 ms post-stimulus intervals), baseline corrected (200 ms pre-stimulus interval), and averaged. The ERP data was further divided into six parts corresponding to three learning stages [stage 1 (Block 1, 2, and 3), stage 2 (Block 4, 5, and 6), and stage 3 (Block 7, 9, and 10)] and three transfer stages [stage 4 (Block 11 and 12), stage 5 (Block 13 and 14), and stage 6 (Block 16)]. Following previous studies (Eimer et al., 1996; Schlaghecken et al., 2000), EPR data on the four central electrodes (i.e., Fz, FCz, Cz, Pz) was used in the analysis. The amplitudes of N200 (230–310 ms) and P300 (340–530 ms) were extracted separately for the 250 ms RSI condition and 750 ms RSI condition.

# RESULTS

#### Behavioral Data

The oral reports showed that none of the participants were able to accurately describe the implicit rules, indicating that the learning process was implicit for all participants. Behavioral data containing more than 10% error responses and extreme reaction time (i.e., shorter than 100 ms or longer than 1000 ms) was excluded.

#### Occurrence of Implicit Learning on SOC1 Sequences for the Two Experimental Groups

Participants' reaction time showed an increase during both the 250 and 750 ms RSI conditions (see **Figure 4**). Based on the commonly used index of implicit learning magnitude [average reaction time difference between the random block and its proximate blocks in the training stage, i.e., RT8-(RT7+RT9)/2 for the present study], average reaction time for Block 8 (RT8) was compared to average reaction time of its proximate blocks [(RT7+RT9)/2]. The occurrence of implicit learning would be indicated by a significantly longer average reaction time of Block 8 than that of its proximate blocks. Repeated ANOVAs with Block as the independent variable showed that the averaged reaction time of Block 8 (404.32 ms ± 54.28 ms) was significantly longer than the average reaction time of its proximate blocks (379.99 ms ± 60.58 ms) for the 250ms RSI experimental group [F(1,12) = 22.27, p < 0.001, η <sup>2</sup> = 0.65], and that the averaged reaction time of Block 8 (357.98 ms ± 23.56ms) was significantly longer than the average reaction time of its proximate blocks (345.94 ms ± 29.87 ms) for the 750 ms RSI experimental group [F(1,12) = 10.73, p < 0.01, η <sup>2</sup> = 0.47]. These results suggest that effective implicit learning occurred in both experimental groups.

#### Comparison of Implicit Learning Magnitudes between the Two Experimental Groups

To further explore the relative amount of implicit learning magnitudes between the two experimental groups, a one-way ANOVA with Group as the independent variable and learning magnitude as the dependent variable showed that the 250 ms RSI and 750ms RSI experimental groups (24.33 ms ± 18.59 ms vs. 12.04 ms ± 13.29 ms) did not differ significantly on learning magnitude [F(1,12) = 2.38, p > 0.05, η <sup>2</sup> = 0.17].

#### Comparison of Implicit Transfer Magnitudes between the Two Experimental Groups

To explore transfer effects in the two experimental groups, repeated ANOVAs with Block as the independent variable were conducted to compare the mean reaction time of Block 14 (RT14) to the mean reaction time of its proximate blocks [(RT<sup>13</sup> + RT15)/2]. These analyses yielded a significant main effect for the 750 ms RSI experimental group [F(1,12) = 9.09,

p < 0.05, η <sup>2</sup> = 0.43], showing longer reaction time in Block 14 (404.32 ms ± 70.99 ms) than that in its proximate blocks (376.17 ms ± 46.06 ms). No significant difference was found for the 250 ms RSI experimental group [F(1,12) = 0.01, p > 0.05, η <sup>2</sup> = 0.001]. These results suggest that effective transfer occurred in the 750 ms RSI experimental group, but not in the 250 ms RSI experimental group.

#### Implicit Learning Magnitudes in the Control Groups

In the above analysis, the observed significant reaction time changes in the 750 ms RSI experimental group during the transfer phase can be caused by both a transfer of SOC1 knowledge to SOC2 sequences (i.e., a transfer effect) and the learning process on SOC2 sequences (i.e., a learning effect). To examine whether it is a transfer effect or a learning effect, data of the control groups must be compared to that of the experimental groups. Notably, participants in the 250 and 750 ms RSI control groups were assigned to the transfer phase without any previous training. If the control groups fail to show a significant learning effect on SOC2 sequences, it can be inferred that the observed significant reaction time changes in the 750 ms RSI experimental group is a transfer effect rather than a learning effect.

Repeated ANOVAs for the control groups revealed no significant learning effects in both the 250ms RSI control group (493.81 ms ± 79.48 ms vs. 481.94 ms ± 76.02 ms) [F(1,11) = 2.54, p > 0.05, η <sup>2</sup> = 0.19] and the 750 ms RSI control group (425.44 ms ± 44.17 ms vs. 423.09 ms ± 47.42 ms) [F(1, 11) = 0.13, p > 0.05, η <sup>2</sup> = 0.01]. These results suggested that 4 blocks of SOC2 sequence training were not enough for the control groups to capture the structural rule, and confirmed that the reaction time decrease over the transfer phase in the 750 ms RSI experimental group was a transfer effect (i.e., the transfer of their previous SOC1 knowledge to SOC2 sequence learning), rather than a learning effect (i.e. structural rule acquisition by exposure to the 4 blocks of SOC2 sequences).

#### ERP Data

#### ERP Data for the Training Phase

For the experimental groups, a 2 (Condition: 250 ms vs. 750 ms) × 4 (Electrode: Fz, Cz, Pz, and FCz) × 3 (Stage: 1, 2, and 3) mixed ANOVA with N200 amplitude as the dependent variable was conducted. This analysis showed a significant main effect for Electrode [F(3,216) = 12.21, p < 0.001, η <sup>2</sup> = 0.15], with N200 amplitude over Fz being significantly stronger than those over the other three electrodes (ps < 0.001). No other significant effect was found. Notably, the lack of a significant main effect of Condition suggests that N200 component is not sensitive for differentiating the implicit learning processes between the 250 and 750 ms RSI conditions.

The same mixed ANOVA with P300 amplitude as the dependent variable was performed for the experimental groups. This analysis showed a significant main effect for Electrode [F(3,216) = 12.09, p < 0.001, η <sup>2</sup> = 0.14], with P300 amplitude over Fz being significantly weaker than those over the other three electrodes (ps < 0.001; Bonferroni corrected). A significant main effect for Condition was also found [F(1,72) = 6.38, p < 0.05, η <sup>2</sup> = 0.08], with stronger P300 amplitude shown in the 750 ms RSI condition (0.86 µV ± 1.41 µV) than that in the 250 ms RSI condition (0.10 µV ± 0.88 µV) (see **Figures 5**, **6**), suggesting that P300 amplitude was sensitive to different learning processes between the two RSI conditions. No other significant effect was found.

#### ERP Data for the Transfer Phase

To explore the sensitivity of N200 and P300 in detecting transfer effect, experimental groups and control groups were compared on their ERP signals in the transfer phase.

#### **N200 amplitude differences over the transfer phase**

For the 250 ms RSI condition, a 2 (Group: experimental group vs. control group) × 4 (Electrode: Fz, Cz, Pz, FCz) × 3 (Stage: 4, 5, and 6) mixed ANOVA with N200 amplitude as the dependent variable was conducted. This analysis showed a significant main effect for Electrode: F(3,276) = 13.16, p <0.001, η <sup>2</sup> = 0.16, with N200 amplitude over Fz (−2.06 µV ± 0.29 µV) being significantly stronger than those over the other three electrodes (−0.62 µV ± 0.31 µV; −0.22 µV ± 0.24 µV; −0.88 µV ± 0.19 µV) (ps < 0.001, Bonferroni corrected). A significant Group × Electrode interaction effect was also found: F(3,276) = 9.13, p < 0.001, η <sup>2</sup> = 0.12. Follow-up independent t tests for this interaction effect showed that N200 amplitude of the experimental group (−3.22 µV ± 3.07 µV) was significantly stronger than that of the control group (−0.89 µV ± 1.59 µV) over Fz [t(73) = −4.06, p < 0.001, Bonferroni corrected].

For the 750 ms RSI condition, the same mixed ANOVA with N200 amplitude as the dependent variable was conducted. This analysis showed a significant main effect for Electrode [F(3,276) = 27.38, p < 0.001, η <sup>2</sup> = 0.28], with N200 amplitude over Pz (0.24 µV ± 2.45 µV) being significantly weaker than that over the other three electrodes (−1.04 µV ± 2.04 µV; −0.85 µV ± 2.23 µV; −1.11 µV ± 2.26 µV) (p < 0.001, Bonferroni corrected). A significant main effect for Group was found [F(1,69) = 44.38, p < 0.001, η <sup>2</sup> = 0.39], with N200 amplitude of the experimental group being stronger than that of the control group. A significant Group × Electrode interaction effect was also found: F(3,276) = 10.6, p < 0.001, η <sup>2</sup> = 0.13. Follow-up independent t-tests for this interaction effect revealed that N200 amplitudes of the experimental group (−1.86 µV ± 1.50 µV; −1.49 µV ± 1.31 µV; −2.35 µV ± 1.29 µV) were significantly stronger than those of the control group (0.23 µV ± 2.39 µV; 2.13 µV ± 1.97 µV; 0.23 µV ± 2.32 µV) over Cz, Pz, and FCz [t(73) = −4.57, t(73) = −9.46, t(73) = −6.02, ps < 0.01; Bonferroni corrected].

#### **P300 amplitude differences over the transfer phase**

For the 250 ms RSI condition, 2 (Group: experimental group vs. control group) × 4 (Electrode: Fz, Cz, Pz, FCz) × 3 (Stage: 4, 5, and 6) mixed ANOVA with P300 amplitude as the dependent variable yielded a significant main effect for Electrode [F(3,276) = 14.57, p < 0.001, η <sup>2</sup> = 0.17], with P300 amplitudes over FCz (0.43 µV ± 2.13 µV) and Pz (0.01 µV ± 1.62 µV) being significantly stronger that those over Fz (−1.50 µV ± 2.29 µV) and Cz (−0.58 µV ± 1.79 µV) (ps < 0.001, Bonferroni corrected). Another significant Group × Electrode interaction effect was also found: F(3,276) = 9.65, p < 0.001, η <sup>2</sup> = 0.12. Follow-up independent t-tests for this interaction effect showed that P300 amplitude in the control group (−0.56 µV ± 1.44 µV) was stronger than that in the experimental group (−2.38 µV ± 2.59 µV) over Fz [t(73) = −3.73, p < 0.01, Bonferroni corrected].

For the 750 ms RSI condition, the same mixed ANOVA mixed ANOVA with P300 amplitude as the dependent variable yielded a significant main effect for Electrode [F(3,276) = 22.56, p < 0.001, η <sup>2</sup> = 0.25], with P300 amplitude over Pz (0.47 µV ± 1.28 µV) being significantly stronger than those over the other electrodes (−0.54 µV ± 1.37 µV; −0.34 µV ± 1.12 µV; −0.57 µV ± 1.35 µV; ps < 0.001). A significant main effect for Group was also found [F(1,69) = 14.24, p < 0.001, η <sup>2</sup> = 0.17], which was due to stronger P300 amplitude in the control group than the experimental group. Another significant Group × Electrode interaction effect was also found: F(3,276) = 8.81, p < 0.001, η <sup>2</sup> = 0.11. Follow-up independent t tests for this interaction effect showed that P300 amplitudes over Fz, Pz, and FCz (−0.26 µV ± 1.53 µV; 1.38 µV ± 0.92 µV; −0.26 µV ± 1.66 µV) were significantly stronger in the control group than those in the experimental group (−0.81 µV ± 1.17 µV; −0.36 µV ± 0.97 µV; −0.86 µV ± 0.92 µV) [t(73) = 2.00; t(73) = 7.41; t(73) = 2.05, ps < 0.05; Bonferroni corrected].

# DISCUSSION

# Impact of RSI on Non-local Dependent Transfer

Conflicts regarding the transferability of implicit knowledge can be found in early artificial grammar learning studies.

One of the main questions is whether implicit learning involves the acquisition of perceptual patterns or abstract rules. Reber (1967) argued that knowledge obtained from implicit learning was the representation of an abstract rule. However, some other researchers believed that people could only acquire perceptual patterns in implicit learning, such as repeated letter chunks (Perruchet and Vinter, 1998), or specific artificial grammar examples (Vokey and Brooks, 1992; Jamieson and Mewhort, 2009). Both points of view have been supported by some empirical studies (Kuhn and Dienes, 2005, 2006, 2008; Pothos, 2007). However, neither of them has been fully backed up. In light of this, we used a completely abstract non-local dependent rule in the present study. In this non-local dependent design, the SOC2 rule in the transfer phase shares a higher-order structural rule (i.e., the location of a third stimulus is determined by the locations of previous two stimuli), rather than superficial representations with the SOC1 rule in the training phase. Therefore, a transfer effect depends on the acquisition of the higher-order structure, rather than superficial patterns of the training sequences. Our behavioral results showed a significant transfer effect in the 750 ms RSI condition, with no accurate description of the implicit rule being reported, indicating that participants were able to acquire the deep structural rule embedded in repeated sequences without full consciousness. The fact suggests that implicit knowledge can be abstract and transferable in nature under an appropriate RSI condition.

Notably, no significant non-local dependent transfer was found in the 250 ms RSI condition. Previous studies suggest that 250 ms RSI elicits primarily unconscious knowledge, which is bounded to perceptual patterns of sequences and therefore hard to transfer (Cleeremans and Jiménez, 2002; Kuhn and Dienes, 2006; Zhang and Liu, 2014). With elongated RSI, the implicit learning process tends to involve more and more conscious knowledge (i.e., abstract and transferable knowledge), and eventually leads to an effective transfer (Kuhn and Dienes, 2006). Consistently, the absence of non-local dependent transfer in the 250 ms RSI condition in the present study may be related to overly low representation quality of implicit knowledge caused by a limited RSI.

Interestingly, in the present study, no significant learning effects were found in the control groups with 3 blocks of training on SOC2 sequences. This result stands contrast to the significant learning effects with 7 blocks of training on SOC1 sequences in the experimental groups (i.e., with relatively similar sample sizes in the experimental groups and control groups). This finding is consistent with the reports in previous studies, as such that effective implicit learning could only occur with appropriate length of training (Cleeremans and Jiménez, 2002; Destrebecqz and Cleeremans, 2003; Goldstone and Sakamoto, 2003; Cleeremans, 2006). Our finding suggests that a qualitative difference of non-local dependent learning effect can occur between 3 blocks (i.e., 288 trials) and 7 blocks of training (672 trials). However, since no detection of learning effect in the early training stage was set in the present study experimental design, it could not tell us the exact numbers of training trial under which the transition occurred. This query may be explored in future studies using a revised design to detect the occurrence of effective learning in early stage of non-local dependent training.

# Sensitivity of P300 to Changes in the Training Phase

N200 has been though as an index of early processing (i.e., coding and storing) of sequence information (Fu et al., 2013). In the present study, N200 amplitude during the training phase was not significantly different between the 250 and 750 ms RSI experimental groups, suggesting that it is not sensitive for detecting RSI variation. However, this result may also suggest that RSI variation does not affect early processing of sequence information.

P300 has been suggested as indicating participants' subjective estimation of the material being processed, which is closely related to increased consciousness (Stadler et al., 2006). Similar to previous reports (Rugg et al., 1998; Atienza et al., 2003; Chu and Liu, 2010), P300 amplitude was found to be significantly stronger in the 750 ms RSI experimental group than that in the 250 ms RSI experimental group in the present study, which may be caused by the increase of consciousness in the 750 ms RSI experimental group (i.e., despite that no full consciousness was achieve in this group). Moreover, given that effective transfer effect was only found in the 750 ms RSI experimental group, its strengthened P300 amplitude relative to that in the 250 ms RSI experimental group during the training phase may suggest a learning process involves abstract rules rather than perceptual patterns.

# Dissociated Transfer Effects Indexed by ERP Components

Our ERP data of the transfer phase showed that N200 amplitude in the experimental group was significantly stronger than that in the control group over Fz in the 250 ms RSI condition, and over all central electrodes in the 750 ms RSI condition According to Mecklinger (2010), N200 is related to participants' prediction in sequence learning. Strengthened N200 amplitudes in the experimental groups may indicate that the experimental groups were more prone to making prediction of stimulus' location in the SOC2 sequences based on their knowledge of the SOC1 sequences, while the control groups tended to perceive the SOC2 stimuli as randomly presented.

In contrast, P300 amplitudes in the experimental groups were significantly weaker than those in the control groups during the transfer phase in both 250 and 750 ms conditions. According to Kok (2001), increase of P300 amplitude over the central electrodes indicates an increase of attention resources. Therefore, this result may suggest that the control groups activated more attention resources than the experimental groups to process the sequences, which was completely novel to them.

# CONCLUSION

fpsyg-08-02107 December 4, 2017 Time: 17:38 # 9

Overall, the present study preliminarily proved that a non-local dependent design was useful for differentiating the confounding effects of perceptual similarity and structural identical in traditional study paradigms.

Our behavioral results showed that, with enough processing time (i.e., a 750 ms RSI), participants were capable of acquiring abstract and transferable implicit knowledge of non-local dependencies. Our ERP data showed that both N200 and P300 were useful for detecting non-local dependent transfer effect. Increase in N200 amplitude indicates enhanced ability to predict stimuli sequences and decrease in P300 amplitude indicates less attention and effort needed. Moreover, Increase in P300 amplitude may suggest that the learning process involves the acquisition of abstract knowledge (i.e., implicit rules) rather than perceptual knowledge. Moreover, Both N200 amplitude and P300 amplitude varied significantly between the experimental and the control groups, suggesting that they are useful in differentiating a transfer process and a learning process.

Despite the fact that none of the participants achieved full consciousness (i.e., based on their oral reports), training on SOC1 sequences was found to facilitate study on SOC2 sequences under the 750 ms RSI condition, suggesting that abstract knowledge can be acquired with partial consciousness. However, it is unknown whether SOC1 training could continually facilitate the study on SOC2 sequences, or only be temporally effective in the early stage. This question can be explored in follow-up studies using more SOC2 blocks.

### REFERENCES


#### BULLET POINTS


#### AUTHOR CONTRIBUTIONS

Guarantor of integrity of entire study: DL. Study concepts: JH. Study design: JH and HD. Literature research: JH. Clinical studies: JH. Experimental studies: JH and HD. Data acquisition: JH, HD, and JY. Data analysis/interpretation: JH and CZ. Statistical analysis: JH. Manuscript preparation: JH and CZ. Manuscript definition of intellectual content: JH, YL, and DL. Manuscript editing: JH, YL, and DL. Manuscript revision/review: YL and DL. Manuscript final version approval: JH, YL, and DL.

#### ACKNOWLEDGMENT

This research was supported by National Natural Science Foundation of China under Grant 31271084 and Grant 31540025.

Rohrmeier, J. A. Hawkins, and I. Cross (Oxford: Oxford University Press), 156–168.



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Huang, Dai, Ye, Zhu, Li and Liu. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The Emergence of Explicit Knowledge in a Serial Reaction Time Task: The Role of Experienced Fluency and Strength of Representation

#### Sarah Esser\* and Hilde Haider

General Psychology 1, Department of Psychology, University of Cologne, Cologne, Germany

#### Edited by:

Marcus Rothkirch, Charité Universitätsmedizin Berlin, Germany

#### Reviewed by:

Vasil Kolev, Bulgarian Academy of Sciences, Bulgaria Michał Wierzchon,´ Jagiellonian University, Poland

> \*Correspondence: Sarah Esser sarah.esser@uni-koeln.de

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 17 October 2016 Accepted: 17 March 2017 Published: 04 April 2017

#### Citation:

Esser S and Haider H (2017) The Emergence of Explicit Knowledge in a Serial Reaction Time Task: The Role of Experienced Fluency and Strength of Representation. Front. Psychol. 8:502. doi: 10.3389/fpsyg.2017.00502

The Serial Reaction Time Task (SRTT) is an important paradigm to study the properties of unconscious learning processes. One specifically interesting and still controversially discussed topic are the conditions under which unconsciously acquired knowledge becomes conscious knowledge. The different assumptions about the underlying mechanisms can contrastively be separated into two accounts: single system views in which the strengthening of associative weights throughout training gradually turns implicit knowledge into explicit knowledge, and dual system views in which implicit knowledge itself does not become conscious. Rather, it requires a second process which detects changes in performance and is able to acquire conscious knowledge. In a series of three experiments, we manipulated the arrangement of sequential and deviant trials. In an SRTT training, participants either received mini-blocks of sequential trials followed by mini-blocks of deviant trials (22 trials each) or they received sequential and deviant trials mixed randomly. Importantly the number of correct and deviant transitions was the same for both conditions. Experiment 1 showed that both conditions acquired a comparable amount of implicit knowledge, expressed in different test tasks. Experiment 2 further demonstrated that both conditions differed in their subjectively experienced fluency of the task, with more fluency experienced when trained with mini-blocks. Lastly, Experiment 3 revealed that the participants trained with longer mini-blocks of sequential and deviant material developed more explicit knowledge. Results are discussed regarding their compatibility with different assumptions about the emergence of explicit knowledge in an implicit learning situation, especially with respect to the role of metacognitive judgements and more specifically the Unexpected-Event Hypothesis.

Keywords: implicit learning, conscious awareness, fluency, associative strength, serial reaction time task, unexpected events

# INTRODUCTION

Implicit learning refers to our ability to adapt to more or less complex statistical structures in the environment without possessing conscious access to the content of the acquired knowledge, often even lacking any conscious knowledge that something has been learned at all. An accessible example is the knowledge about grammatical rules which develops early and is hard to verbalize even for adults (Reber, 1989; Dienes et al., 1991). But implicit learning processes are also important for sequences of movements (e.g., typing on a keyboard; Nissen and Bullemer, 1987; Willingham, 1998) or stimuli (e.g., visual, spatial, or auditive; Haider et al., 2012; Ling et al., 2016), as well as for social interactions (Heerey and Velani, 2010; Norman and Price, 2012).

There are two very prominent paradigms for studying these so-called implicit learning processes: One is the Artificial Grammar Learning Task (AGLT; Reber, 1989) and the other one is the Serial Reaction Time Task (SRTT; Nissen and Bullemer, 1987). In an AGL task participants are confronted with letter strings that are constructed from artificial probabilistic grammar rules. In a following test task participants are usually asked to classify test strings as following or not following this rule. In a SRTT participants react to a series of different target stimuli by choosing a corresponding response. Unbeknownst to the participant the stimuli and/or the responses follow a certain probabilistic or deterministic sequence. Test tasks can, mostly depending on the definition of consciousness, range from recognition (Shanks and Johnstone, 1999), over free generation (Wilkinson and Shanks, 2004), process dissociation (Jacoby, 1991; Destrebecqz and Cleeremans, 2001), and subjective measures (Persaud et al., 2007; Haider et al., 2011) to verbal report (Eriksen, 1960; Rünger and Frensch, 2010). In both paradigms, the SRTT and the AGLT, participants are usually unaware of the fact that their task followed a certain rule or sequence, while their performance shows that, in fact, they have acquired some knowledge about these [see e.g., Rünger and Frensch, 2010, for a discussion about the adequacy of the different measures for (un-)conscious knowledge].

In implicit learning research, a lot of effort is dedicated to the question about the complexity, the flexibility and the structure of the knowledge we extract from predictable structures (see e.g., Abrahamse et al., 2010, for an overview). Here, our goal is to focus not on implicit learning processes per se, but on the question: Why and how do some participants acquire conscious knowledge about the unconsciously learned sequences? The paradigms used in implicit learning research can constitute very useful tools for addressing exactly this important topic.

One undoubtedly important aspect that divides unconscious from conscious information processing is signal strength (Kanwisher, 2001; Cleeremans and Jiménez, 2002). While research on subliminal priming mostly leads to the conclusion that perception seems to be unconscious when the signal strength is very weak, there is more disagreement about whether having a high signal strength is a sufficient condition for a bottom-up signal to become conscious (Dehaene et al., 2006). Depending strongly on the definition of consciousness and a potential division between phenomenal and access consciousness (Block, 2005; Kouider et al., 2010; Cohen and Dennett, 2011), some claim that strong activity in specialized local circuits (e.g., in extrastriate areas; Lamme, 2003, 2010; Zeki, 2003; Block, 2007) is sufficient and, thereby, allowing different, gradual qualities of conscious perception (Overgaard et al., 2006; Nieuwenhuis and de Kleijn, 2011; Windey et al., 2014). Otherresearchers assume that a global, cortical "ignition," involving parieto-frontal networks which allow for a top-down amplification (Dehaene and Naccache, 2001; Dehaene et al., 2003; Lau and Passingham, 2006; Dehaene and Changeux, 2011), is correlated with conscious information processing. In this latter class of theories, consciousness is usually seen as an all-or-none matter (Sergent and Dehaene, 2004; Kouider et al., 2010). While opinions still strongly differ about the definition of conscious processing, there is more agreement that global processing is associated with reportability and highly flexible, strategic usage of the respective knowledge (Block, 2005; Lamme, 2010; Cohen and Dennett, 2011). When we further speak of conscious or explicit knowledge in this article, we refer to this highly flexible, reportable form of knowledge.

The debate about the role of bottom-up signal strength and additional higher-order top-down processes is also reflected in the proposed mechanisms for the transformation from implicit to explicit knowledge which shall further be the subject of this paper. While there certainly are more diverging theories than can be discussed here in detail, most of them can roughly be differentiated by the role of the strength of associative weights of the implicitly learned structure. Some theories, implying a quantitative, gradual difference, assume that representational strength and stability is the deciding factor for separating unconscious from conscious processing. By contrast, others assume qualitative differences where additional, higher order processes are necessary to transfer unconscious into conscious knowledge.

The former of these accounts is the more parsimonious one. No qualitative separation between conscious and unconscious representation is assumed, but rather a gradual transition. These theories do not need any additional, hierarchically higher, mechanisms that transform unconscious into conscious representations. Therefore, these models are often referred to as single-system views. One and the same learning, respectively memory system can explain fast reaction times or simple discriminative decisions when its associations are relatively weak and, as these grow stronger throughout training, can account for verbally accessible and highly flexible knowledge (Cleeremans and Jiménez, 2002; Perruchet and Vinter, 2002; Shanks and Perruchet, 2002; Destrebecqz and Cleeremans, 2003).

In contrast to these theories, there are so-called multiplesystem views which assume a qualitative distinction between unconscious and conscious representations. According to these accounts, representational strength is not enough to explain the differences between both forms of knowledge. Instead, consciousness is defined by a distinct form of information processing. Therefore, it either has to be explained how unconscious information is granted access to this particular form of processing, potentially involving top-down amplification and global processing (Keele et al., 2003), or how an independent learning system builds explicit knowledge on its own (Reber, 1989; Willingham, 1998; Sun et al., 2001; Frensch et al., 2003; Haider and Frensch, 2005; Scott and Dienes, 2010).

There are hierarchical models that build a bridge between stricter single-system and multiple-systems views. An interesting account that fits into this position comes from Cleeremans et al. (Cleeremans, 2011; Timmermans et al., 2012). Similar to singlesystem views, they regard the gradually developing strength, stability, and distinctiveness of representations as necessary conditions for gradually developing explicit knowledge. Nevertheless, they also include Higher-Order Thought (HOT) Theories of consciousness (Rosenthal, 1997; Lau, 2008) into their theory. HOT Theories assume that consciousness develops if a hierarchically built system is able to represent that it knows or does not know something (metacognition). In Cleeremans (2011) theory, a simple feed-forward backpropagation network develops first-order knowledge about any sequential structure which enables predictive behavior. A similarly built, hierarchical higher network simultaneously learns about the states of the first-order system when this has made a correct or an incorrect prediction and thereby develops second-order knowledge about the first-order system possessing or not possessing knowledge in a given situation. The assumption that the higher order mechanism operates in the same way as the first order mechanism makes this model a hybrid between single- and multiple-system views. While multiple hierarchical layers of learning are required for consciousness to develop, the same strengthening mechanism is assumed for the generation of implicit and explicit knowledge. Consciousness about the implicitly learned sequence is still a gradual state, changing with every trial from not-knowing to knowing whether something is known, depending on the strength of the associative weights in the higher-order network.

Metacognition also plays an important role in other multiplesystem views about the transition from implicit to explicit knowledge. In the Unexpected Event Hypothesis (UEH), implicitly learned contents are assumed to be encapsulated in local modules. They can neither become conscious themselves just by increasing strength throughout the learning process, nor does the explicit learning process have direct access to the implicit information. Instead, an indirect link is assumed, by which explicit knowledge can develop when implicit learning leads to observable, consciously perceivable, unexpected changes in a person's behavior (Frensch et al., 2003; Haider and Frensch, 2005, 2009; Rünger and Frensch, 2008). For example, having learned a motor sequence implicitly in an SRT task might lead to premature responses before the next target stimulus appears (Haider and Frensch, 2009). This in turn might surprise the participant, who might then start an attributive search process about the reason for their ability to respond prematurely which often leads to the detection of the underlying sequence. Still, as long as there are other more obvious options the unexpected change in behavior might be attributed to, search processes might be terminated and no explicit knowledge might develop (Haider and Frensch, 2005). Importantly, while the implicit learning process is assumed to develop knowledge by a rather slow strengthening of associative weights (Cleeremans and Dienes, 2008), the explicit learning mechanism does not operate in the same way. Instead of accumulating strength via feedback about the correctness of predictions, explicit attributional search processes result in sudden insights following the unexpected occurrence of predictive behavior. In an SRT task these sudden insights are characterized by an abrupt drop in the reaction times (Haider and Rose, 2007; Haider et al., 2011) as well as an increased coupling of gamma-band activity between the right prefrontal and occipital regions (Rose et al., 2010; Wessel et al., 2012). Hence, according to the UEH, consciousness about the sequence is not seen as a gradual matter, relying on a slow strengthening of associations on a trial-by trial basis, as proposed by singlesystem views or Cleeremans' hybrid model (Cleeremans, 2011). Nevertheless, the UEH would not dispute the idea that there are higher-order learning processes which develop knowledge about first-order states and are relevant for assessing what the system knows or does not know. These learning processes might for example be important for the accuracy of judgements of knowledge. Increasing accuracy of judgements of knowledge might in turn serve as a consciously perceivable, unexpected change in behavior, which again leads to conscious sequence knowledge. In the UEH any metacognitive judgement about one's own behavior (perceiving increased fluency, accuracy, speed etc. or unexpected mismatches in these aspects when the sequential structure is suddenly replaced with irregular trials) can serve as an unexpected event, triggering attributive processes.

Scott and Dienes (2008, 2010) proposed a similar account with implicit learning leading to explicit judgement knowledge (gradual improvement of the accuracy of second-order judgements). Experiencing this meta-knowledge about one's own behavior can trigger a second attributional process leading to explicit structure knowledge (insight into the sequential rules). Within the AGL paradigm, they have shown that participants first notice an increasing correlation between their feeling of familiarity for learned grammar strings and their ability to discriminate learned from new grammar strings (explicit judgement knowledge). Perceiving this surprising ability, a second explicit learning process is triggered, leading to explicit structure knowledge of the grammatical rules.

It is beyond the scope of this paper to discriminate between the finer differences of these proposals. Rather, the aim is to offer some insights into the fundamental assumptions about the role of the strengthening of associative weights for the transition from implicit to explicit knowledge. So far, not many studies have empirically explored whether the same associative strength can result in different development of explicit sequence knowledge. While single-system, respectively strengthening accounts would not expect such a difference in explicit knowledge, multiplesystem accounts like the UEH would allow different extents of explicit knowledge for the same associative strengths, as long as the learning situations lead to different metacognitive judgements of one's own behavior.

### Rationale of the Experiments

In order to test the role of the strengthening of associative weights against the role of metacognitive knowledge, we aimed to create a situation where the associative strength is kept constant between two conditions but the metacognitive judgements, or more specifically here, the subjective feelings of fluency differ. We assume that participants who have the opportunity to experience unexpected differences in their feeling of fluency are likely to use this feeling as a trigger for explicit search processes, ultimately resulting in more explicit sequence knowledge.

For testing this, we used an SRTT with the important modification of the arrangement of trials that follow the sequence (regular trials) and those that violate the sequence (deviant trials). More precisely, we aimed to keep the number of correct and incorrect transitions throughout the experiments equal between all conditions. However, in one condition the regular trials and deviant trials were arranged in alternating mini-blocks, while they were arranged randomly in the other condition. It is assumed that, due to the equal number of correct and incorrect transitions, all conditions should be able to develop the same strength of associative weight and gain a comparable rate of implicit knowledge (Experiment 1). Concurrently, the difference in the arrangement of trials should lead to differences in the experienced fluency of the task. Participants working with miniblocks of regular and random trials should tend to experience fluency differences between regular and random material, while participants with randomly arranged material should not experience such differences (Experiment 2). Participants who experience differences in the fluency of the task should have a tendency to search for the reasons of these experienced differences and hence show a greater likelihood to develop explicit sequence knowledge (Experiment 3). All experiments used the same method.

# GENERAL METHOD

#### Stimulus and Apparatus

In our version of the SRTT (Haider et al., 2012), a squared target stimulus appeared in the upper third of the screen. The target stimulus had a size of about 3 × 3 cm on a 17-inch screen and contained a picture of one of six different colored circles. In the lower two thirds of the screen six squared response stimuli were presented in a triangular space, with each stimulus having a size of about 2.8 × 2.8 cm. On each trial, the response stimuli displayed the six possible target stimuli. The arrangement of the six possible target stimuli within the response squares was changed from trial to trial (**Figure 1A**). The locations of the response stimuli were spatially mapped to the Y, X, C, B, N, and M keys of a German QWERTZ-keyboard. The participants were instructed to let their index-, ring-, and middle-fingers rest on the six keys for the entire experiment and to press the key that spatially corresponded to the response stimulus which contained the target stimulus.

In all three experiments, 50% of the locations of the correct response stimulus followed a regular sequence, which resulted in a motor sequence of key strokes for the participants. Described as response positions 1–6, the resulting sequence was 1-6-4-2- 3-5. The colored target stimuli were presented randomly with the constraints that all six possible targets had occurred before being shown again and that no response stimulus position would contain the same stimulus more than twice consecutively. In the other 50% of the trials, both the colored target stimuli and the response stimulus locations were presented randomly, with the constraints that no position would contain the target stimulus successively and that all six positions were to be used equally often in each of the training blocks. The crucial manipulation in all three experiments was the arrangement of the regular and random trials. In the Blocked-Order Condition (BO-Condition), 22 regular trials were always followed by 22 random trials in the first two experiments (**Figure 1B**). In Experiment 3, the length of these mini-blocks was extended to 88 trials (**Figure 1C**). In the Random-Order-Condition (RO-Condition) random and sequential trials were mixed randomly (**Figure 1D**). All participants received the same material, but the starting point was assigned randomly for each participant.

There were very slight differences in the proportion of regular and deviant trials in both conditions. While the BO-Condition had 49% regular and 51% deviant trials, the RO-Condition contained 53% regular and 47% deviant trials in all experiments. If these small differences, against our intention and expectation, made a relevant difference for the associative strengths, then the RO-Condition would build stronger associations, which would not be in favor of our hypotheses.

#### Procedure

Each experiment consisted of a training and a test phase. The training started with the presentation of the instructions on the computer screen. Participants were instructed to react as fast as possible while also avoiding mistakes. They were informed that they would first receive 20 practice trials to get a first impression of the task. These practice trials consisted of random material that followed the same constraints as the random material described above. Each trial of the task began with the presentation of the six response stimuli. After 100 ms the target stimulus appeared for 150 ms. After the participant's response the screen went black for 300 ms before showing the next six response stimuli in a different arrangement. The course of a trial was identical in the test phases. After the respective test, participants were debriefed and received their financial reward or their course credit.

## EXPERIMENT 1

The aim of Experiment 1 was to show that the extent of implicit knowledge depends on the amount of regular transitions presented during training, not on the respective arrangement of regular and random trials. As it is conceivable that the different arrangements of the training material cause participants to behave differently in the training tasks, even though they possess comparable amounts of knowledge, we assessed the extent of acquired knowledge with two different test tasks after training. Half of the participants were tested with a wagering task (Persaud et al., 2007; Haider et al., 2011). The other half received additional test blocks in which they were confronted with blocks of either sequential or random trials only. With the use of two different test tasks we aimed to assess different aspects of implicit learning. The wagering task assesses implicit knowledge without resorting to reaction times by asking participants to predict the next response. By additionally asking participants to wager on the correctness

sequential trials mixed randomly.

of their guess, this test enables us to detect whether participants possess explicit knowledge. The three additional test blocks in the second test are especially important as they assess knowledge in terms of the usual performance differences between random and regular blocks. Thus, it should detect knowledge with the same sensitivity as the training phase (Shanks and St. John, 1994). This would allow us to test whether any possible training differences between the two conditions reflect different amounts of knowledge or different expressions of a comparable extent of learning. If it is only the amount of regular transitions that influences learning, the two conditions should not differ in the two tests. By contrast, if the arrangement of trials matters, then the two conditions should differ in the two test tasks.

#### Method

#### Participants

One hundred-twenty students of the University of Cologne (81 women) participated either in fulfillment of course credit or in return for payment, with 60 participants in the Blocked-Order and 60 participants in the Random-Order Condition which they were randomly assigned to. The mean age was 24.4 years (range: 18–47, SD = 5.28). All participants reported normal or correctedto-normal vision.

#### Procedure

The procedure of the SRTT training was as described in the General Method Section. The training task consisted of seven blocks with 176 trials (4 × 22 sequential trials alternating with 4 × 22 random trials in the BO-Condition).

For 60 participants (30 in each condition) the training ended after these seven training blocks and they received the instructions for the following wagering task. The wagering task had the same design as the training task with two important exceptions. First, it only contained regular trials. Second, on 36 of the 176 trials, the target stimulus was replaced with a question mark. Participants were instructed to respond with the key they considered to be the most likely response after the previous trial. The question mark remained on the screen until the participant gave a response. Subsequently, they were asked to indicate how sure they were about the correctness of their prediction by wagering either 1 or 50 Cent on their response. They were informed that they would earn the amount if their prediction was correct and to lose it if it was incorrect. Following the logic of the zero-correlation criterion (Dienes and Perner, 1999; Dienes, 2008), this task allows to assess whether participants possess implicit or explicit knowledge about the sequence. Participants with implicit knowledge give more correct answers than to be expected by mere guessing, while showing no correlation between the correctness of their response and their confidence (the amount wagered). By contrast, participants with explicit knowledge should demonstrate a high rate of correct responses that is above chance-level and a strong correlation between the correctness of their response and their confidence.

For 60 participants (30 in each condition) the seven training blocks were followed, without any special notice or new instruction, by three test blocks. Their only difference to the training blocks was that the first and last test block contained purely random trials, whereas the second block consisted of solely regular trials.

#### Results

We had to exclude six participants due to a technical error with one computer (four in the BO-Condition). For the remaining 114 participants we analyzed the mean error-rate for each participant. Participants were excluded if their error rate, averaged over all seven blocks, exceeded 15%. This led to the exclusion of four participants in the BO-Condition and six participants in the RO-Condition. Also one person of the BO-Condition had to be excluded, because they did not finish the test task. For the remaining 51 participants in the BO-Condition and 52 participants in the RO-Condition, individual median reactions times (median RTs) were computed for each block. We excluded errors and post-error trials from this computation (10.027%). We excluded post-error trials because of post-error slowing (Ruitenberg et al., 2014) and because trials after an erroneous response also are based on a wrong transition. Also RTs larger than 3000 ms (0.346%) were excluded.

#### Training Phase

**Tables 1A,B** depict the mean percent error rates and the means of the median RTs of the training. For both dependent variables, we conducted a 2 (Condition) × 2 (Trial Type: regular vs. deviant trial) × 7 (Block) mixed-design ANOVA.

For mean error rates, a 2 (Condition) × 2 (Trial Type: regular vs. deviant trial) × 7 (Block) repeated measures ANOVA yielded a main effect of Block [F(6, 606) = 23.77, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.190] and of Trial Type [F(1, 101) = 76.18, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.430]. As can be seen from **Table 1A**, error rates decreased over the course of training and error rate was higher on deviant trials. There also was a main effect of Condition [F(1, 101) = 5.57, p = 0.020, η<sup>p</sup> 2 = 0.052], due to the RO-Condition making more errors (6.3% for the RO-, 4.9% for the BO-Condition). The interaction did not reach the level of significance.

For median RTs a 2 (Condition) × 7 (Block) × 2 (Trial Type: regular/deviant) mixed-design ANOVA revealed a significant main effect of Block [F(6, 606) = 140.33, p < 0.0001, η<sup>p</sup> <sup>2</sup> = 0.581] and of Trial Type [F(1, 101) = 138.34, p < 0.0001, η<sup>p</sup> <sup>2</sup> = 0.578]. **Table 1B** shows that participants in both conditions became faster over the course of the training and responded faster to regular than to random trials. There also was a significant Block × Trial Type interaction [F(6, 606) = 4.34, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.041], indicating that the differences between regular and deviant trials increased over the course of the training, representing a general learning effect. Follow-up interaction contrasts (Block 1 vs. 7 × Trial Type) revealed that both conditions showed a sequence learning effect [F(1, 101) = 5.41, p = 0.022, η<sup>p</sup> <sup>2</sup> = 0.051 for the RO-Condition and F(1, 101) = 9.60, p = 0.003, η<sup>p</sup> <sup>2</sup> = 0.087 for the BO-Condition]. No other interaction reached the level of significance. There was a trend toward a significant Condition × Trial Type interaction [F(1, 101) = 3.07, p = 0.083, η<sup>p</sup> <sup>2</sup> = 0.030]. This trend might be due to a somewhat larger overall difference between regular and deviant trials in the BO-Condition than in the RO-Condition. Further, post-hoc tests point to the circumstance that the larger difference between regular and deviant trials within the BO-Condition is only present in the last three [F(1, 101) = 8.49, p = 0.004, η<sup>p</sup> <sup>2</sup> = 0.078] but not in the first three blocks (F < 1). Taken together the trend toward a Condition × Trial Type interaction might indicate that, against our assumptions, there could be a difference in the extent of acquired knowledge between the two conditions with the BO-Condition acquiring more knowledge than the RO-Condition.

#### Test Phase: Wagering Task

We first tested whether the participants' acquired sequence knowledge is better than expected by chance. For this purpose, we tested the proportion of correct predictions in each condition against a chance level of 20% (which implies that participants only know that the same button is never to be pressed successively). In both conditions the proportion of correct predictions was significantly higher than chance level [t(26) = 2.70, p = 0.006, one-tailed, for the RO-Condition; t(24) = 2.54, p = 0.009, one-tailed, for the BO-Condition; see **Table 2**]. This further confirms that both conditions acquired knowledge about the sequence which has already been shown in the reaction times of the training phase. Furthermore, the amount of knowledge did not differ between conditions [t(50) = 0.70, p = 0.487].

Since our aim of the experiment was to show that the ROand the BO-conditions did not differ with regard to the strength of their representations of the regular sequence, our main focus was on this Null-hypothesis testing. Therefore, we additionally conducted a Bayes analysis. Following Dienes (2014) we specified our roughly expected maximum effect-size if the hypothesis was true that both conditions differ. Based on the data of Experiment 3 in this paper and data from previous studies (Haider et al., 2011, 2012, 2014) we estimated that the maximum expected difference between the percent correct predictions in the BO- and the RO-Condition can reach 30% [presented as B<sup>H</sup> (0.30%)]. With this estimated effect size the Bayes factor was B<sup>H</sup> (0.30%) = 0.02. According to the conventions of Jeffreys (1939), a B smaller than 1/3 can be taken as substantial evidence for the H<sup>0</sup> (a B > 3 is taken as substantial evidence for the H1). Taken together, these results confirm that both conditions acquired knowledge about


TABLE 1A | Percent error rates and their respective standard deviations (in brackets) by training block, condition, and trial type (regular vs. deviant) in Experiment 1.

TABLE 1B | Means of the median RTs and their respective standard deviations (in brackets) by training block, condition, and trial type (regular vs. deviant) in Experiment 1.


TABLE 2 | Percent correct predictions, percent high wagers given when the prediction was correct, percent high wagers given when the prediction was false, and their respective standard deviations (in brackets) by condition in Experiment 1.


the sequence and that the amount of acquired knowledge did not differ between both conditions.

Next, we investigated whether this knowledge was explicit. If so, participants should be able to strategically use their knowledge to maximize their gains. We compared the proportion of high wagers when participants made correct predictions with the proportion of high wagers when they made a false prediction (see **Table 2**). A 2 (Condition) × 2 (Wager Type: high|false vs. high|correct) mixed-design ANOVA yielded no significant effect (all Fs < 1). Hence, it seems that both groups acquired a comparable amount of merely implicit sequence knowledge.

To exclude that the participants acquired explicit knowledge during the test task, we contrasted the difference between high|correct and high|false wagers in the first 12 wager trials with that of the last 12 wager trials for both conditions. No Condition showed a significant increase in the difference between high|correct and high|false wagers (both ts < 1).

#### Test Phase: Transfer Blocks

For the error rates, a 2 (Condition) × 2 (Test Block 8/10 vs. 9) mixed-design ANOVA revealed a significant effect of Block [F(1, 49) = 18.93, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.279] indicating less errors in the regular than in the random blocks. Neither the main effect of condition (F < 1) nor the Condition × Block interaction reached the level of significance [F(1, 49) = 2.79, p = 0.101, η<sup>p</sup> <sup>2</sup> = 0.054].

**Figure 2** shows the mean median reaction times of both conditions for the test phase. A 2 (Condition) × 2 (Block Type 8/10 vs. 9) mixed-design ANOVA only revealed a significant main effect for Block [F(1, 49) = 17.84, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.267]. As can be seen from **Figure 2**, participants in both conditions responded slower in the random blocks than in the sequential block.

In order to test whether the data of the transfer test speak for our null-hypothesis that both conditions do not differ in their acquired knowledge we again used Bayes statistics. Therefore, we needed to specify the maximum plausible effect for the difference between regular and deviant transfer test blocks for both conditions. If both conditions did differ in their acquired knowledge, the BO-Condition should show a greater difference than the RO-Condition. We set 50 ms as the maximum plausible difference between the difference of regular and deviant trials

in the RO-, respectively, the BO-Condition. These 40 ms were taken as an estimation because of (a) the data from the last three training blocks of Experiment 3 and (b) the data of two Experiments of Eberhardt et al. (manuscript submitted for publication). The former was chosen because there we predicted a difference in the amount acquired knowledge. In the latter two, participants were also trained with material arranged similar to our RO-Condition and later tested with a comparable transfer test, with the difference that we predicted and found a difference in the amount of acquired knowledge due to different instructions. With this estimated effect size the Bayes factor was B (0.40 ms) = 0.14. A B < 1/3 indicates that the data of the transfer block are in favor of the null-hypothesis. Thus, both conditions do not seem to differ in the amount of the participants' acquired knowledge.

### Discussion

The results of Experiment 1 show that participants in both conditions acquired sequence knowledge during training, indicated by the decrease of RTs and the significant Block × Trial Type interaction. However, there was a statistical trend in the Condition × Trial Type interaction with the BO-Condition showing a somewhat larger difference between regular and deviant trials than the RO-Condition, particularly in the last three training blocks. At first glance, this might contradict our hypothesis that the amount of correct and incorrect transitions determines the extent of acquired sequence knowledge, not the arrangement of these trials.

However, when, in the two test phases, participants in both conditions were given the same chance to express their knowledge the results cast a different impression on the data. The wager task shows that both conditions acquired knowledge about the sequential structure. More importantly, it also reveals no difference in the amount of acquired knowledge between the two conditions as they made a comparable amount of correct predictions. The data of the transfer test also support this finding. Both conditions show similar differences between regular and deviant trials and neither the effect of Condition nor the Condition × Block interaction was significant. This is especially important as it might be argued that the wager task is less sensitive and therefore not able to detect small differences in participants' knowledge (Shanks and St. John, 1994). The transfer test gives the participant the possibility to express their knowledge in the exact same way as they did in the training phase. This further fortifies the impression that both conditions acquired a comparable amount of knowledge.

Thus, it seems likely that the differences between conditions found in the training task do not indicate a difference in the amount of acquired knowledge, but rather in the way participants expressed this knowledge in the training task. It is conceivable that the participants in the BO-Condition might have noticed the blocked structure of the training blocks and also might have experienced a fluency difference between the random and the regular mini-blocks. This might have helped them to adjust their performance accordingly. When a participant experiences more fluency on the regular trials and less on the deviant trials, they might, for example, be able to focus more on speed on the regular and on accuracy on the deviant trials.

Taken together, the results of Experiment 1 suggest that the different arrangement of the trial types does not affect the amount of implicitly acquired knowledge and therefore provides a solid basis for further investigating whether both groups indeed experience a difference in the subjective feeling of fluency.

## EXPERIMENT 2

Experiment 2 tested whether the different arrangement of trials types in the BO- and the RO-conditions affect the experienced subjective feelings of fluency. The findings of Experiment 1 build an important precondition for testing this as they exclude the possibility that a potential difference in experienced fluency can simply be explained by a different amount of acquired knowledge in the two conditions. It is assumed that the blocked arrangement of the trial types leads to perceivable differences in the fluency of deviant and regular trials while the random arrangement should make it unlikely to experience such differences. In order to assess the subjective feeling of fluency, we used the same training phase as in Experiment 1 but a different test phase. Here, participants were asked to compare the subjective fluency of short blocks of regular and deviant trials. If our hypothesis is correct, participants in the BO-Condition should be sensitive to the differences in the fluency of the regular and deviant test trials whereas participants in the RO-Condition should not, or at least to a lesser extent. Since it cannot be excluded that the participants in the BO-Condition adapt to this blocked structure of the test task rather quickly, we also included a control condition in which participants were trained with only random material (All-Random; AR-Condition).

# Method

#### Participants

Sixty students of the University of Cologne (48 women) participated either in fulfillment of course credit or in return for payment, with 20 participants in each of the three conditions, which they were randomly assigned to. The mean age was 23.65 years (range: 17–33; SD = 3.68). All participants reported normal or corrected-to-normal vision.

#### Procedure

The training task was designed as described in the General Method Section. There were only two slight differences to Experiment 1. One difference was the stimuli being used; instead of the rather complex colored circles from Experiment 1, we now used simpler colored squares (the colors were: green, pink, cyan, yellow, blue, and red). The other difference was that the training task only consisted of six blocks this time.

In the test task, all participants were asked to compare two consecutive mini-blocks of 18 trials each and rate which of the two blocks felt more fluent to them. One mini-block contained the learned sequence and the other block consisted of random trials only. In order to reach maximal similarity to the sequential trials, the random trials now had the additional constraint that all six positions had to be used, before they could be repeated and no "runs" (e.g., position 1 to 2, 2 to 3, etc.) were allowed. The order of the mini-blocks was random. The first of the mini-blocks was always announced with the notion "1st half " and the second with "2nd half " appearing on the center of the screen. After having responded to the two mini-blocks, participants were asked to rate which of the two halves felt more fluent to them by pressing "1" for the first and "2" for the second half. In total, there were five pairs of such mini-blocks, so participants gave five comparative fluency ratings. The first pair of mini-blocks did not contain any regular trials. It only served to sensitize the participants for experiencing different subjective feelings of fluency: The miniblock that was designed to feel more fluent was structurally equal to the training sequence (i.e., all six keys had to be used before a repetition was allowed, the fingers that had to be used alternated between the left and the right hand). Additionally, the response stimulus containing the target color on trial t already appeared on the correct location on trial t–1. The mini-block designed to feel less fluent did not have any of these properties and the only constraints were avoidance of repetitions and runs. The data of this block were excluded from further analyses. Besides a general explanation of the test task, participants were instructed to focus on their subjective feeling of fluency and to try not to focus on the factors in the background of the task that might have an influence on this feeling.

If the hypothesis that the participants in the BO-Condition experience differences in the fluency of regular and deviant trials during training, while participants in the RO-Condition do not, is correct, then participants in the BO-Condition should rate the regular mini-blocks as feeling more fluent more often than the RO- or AR-Condition.

#### Results

One participant in the AR-Condition had to be excluded because they did not finish the test task. For the remaining 59 participants we analyzed the individual mean error-rates. According to our error criterion of Experiment 1, two participants in the AR-Condition and one participant in the BO-Condition were excluded. For the remaining 56 participants, individual median RTs were computed for each block. Again, we excluded errors and post-error trials (11.43%) as well as RTs > 3000 ms (0.07%) from this computation.

#### Training Phase

**Tables 3A,B** show mean percent error rates and mean of the median RTs for each condition and for each training block. We conducted 2 (Condition) × 6 (Block) × (Sequence Type: regular/deviant) mixed-design ANOVAs for each of the dependent variables to compare the two experimental conditions. In a second step, we then analyzed the difference between these experimental conditions and the AR-Condition with a 3 (Condition) × 6 (Block) mixed-design ANOVA.

For the error rates the first ANOVA revealed a significant main effect of Block [F(5, 185) = 3.39, p = 0.006, η<sup>p</sup> <sup>2</sup> = 0.084]. This effect could not be attributed to specific differences between single blocks by post-hoc tests, though there seemed to be a numerical trend for participants making slightly more errors in the middle of the experiment (lowest mean error rate: Block 1 = 5.7%, highest mean error rate: Block 3 = 7.7%, Block 6 = 6.8%). There also was a significant main effect of Sequence Type, with more errors being made on deviant than on regular trials [F(1, 37) = 26.37, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.416]. There were no significant interactions.

The second ANOVA (with all three conditions included) yielded a main effect of Block [F(5, 265) = 0.022, p = 0.090, η<sup>p</sup> <sup>2</sup> = 0.035; all other Fs < 1]. Post-hoc tests suggest that all conditions showed slightly more errors in the sixth than in the first block [F(1, 53) = 3.52, p = 0.066, η<sup>p</sup> <sup>2</sup> = 0.062]. There was no Condition × Block Interaction (F < 1).

For the RT, the first ANOVA revealed significant main effects of Block [F(5, 185) = 28.35, p > 0.001, η<sup>p</sup> <sup>2</sup> = 0.434] and Trial Type [F(1, 37) = 101.21, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.732]. In addition, the Condition × Trial Type interaction [F(1, 37) = 14.19, p < 0.001, ηp <sup>2</sup> = 0.277], and the three-way interaction [F(5, 412) = 2.58, p


TABLE 3A | Percent error rates and their respective standard deviations (in brackets) by training block, condition, and trial type (regular vs. deviant) in the training phase of Experiment 2.

TABLE 3B | Means of the median RTs and their respective standard deviations (in brackets) by training block, condition, and trial type (regular vs. deviant) in the training phase of Experiment 2.


= 0.028, η<sup>p</sup> <sup>2</sup> = 0.065] were significant (all other effects Fs < 1.5, p > 0.4).

Thus, the picture of the results is a bit more complex than it was in Experiment 1. As can be seen from **Table 3B**, the mean RTs in both conditions decreased due to practice and deviants led to generally slower reaction times than regular trials. However, we did not find a significant Block × Trial Type interaction (F < 1). This might be partially due to the BO-Condition showing large RT-differences between regular and deviant trials from Block 1 on [F(1, 37) = 17.68, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.323] whereas the RO-Condition merely showed a trend of an increasing difference between regular and deviant trials, when comparing Block 1 with Block 6 [F(1, 37) = 3.24, p = 0.080, η<sup>p</sup> <sup>2</sup> = 0.081]. In parts, these differences between conditions already explain the significant Condition × Trial Type and the Condition × Block × Trial Type interactions. Post-hoc comparisons additionally showed that the participants in the BO-Condition responded faster to regular trials than participants in the RO condition [F(1, 37) = 3.73, p = 0.060, η<sup>p</sup> <sup>2</sup> = 0.092], whereas the deviant trials showed no such difference [F(1, 37) = 1.74, p = 0.190).

The second ANOVA only showed a main effect for Block [F(5, 265) = 24.37, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.315], suggesting a general practice effect for all conditions. No other effect was significant (Fs < 1.50, ps > 0.3).

#### Test Phase: Reaction Times

As a manipulation check, we first computed a 3 (Condition) × 4 (Block) × 2 (Mini-Block: regular/deviant) mixed-design ANOVA with mean RTs in the test phase as dependent variable. The ANOVA yielded a significant main effect of Mini-Block, with regular mini-blocks leading to shorter median RTs than deviant mini-blocks [F(1, 53) = 4.12, p = 0.047, η<sup>p</sup> <sup>2</sup> = 0.072]. This main effect was qualified by a significant Condition × Mini-Block interaction [F(2, 53) = 8.85, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.250] and a significant Condition × Block × Mini-Block interaction [F(6, 159) = 2.46, p = 0.0264, η<sup>p</sup> <sup>2</sup> = 0.085]. Additionally, the Block × Mini-Block interaction was significant [F(3, 159) = 3.31, p = 0.022 η<sup>p</sup> 2 = 0.059; all other Fs < 1].

As can be seen from **Table 4**, the first two interactions were mainly due to different performance patterns in the ARvs. the BO- and the RO-Conditions. Participants in the AR-Condition showed faster median RTs for the deviant miniblocks than for the regular mini-blocks [F(1, 53) = 4.81, p = 0.033, η<sup>p</sup> <sup>2</sup> = 0.083]. By contrast, participants in the RO- and the BO-Conditions showed the expected reversed pattern [faster responses for regular than for random trials: F(1, 53) = 12.05, p = 0.001, η<sup>p</sup> <sup>2</sup> = 0.185, F(1, 53) = 6.22, p = 0.016, η<sup>p</sup> <sup>2</sup> = 0.105 for the RO- and the BO-Condition, respectively]. They did not differ significantly from each other (F < 1). This resembles the results of the transfer test of Experiment 1 and strengthens the assumption that both conditions developed at least some sequence knowledge. In the RO-Condition, this difference between mini-block type increased with practice showing larger RT-differences in the last two blocks than in the first two blocks [F(1, 53) = 10.05, p = 0.002, η<sup>p</sup> <sup>2</sup> = 0.159]. This was not true for the BO-Condition, which showed comparable

TABLE 4 | Means of the median RTs and their respective standard deviations (in brackets) by training block, condition, and trial type (regular vs. deviant) in the test phase of Experiment 2.


differences between these blocks (F < 1). Hence, it could be assumed that participants in the RO-Condition needed some time to adapt their behavior to the new structure of the material.

Because Experiment 2 has no test for the learning effects on its own and mostly relies on the interpretation of Experiment 1, the non-significant difference between the RTs of the BO- and the RO-Condition in this test phase is particularly important. Therefore, we calculated a Bayes factor with the same estimated maximum effect of the transfer test of Experiment 1 [BH(0.40 ms) = 0.30]. Though not specifically designed to test the participants' knowledge, the test phase of Experiment 2 can help to support the assumption that the BO- and the RO-Condition do not substantially differ in their acquired knowledge.

#### Test Phase: Preference Judgements

To test whether the arrangement of the trial types in the training task influenced the perceived fluency of the task, we analyzed how often the regular or the deviant mini-block was chosen as feeling more fluent. A 2 (Fluency-Judgement: regular/deviant) × 3 (Condition) Chi-Square Test showed a significant difference between the conditions [χ(2) = 7.46, p = 0.02, V = 0.09]. Most important for our hypotheses, the RO- and the BO-Condition differed significantly [χ(1) = 3.92, p = 0.04, ϕ = 0.16]. The BO-Condition rated the regular sequence as feeling more fluent more often (see **Table 5**). The RO- and AR-Condition did not differ in their choices (p = 0.45).

In addition, we also tested the relation between preference judgements and the mean RT-differences. If participants rely their judgement about the feeling of fluency on RT-differences between regular and the random mini-blocks (median RT deviant mini-block—median RT regular mini-block), we should find a positive correlation between preference judgements and RT-differences. Indeed, this correlation was positive (r = 0.22, p < 0.05).

#### Discussion

Overall, Experiment 2 confirmed our main hypothesis that participants in the BO-condition developed a stronger sensitivity

TABLE 5 | Frequencies of sequential and random mini-blocks rated as feeling more fluent in the test phase of Experiment 2.


for the fluency differences between the random and the regular material than the RO condition. However, the analysis of the median RTs of the training phase showed some unexpected patterns. Only the RO-Condition showed a trend toward a Block × Trial Type interaction, while the BO-Condition showed a significant difference between random and regular trials from Block 1 on. We have no really convincing explanation for this unexpected finding as we used, with the exception of the presented stimuli and the somewhat shorter training phase, exactly the same training phase as in Experiment 1. Since participants' knowledge was not assessed in Experiment 2, the interpretation of the reaction time data is difficult. Even though the reaction times in the RO- and BO-Conditions seem to suggest that knowledge about the sequence has been acquired, the data are not very clear cut and it cannot fully be excluded that both conditions did learn the sequence to a different extent. However, the results of Experiment 1 suggested that both conditions did not differ in the extent of learning, but differed in the expression of this learned knowledge during training. Like already suggested by a statistical trend in Experiment 1, there was a significant Condition × Trial Type interaction in Experiment 2. Therefore, replicating this performance difference between the BO- and the RO-Condition is an interesting finding. These preliminary considerations seem to be supported by the reaction times of the test task. Comparable to the transfer test in Experiment 1, the RO- and the BO-Condition showed significant and comparable differences between regular and deviant miniblocks as the material changed to a blocked arrangement for all participants. Additionally, the RO-Condition showed larger differences between regular and deviant trials in the last two blocks of the test task, which could mean that this group needed a few trials to adjust to the blocked arrangement until their performance matched the performance of the BO-Condition. The AR-Condition did not show any adjustment to the material over the four test blocks.

Most important to our hypothesis, the BO-Condition did rate the regular mini-blocks as feeling more fluent than the deviant mini-blocks. This was not the case in the RO-Condition. It seems that the training with the blocked material leads to the development of a higher sensitivity toward the differences in the experienced fluency of deviant and regular trials. Additionally, we found a correlation between the reaction time differences and the tendency to rate the regular mini-blocks as feeling more fluent. This might be a hint that the RTs provide a basis for the feeling of fluency, with faster RTs being experienced as being more fluent. However, it is also possible that the feeling of fluency relies on other cues like the feeling of a repeating movement. Alternatively, the feeling of fluency might also influence the reaction times as suggested by the observed Condition × Trial Type interactions.

## EXPERIMENT 3

Experiments 1 and 2 demonstrated that the different arrangements of the training material did not affect the extent of acquired implicit learning, but it did affect the sensitivity toward the differences in the experienced fluency between regular and deviant trials. According to the UEH these (consciously) perceivable differences in the fluency of the task are a violation of the expectancies of a participant who concurrently perceives no differences on the surface of the task. This unexpected violation of expectancies triggers an attributional process which in turn can lead to explicit knowledge of the underlying sequence. Therefore, Experiment 3 is designed to test whether the blocked arrangement of the different trial types leads to more conscious knowledge than the random arrangement. In Experiment 1, we already tested for explicit knowledge using the Wager Task. This task did not reveal any signs of explicit knowledge. We assume that the short intervals (22 trials each) make it difficult for explicit search processes to successfully result in finding the sequence as any traces of emerging explicit knowledge are rejected quickly because of the following deviant trials. Therefore, we decided to prolong the intervals of regular and deviant trials in the BO-Condition to make it easier for the participants to detect the underlying sequence.

# Method

#### Participants

Sixty-four students of the University of Cologne (39 women) participated either in fulfillment of course credit or in return for payment. Their mean age was 24.50 years (range: 18–44, SD = 5.0). All participants reported normal or corrected-to-normal vision. Thirty-two participants were in the BO-Condition and 32 participants in the RO-Condition<sup>1</sup> .

#### Procedure

The training task in Experiment 3 was identical to the training task of Experiment 1 with two exceptions. First, the mini-blocks of regular and deviant trials now contained 88 instead of 22 trials each. Second, the training task lasted eight instead of seven blocks. The stimuli again were the more complex stimuli of Experiment 1. Again, all participants received the wager task of Experiment 1 after training.

### Results

Two participants in the RO-Condition had to be excluded, due to technical problems with the computers.

For the remaining 62 participants we analyzed the individual mean error-rates. Our criterion led to the exclusion of two participants in the RO-Condition and one in the BO-Condition. For the remaining participants individual median RTs were

#### Training Phase

**Tables 6A,B** present the mean percent error rates and the means of the median RTs per condition, block, and trial type. We conducted 2 (Condition) × 8 (Block) × (Sequence Type: regular/deviant) mixed-design ANOVAs separately for the two dependent variables.

For the error rates, the ANOVA yielded a significant main effect of Block [F(7, 399) = 4.78, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.463], and of Sequence Type [F(1, 57) = 42.31, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.415]. The effect of Block indicated that participants in both conditions made fewer errors over the course of training. The main effect of Sequence Type was caused by higher error rates for deviant than for regular trials. No interaction reached the level of significance.

Concerning the median RTs the ANOVA revealed significant main effects of Block [F(7, 399) = 74.64, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.557] and of Trial Type [F(1, 57) = 21.50, p < 0.001, η<sup>p</sup> <sup>2</sup> = 0.274]. Participants responded faster over the course of the training and were generally slower on deviant trials. In addition, the Block × Trial Type interaction [F(7, 399) = 2.42, p = 0.020, η<sup>p</sup> <sup>2</sup> = 0.041] was significant, showing that the difference between regular and deviant trials increased over the course of training. Post-hoc tests suggest that the Block × Trial Type interaction was mainly caused by the BO-Condition showing a large difference between the first and the last block [F(1, 57) = 5.60, p = 0.021, η<sup>p</sup> <sup>2</sup> = 0.090], whereas the RO-Condition surprisingly did not show such a difference (F < 1). This, again could be due to the RO-Condition either gaining no knowledge about the sequence or because the arrangement of trial types made it difficult to express sequence knowledge. No other interaction reached level of significance.

#### Test Phase: Wager Task

As in Experiment 1, we first tested if the participants in the two conditions gave more correct predictions than was expected by chance (20% correct predictions). This was especially important as the participants in the RO-Condition did not show a significant learning effect during the SRT training. Both conditions showed significantly more correct predictions than expected by guessing [t(27) = 1.74, p = 0.05, one-tailed, for the RO-Condition; t(30) = 4.24, p < 0.001, one-tailed, for the BO-Condition]. Thus, the wager task revealed that both conditions acquired a significant amount of sequence knowledge. As in Experiment 1, this finding suggest that the RO-Condition did not express their acquired knowledge in the same way as the BO-Condition did in the training task.

Different to Experiment 1 but in accordance with our hypothesis, the BO-Condition gave significantly more correct

<sup>1</sup>We would like to mention that we initially only had 22 participants in each condition and later added 10 more to each condition. The reason is that we failed to find a significant learning effect in the wager task of the RO-Condition after the first 22 participants. Because the learning effect under the RO-Condition was present in Experiment 1 where we had more participants, and because a learning effect was marginally present after the first twenty-two participants, we decided to collect more data (for a debate about the pros and cons of this practice see: Simmons et al., 2011; Murayama et al., 2014).


TABLE 6A | Percent error rates and their respective standard deviations (in brackets) by training block, condition, and trial type (regular vs. deviant) in Experiment 3.

TABLE 6B | Means of the median RTs and their respective standard deviations (in brackets) by training block, condition, and trial type (regular vs. deviant) in Experiment 3.


predictions than the RO-Condition [RO-Condition: 23%, BO-Condition: 42%; t(57) = 3.21, p < 0.001]. Albeit, both conditions received the same amount of regular trials, the BO-Condition acquired much more knowledge about the sequence than the RO-Condition.

The second analysis aimed to test if this larger amount of knowledge in the BO-Condition concerns only implicit or also explicit knowledge (see **Table 7**). A 2 (Condition) × 2 (Wager Type: high|correct vs. high|false) mixed design ANOVA yielded no significant main effect for Condition, implying that no condition had a greater tendency to give high wagers (F < 1). The main effect for Wager Type was significant and showed that more high wagers were given when the prediction was correct [F(1, 57) = 5.87, p = 0.019, η<sup>p</sup> <sup>2</sup> = 0.093]. Most importantly, there was a significant Condition × Wager Type interaction [F(1, 57) = 4.61, p = 0.036, η<sup>p</sup> <sup>2</sup> = 0.075], showing that the participants in the BO-Condition were better able to use their knowledge strategically. Post-hoc tests further showed that only the participants in the BO-Condition put more high wagers on correct than on false predictions [F(1, 75) = 11.01, p = 0.002, η<sup>p</sup> <sup>2</sup> = 0.162, RO-Condition: (F < 1)]. This implies that the participants in the BO-Condition possessed more explicit knowledge than the participants in the RO-Condition.

TABLE 7 | Percent correct predictions, percent high wagers given when the prediction was correct, percent high wagers given when the prediction was false, and their respective standard deviations (in brackets) by condition in Experiment 3.


Like in Experiment 1, we wanted to exclude that the participants acquired explicit knowledge during the test task. Therefor we contrasted the difference between high|correct and high|false wagers in the first 12 wager trials with that of the last 12 wager trials for both conditions. No Condition showed a significant increase in the difference between high|correct and high|false wagers [RO-Condition: t(27) = 1.38, p = 0.178; BO-Condition: t(30) = 1.12, p = 0.271].

#### Discussion

Experiment 3 provided one important result: Prolonging the mini-blocks from 22 trials to 88 trials led to explicit knowledge in the BO-Condition, even though the number of sequence trials was not increased. This finding comes along with one weakness, however. The data of the training did not reveal any sequence learning in the RO-Condition (even after increasing the number of participants). Only the wagering task suggested that participants in this condition did possess some implicit sequence knowledge.

Analyzing the reaction times of the training task could only confirm a learning effect for the BO-Condition while the RO-Condition did not show any signs of sequence learning in their median RTs. As argued before, this could either mean that the RO-Condition did not acquire any knowledge of the training sequence or that they controlled their behavior differently than the participants in the BO-Condition. Because the participants in the RO-Condition never experience longer episodes of regular trials where they can express their knowledge fluently, they might not show their knowledge in their overt behavior and act more stimulus-dependent. This is also the only experiment in this series that could not reveal any signs of a learning effect in the reaction times of the RO-Condition, possibly hinting at a power problem. Supportive for the assumption that the participants in both conditions did acquire some sequence knowledge, the wager task showed that both groups were able to make more correct predictions than to be expected by mere guessing. This is important for further testing our hypothesis that it is the difference in the experienced fluency that leads to differences in the generation of explicit knowledge rather than the difference in the strength of associative weights acquired in the training task. Still, as Experiment 3 was designed to show differences in acquired explicit knowledge, we are not able to show that the implicit knowledge base is comparable in both conditions because if the BO-Condition indeed did acquire more explicit knowledge, this will also lead to more correct predictions in the wager task. This is essentially what we found, analyzing the wager task. Not only did the BO-condition make far more correct predictions about the next response, they also were able to use this knowledge strategically. Giving more high wagers when a prediction was correct while giving less high wagers when a prediction was false indicates that the participants not only developed first-order knowledge about the sequence but also second-order knowledge of knowing if they knew or did not know the correct prediction. Taken together, Experiment 3 showed that being trained with the blocked arrangement of the trial types leads to more explicit knowledge than being trained with the random arrangement of the trial types.

### GENERAL DISCUSSION

It was the aim of this article to test two important theories on the generation of explicit knowledge in an implicit learning situation against each other. Single-system accounts on the one hand, respectively strengthening views, assume a gradual difference between unconscious, implicit and conscious, explicit knowledge (Cleeremans, 2008, 2011; Timmermans et al., 2012). Multiplesystem theories on the other hand disagree with the idea that representational strength is a sufficient factor to explain the transition from implicit to explicit knowledge (Sun et al., 2001; Frensch et al., 2003; Scott and Dienes, 2008). Here we focused on one of these multiple-system theories, namely the Unexpected Event Hypothesis (Frensch et al., 2003; Haider and Frensch, 2005, 2009; Rünger and Frensch, 2008). The UEH assumes that implicit learning leads to changes in behavior which can be consciously perceived as unexpected changes. Once an unexpected change in one's own behavior is observed, attributional processes can lead to the generation of explicit knowledge. Various unexpected events, here the unexpected difference in the experienced subjective fluency, can serve as a trigger to start these attributional search processes. The UEH views consciousness as an absolute, dichotomous state, resulting from sudden insight rather than a gradual developing state.

To test the UEH against strengthening accounts, we aimed to create an experimental situation in which two groups should not differ in their acquired associative strength because they experienced the same amount of correct and incorrect sequence transitions during a training task. The only difference between the two groups was supposed to be in their opportunities to experience differences in their feelings of fluency due to the different arrangement of the regular and deviant trials. If a simple strengthening account was correct, both groups should show a comparable amount of explicit knowledge. If instead the assumption that it needs an unexpected event to develop explicit knowledge was correct, the group that experiences differences in their feelings of fluency should develop more explicit knowledge. In Experiment 1 we were able to show that the difference in the arrangement of regular and deviant trials led to comparable amounts of acquired knowledge, demonstrated both in a transfer and in a wager task as well. This knowledge seemed to be mainly implicit, as the wager task revealed that participants of both conditions were unable to strategically use their knowledge. With Experiment 2, we were further able to show that the differences in the arrangement of the trial types led to the predicted differences in the feeling of fluency between the groups. Together these results provided the necessary preconditions for Experiment 3 in which we aimed to show that these differences in the experienced fluency eventually lead to differences in the resulting explicit knowledge. Experiment 3 demonstrated that the participants in the BO-Condition generated more explicit knowledge than the participants in the RO-Condition.

Taken together, these results favor the UEH over a simpler strengthening account. We assume that in our experiments it is the difference of fluency between regular and deviant trials, which can only be experienced in the BO-Condition that serves as an unexpected event. Our results fit nicely with a recently published study by Yordanova et al. (2015) who found that participants who gain explicit knowledge show a greater variability in their RTs which the authors interpret as an unexpected mismatch that may serve as an offline trigger for explicit learning processes.

Nevertheless, there are several critical empirical points in our experiment that need further discussion and subsequent research. First of all, we could not assume comparable performances in the training as the reaction times differed between the BO- and the RO-Condition. The interpretation of our data would be simpler and more clear-cut if the reaction times of Experiments 1 and 2 could support the assumption that both conditions do not differ in their acquired knowledge. Instead, for all three experiments, the RT-difference between regular and deviant trials was smaller in the RO- than in the BO-Condition. In fact, Experiment 3 even failed to show a significant Block × Trial Type contrast for the RO-Condition. If these results are interpreted as showing that the RO-Condition learned less about the sequence, the interpretation of the remaining results would be ambiguous, favoring the more parsimonious strengthening account.

However, due to the design of the training, the knowledge tests in Experiment 1 and 3 seem to provide more reliable information about the extent of sequence knowledge. Different studies have shown that the performance in an SRTT can be an unreliable measure of sequence knowledge, as it can, for example, vary with different instructions for speed or accuracy (Hoyndorf and Haider, 2009), with perceived conflict (Jiménez et al., 2009), or task demands (Frensch et al., 1999). It seems plausible to assume that the differences in experienced fluency are accompanied by differences in performance during the training. It might be that in the BO-Condition, the more fluent experience on regular trials leads to a focus on speed while the less fluent deviant trials favor a focus on accuracy. Complementary, in the RO-Condition participants might show less decreased RTs on regular trials as the discontinuous structure of the task leads to an extenuated expression of the knowledge. The transfer test in Experiment 1 supports this argument as the BO- and the RO-Conditions show the same performance once the structure of the task was identical for both conditions.

Another aspect that has to be discussed is the representational structure of the implicitly learned sequences. Because the BOand the RO-Conditions differ in the arrangement of the trial types, it is possible that the representational structures differ between the two groups. It is conceivable that the participants in the BO-Condition learn something about the embedded, predictable structure of the training task (Cleeremans and McClelland, 1991; Pacton et al., 2015). Progressing in training, they might learn that there is a deterministic structure embedded after every 22 unpredictable trials and cognitively separate regular and deviant chunks. From then on, the irregular trials would no longer weaken the associative strength of the sequential weights, leading to a stronger sequential representation in the BO- then in the RO-Condition. While further experiments should be conducted to exclude this explanation, we consider it to be the less likely explanation. Research has shown that embedded random sequences can have a weakening effect on the associative weights before the system starts to learn that there is a certain predictive structure in the task and adjusts to this (Cleeremans, 1993). With six to eight blocks, our training task is most likely too short to learn about this predictive structure. Also, again, the data of our test tasks in Experiment 1 seem to speak against the assumption that the participants in the BO- and the RO-Condition have learned the sequence to a different extent.

A third objection might address the results of Experiment 3 which showed that there was more explicit knowledge when the participants were trained with a blocked arrangement. It could be argued that, due to the same strength of implicitly learned sequence knowledge, both groups did try to find a sequence, but only the BO-Condition could be successful in their search, while the RO-Condition desisted from the idea of a possible underlying sequence because the probabilistic structure made it too difficult to find it. This assumption can be supported by the literature comparing probabilistic and deterministic sequence learning. Various authors have demonstrated that a probabilistic sequence does interfere with the acquisition of explicit knowledge. When participants are informed in advance of the training that there is a certain structure or even what this structure is, they are not able to profit from this information and usually show no sign of explicit sequence knowledge (Schvaneveldt and Gomez, 1998; Stefaniak et al., 2008). However, research on probabilistic sequence learning uses this special kind of structure for suppressing explicit learning without addressing the question why it usually does not lead to explicit knowledge. It seems to be the a priori assumption that additional explicit learning processes cannot be successful under probabilistic conditions; this could have been the case in our experiments as well. Alternatively, it is possible that probabilistic learning conditions lead to weaker representations because (a) regular trials are often not equalized between deterministic and probabilistic conditions and (b) even when they are, deterministic conditions lack the weakening effect of additional deviant trials. Our BO-Condition, especially in Experiment 3, might be seen as a deterministic structure interrupted by blocks of deviant trials. Taken together with Experiment 1, our results speak against the role of representational strength for differences in the acquired explicit knowledge between probabilistic and deterministic sequences. This leaves two further explanations: First, as stated above, participants do in fact acquire enough strength to develop explicit knowledge but cannot further strengthen this knowledge because the probabilistic structure is too difficult for an explicit learning process. However, this option is not entirely compatible with a simple strengthening account, as the explicit knowledge should develop depending only on the strength of associations. An additional explicit learning process would have to be assumed that is triggered by representational strength but does not rely on it itself. The second option, in line with our hypothesis, is that the probabilistic structure hinders any metacognitive judgements about one's own behavior during the training task. Therefore, there is no situation where a triggered explicit learning process fails due to the complex structure of the task. A preliminary argument for the second option would be the lack of explicit knowledge at the end of the wager task. If participants had an idea about an underlying sequence during training it seems most likely that, as soon as the wager task starts, they remember their initial assumption and start to search for the sequence again. The data for the RO-Condition showed no signs of developing explicit knowledge during the wager task. Additionally, it would have to be explained why the participants in the RO-Condition could not discriminate between the fluency of regular and deviant mini-blocks in the test of Experiment 2, if both conditions did not differ in their experienced fluency during training. Finally, it has to be mentioned that we do not disagree with the idea that even if we told the participants in the RO-Condition that there was a sequential structure, they would not be able to find it. Our main point is that any additional explicit learning process is not triggered directly by associative strength and that, maybe even

more importantly, associative strength of the transitions does not govern explicit learning processes.

Also, the current experiments are not designed to test the finer predictions of the UEH, like, for example, the assumption that the explicit attributional processes have no direct access to the implicitly learned knowledge. However, while further experiments are certainly needed to precisely test these assumptions, the current study provides another important empirical piece in the puzzle of how explicit knowledge can develop from implicit knowledge. The results so far are difficult to reconcile with a simple explanation that relies on the strengthening of associative weights and rather appear to be in favor of the UEH. We also believe that the way we manipulated the arrangement of regular and deviant trials with the intention to keep the ratio of regular and deviant transitions balanced has not been used in implicit learning research so far, but might be an interesting methodological addition for studying present controversies in this area of research.

#### Conclusion

(1) Experiment 1 demonstrated that our method of manipulating the arrangement of regular and deviant trials does not lead to a difference in the performance in either a transfer test or a wager task. We interpret this as an indicator that the different arrangements do not lead to a difference in the acquired associative strength. (2) Experiment 2 further showed that the participants in the BO-Condition seemed to experience a difference in the fluency of random and regular trials, while the participants in the RO-Condition experienced no such difference. (3) When the intervals of regular and random trials were prolonged in Experiment 3, the BO-Condition demonstrated

REFERENCES


more explicit knowledge in a wager task. We interpret this difference being caused by the difference in experienced fluency. (4) Taken together, these results seem to be difficult to reconcile with a simple strengthening account of the emergence of explicit sequence knowledge. Rather, a metacognitive account that pronounces the role of unexpected, observable changes in one's own behavior (i.e., experienced differences in fluency) seems to be adequate for explaining the results.

#### ETHICS STATEMENT

For the participation in behavioral studies which are not expected to create stress or harm to the participants no special permission is required in Germany, so no approval from an institutional review board was necessary. All participants gave written informed consent and all procedures were performed in full accordance with German legal regulations and the ethical guidelines of the DGPs (Deutsche Gesellschaft für Psychologie; German Society for Psychology).

#### AUTHOR CONTRIBUTIONS

SE: Conception and design of the work, conducting the experiments, data analysis and interpretation, writing of the article. HH: Conception and design of the work, data analysis and interpretation, writing, and revising the article.

#### FUNDING

HH was supported by a grant from the German Research Foundation (HA-5447/10-1).


probabilistic serial reaction time task performance? Mem. Cogn. 36, 1283–1298. doi: 10.3758/MC.36.7.1283


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Esser and Haider. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Subliminal Impending Collision Increases Perceived Object Size and Enhances Pupillary Light Reflex

Lihong Chen1,2, Xiangyong Yuan<sup>1</sup> , Qian Xu1,2, Ying Wang<sup>1</sup> and Yi Jiang1,2 \*

<sup>1</sup> State Key Laboratory of Brain and Cognitive Science, CAS Center for Excellence in Brain Science and Intelligence Technology, Institute of Psychology, Chinese Academy of Sciences, Beijing, China, <sup>2</sup> University of Chinese Academy of Sciences, Beijing, China

Fast detection of ambient danger is crucial for the survival of biological entities. Previous studies have shown that threatening information can bias human visual perception and enhance physiological reactions. It remains to be delineated whether the modulation of threat on human perceptual and physiological responses can take place below awareness. To probe this issue, we adopted visual looming stimuli and created two levels of threat by varying their motion trajectories to the observers, such that the stimuli could move in a path that either collided with the observers' heads or just nearly missed. We found that when the observers could not explicitly discriminate any difference between the collision and the near-miss stimuli, the visual stimuli on the collision course appeared larger and evoked greater pupil constrictions than those on the near-miss course. Furthermore, the magnitude of size overestimation was comparable to when the impending collision was consciously perceived. Our findings suggest that threatening information can bias human visual perception and strengthen pupil constrictions independent of conscious representation of the threat, and imply the existence of the subcortical visual pathway dedicated to automatically processing threat-related signals in humans.

#### Edited by:

Morten Overgaard, Aarhus University, Denmark

#### Reviewed by:

Zhicheng Lin, Ohio State University, USA Guido Hesselmann, Charité, Germany

> \*Correspondence: Yi Jiang yijiang@psych.ac.cn

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 15 June 2016 Accepted: 21 November 2016 Published: 02 December 2016

#### Citation:

Chen L, Yuan X, Xu Q, Wang Y and Jiang Y (2016) Subliminal Impending Collision Increases Perceived Object Size and Enhances Pupillary Light Reflex. Front. Psychol. 7:1897. doi: 10.3389/fpsyg.2016.01897 Keywords: threat, looming, size perception, pupillary light reflex, awareness

# INTRODUCTION

The ability to quickly detect and properly react to potential dangers in the environment is of evolutionary significance to living organisms. One may have to judge whether a looming ball will hit him/her in order to make a proper response (e.g., to avoid collision). Threat-related signals have been found to bias human visual perception and trigger physiological responses. For example, people are inclined to overestimate the perceived proximity (Cole et al., 2013), size (Vasey et al., 2012), and duration (Tipples, 2011) of threatening objects, or to underestimate the time to contact with them (Vagnoni et al., 2012). Observers with fear of heights overestimate the perceived vertical distances and the sizes of objects when looking down from a high place (Teachman et al., 2008; Stefanucci and Proffitt, 2009). Furthermore, pictures of threatening stimuli elicit larger physiological reactions, including enhanced pupil constrictions and skin conductance responses (SCRs) in comparison with non-threatening ones (Naber et al., 2012; Lapate et al., 2014).

As a type of dynamic threatening information, visual looming stimuli have been observed to evoke the escape responses in a variety of species (Schiff et al., 1962; Wang and Frost, 1992;

de Vries and Clandinin, 2012; Yilmaz and Meister, 2013; Temizer et al., 2015), including human infants (Ball and Tronick, 1971). A growing body of evidence suggests that the threat content (e.g., impending collision) conveyed by the looming stimuli biases human visual perception and captures attention. For example, the looming stimuli predominate over receding ones during binocular rivalry (Parker and Alais, 2007), and they appear larger than the contracting ones (Whitaker et al., 1999). Moreover, the looming objects on a collision course with the observers capture attention more strongly than those on a near-miss course (Lin et al., 2008).

It remains to be delineated whether the modulation effects of looming stimuli on visual perception and physiological reactions have to rely on the conscious representation of the threatening information. Some previous studies provide clues to this issue. For instance, the attentional effect induced by impending collision can be observed without observers' explicit discrimination between the collision objects and the near-miss ones (Lin et al., 2009). It would be reasonable to postulate that the modulation of looming stimuli on visual perception and physiological reactions might be automatic and independent of whether the threat elicited by the looming stimuli is consciously perceived and explicitly discriminated. To probe this issue, we used very brief looming stimuli and created two levels of threat by varying the motion trajectories of the looming stimuli, such that the effects evoked by the two types of motion trajectories, one collided with the observers' heads and the other nearly missed, could be directly compared. Since the collision stimuli contained more threatening information than the near-miss ones, it would be expected that the former would bias visual size perception more strongly and evoke greater physiological reactions than the latter. The more critical examination was whether the perceptual and physiological effects would persist even when the observers could not consciously discriminate the collision stimuli from the near-miss ones.

# MATERIALS AND METHODS

# Participants

A total of 44 undergraduates and graduates (22 male) with the mean age of 22.25 (ranging from 19 to 26) received financial compensation for their participation. All the information about participants, stimuli, and tasks was summarized in **Table 1**. In Experiment 1, 10 observers (5 male) participated in the size perception and motion trajectory discrimination tasks. In Experiment 2, 12 observers (6 male) performed the size perception and motion trajectory discrimination tasks. In Experiment 3, another 8 observers (4 male) participated in the two-alternative forced choice (2AFC) tasks on motion trajectory and speed discrimination. In Experiment 4, a new group of 8 observers (4 male) participated in the distance discrimination and the corresponding 2AFC tasks (one of them also participated in Experiment 1). Seven observers (3 male) participated in Experiment 5, which includes size perception, motion trajectory discrimination and location discrimination tasks (with their pupil sizes recorded). All had normal or corrected-to-normal vision and gave written, informed consent in accordance with procedures and protocols approved by the institutional review board of the Institute of Psychology, Chinese Academy of Sciences. All observers were naive to the purpose of the experiments.

# Apparatus, Stimuli, and Procedure

Subjects viewed a CRT monitor binocularly from a distance of 50 cm. A chin rest was used to stabilize their head position. The experimental room had no illumination other than the display screen. Displays were generated using Matlab (Mathworks) together with the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997).

The looming sphere simulated a motion from a distance of 350 cm to 175 cm to the observers (**Figure 1A**), and during this period it expanded uniformly from a small size (1.32◦ ) to the standard size (2.64◦ ; **Figure 1B**). In Experiment 1, the motion duration was 133 ms. The collision sphere had an initial position 7.57 cm (8.68◦ ) from the monitor center and simulated a final impact point 3 cm from the center of the observers' heads. Similarly, the near-miss sphere had an initial position 7.14 cm (8.18◦ ) from the monitor center and simulated a final impact point 6 cm from the center of the observers' heads. At the end of the looming motion, both the collision and the near-miss spheres had identical end position 8 cm (9.17◦ ) from the monitor center. In Experiments 2–5, two parameters were altered to render the trajectory difference indistinguishable. Firstly, the motion duration was reduced to 67 ms. Secondly, the initial position of the collision sphere was 7.43 cm (8.51◦ ) from the monitor center, simulating a point of impact 4 cm from the center of the observers' heads.

In the size perception task, the sphere stayed on the screen for an extra 200 ms after it reached its maximum size, and then followed by a comparative circle presented at the center of


the screen. The observers were asked to adjust the size of the comparative circle to match the final size of the sphere presented before. The perceived size of the sphere was calculated relative to its standard size. The initial size of the comparative circle varied from trial to trial (2.40◦–2.87◦ ) with a step of 0.068◦ . In the motion trajectory discrimination task, one collision or near-miss sphere was presented in each trial. The observers were asked to judge whether the sphere collided with their heads or just nearly missed (subjective judgment task). In 2AFC tasks, two spheres were presented successively, one on the collision path, and the other on the near-miss path. The observers had to judge whose trajectory collided with their heads, the first one or the second one in the 2AFC motion trajectory discrimination task, and to judge which one was faster or nearer to them in the 2AFC speed discrimination and the 2AFC distance discrimination tasks, respectively. In another task of distance discrimination, the observers had to judge whether the successively presented spheres had the same or different distances from them.

### Eye Tracking

We only recorded the observers' pupil sizes in Experiment 5 where the observers were asked to perform the location discrimination task. To exclude the influence of the physical difference of motion trajectories, we included another two conditions in which the spheres moved in the same trajectories as in the collision and near-miss conditions, but from the end point to the starting point, i.e., in a receding manner. Because of the extreme sensitivity of the pupil to the light, we replaced the gray background with a black one in Experiment 5. In each trial, after a variable duration (1500–1900 ms) with a white fixation cross presented at the center of the screen, a looming or receding sphere was presented to the left or right side of the fixation cross with a duration of 67 ms, following by a 4-s blank interval. The observers were asked to judge the location of the sphere relative to the fixation cross using the left and right arrow keys. To maintain an accurate measure of the pupil size, the observers were required to keep their eyes on the fixation cross and to refrain from blinking throughout the trial.

Pupil diameter and two-dimensional eye position of the left eye were measured with a video-based iView X Hi-Speed system (SMI, Berlin, Germany). A standard 5-point calibration was performed at the beginning of each session. Eye tracking data were acquired at 500 Hz. Trials with unrealistic pupil size (<0.3 mm or >1.3 mm) and with the pupil size out of ±3 SDs were excluded from further analysis. There were 20.25% trials on average being excluded in the eye tracking experiment. The data were resampled in time bins of 40 ms each. For each subject and each condition, pupil data were averaged across trials after subtracting the mean pupil diameter in the 500 ms preceding stimulus onset. Horizontal eye position data were analyzed in the same way as the pupil diameter.

There were 80 trials in each of the 2AFC trajectory discrimination task, 2AFC speed discrimination task and the two distance discrimination tasks. There were 32 trials per condition in the remaining tasks.

# RESULTS

fpsyg-07-01897 November 30, 2016 Time: 12:40 # 4

# Threatening Information Increases Perceived Object Size Independent of Conscious Perception

In Experiment 1, the observers could well discriminate the collision sphere from the near-miss one (judgment accuracy = 62.1% ± 8.0%, 95% Confidence Interval (CI) = [56.3%, 67.8%], t(9) = 4.757, p = 0.001, d = 1.504, JZS-BF [alternative/null] = 44.553, mean sensitivity d <sup>0</sup> = 0.683 ± 0.532). JZS-BF is Jeffrey-Zellner-Siow Bayes factor with Cauchy distribution on effect size and is the probability of the data under one hypothesis relative to that under another hypothesis (Rouder et al., 2009; Vadillo et al., 2016). The odds of alternative versus null hypothesis were greater than 44:1 favoring the alternative hypothesis. The collision sphere was perceived to be significantly larger than the near-miss one (3.682% vs. 2.941%, F(1,9) = 7.688, p = 0.022, η 2 <sup>p</sup> = 0.461; **Figure 2A**). In Experiment 2, the observers could not discriminate the trajectory difference any more (50.5% ± 5.4%, 95% CI = [47.1% 54.0%], JZS-BF [null/alternative] = 4.421, mean sensitivity d <sup>0</sup> = 0.036 ± 0.339). The odds of null versus alternative hypothesis were greater than 4:1 favoring the null hypothesis. Remarkably, the collision sphere still appeared larger than the near-miss one (3.409% vs. 2.928%, F(1,11) = 30.858, p < 0.000, η 2 <sup>p</sup> = 0.737; **Figure 2B**). In Experiment 3, to further confirm the results of Experiment 2, we asked another group of the observers to perform a 2AFC motion trajectory discrimination task, and found that the observers could not discriminate the trajectory difference (53.8% ± 8.4%, 95% CI = [46.7%, 60.8%], JZS-BF [null/alternative] = 1.987). Moreover, the discrepancy of perceived size in Experiment 2 could not be explained by the differences in speed perception between the two conditions (52.0% ± 7.7%, 95% CI = [45.6%, 58.5%], JZS-BF [null/alternative] = 3.033,). In Experiment 4, we found that distance perception could not account for the discrepancy of perceived size between the two conditions in Experiment 2 (2AFC performance, 52.2% ± 6.5%, 95% CI = [46.8% 57.6%], JZS-BF [null/alternative] = 2.579; samedifference judgment, 50.2% ± 8.7%, 95% CI = [42.9%, 57.4%], JZS-BF [null/alternative] = 3.910). More interestingly, the perceived size discrepancy of the collision and near-miss spheres in Experiment 1 and Experiment 2 was not different from each other [F(1,20) = 0.997, p = 0.330, η 2 <sup>p</sup> = 0.047, **Figure 2C**], which is in contrast to the observations that the observers' motion trajectory discrimination accuracies in Experiment 1 was significantly higher than those in Experiment 2 [F(1,20) = 16.099, p = 0.001, η 2 <sup>p</sup> = 0.446]. Moreover, the

observers' size overestimation effects were not correlated with their motion trajectory discrimination accuracies [r(22) = 0.248, p = 0.265]. These results together suggest that the modulation effect of threat on perceived size is automatic and independent of conscious perception.

# Subliminal Threat Enhances Pupil Constriction

In Experiment 5, we first replicated the behavioral finding that the observers could not discriminate the motion trajectory difference (53.6% ± 10.6%, 95% CI = [43.8% 63.4%], JZS-BF [null/alternative] = 2.593, mean sensitivity d <sup>0</sup> = 0.296 ± 0.756). The odds of null versus alternative hypothesis were around 3:1 favoring the null hypothesis. The collision sphere was perceived as significantly larger than the near-miss one (6.078% vs. 5.792%, F(1,6) = 11.228, p = 0.015, η 2 <sup>p</sup> = 0.652; **Figure 2D**).

The pupil diameters of the observers began to shrink around 300 ms after the onset of the sphere. We found a marginally significant interaction of presentation mode (looming and receding) and motion trajectory (collision and near-miss) in the minimum vertex of pupil diameters [F(1,6) = 5.601, p = 0.056, η 2 <sup>p</sup> = 0.483]. Further analysis showed that the collision sphere evoked significantly larger pupil constrictions than the nearmiss one in the looming manner (t(6) = –3.309, p = 0.016, d = –1.251, JZS-BF [alternative/null] = 4.769, **Figure 3A**). However, there was no such difference when the spheres moved in the receding manner (t(6) = –0.230, p = 0.826, JZS-BF [null/alternative] = 3.613, **Figure 3B**), even though the receding spheres had the identical motion trajectories as the looming spheres. This pattern of results persisted from 340 ms to 820 ms after the onset of the sphere, as evidenced by a significant interaction between presentation mode (looming and receding) and motion trajectory (collision and near-miss), F(1,6) = 5.991, p = 0.05, η 2 <sup>p</sup> = 0.50. Consistently, the collision sphere elicited significantly larger pupil constrictions than the near-miss one in the looming manner (t(6) = –3.242, p = 0.018, d = –1.225, JZS-BF [alternative/null] = 4.446) but not in the receding manner (t(6) = –0.238, p = 0.820, JZS-BF [null/alternative] = 3.606). Differences in the pupil diameters between the collision and nearmiss spheres were also tested for statistical significance through one-tailed non-parametric permutation test based on the t-max statistic (Blair and Karniski, 1993; Groppe et al., 2011). We performed the test in the time window of 0 ms to 2000 ms after the onset of the sphere (5000 permutations, p = 0.05). The results showed that the collision sphere elicited significantly larger pupil constrictions than the near-miss one from 570 ms to 1460 ms after the onset of the sphere in the looming manner (**Figure 3C**), but not in the receding manner (**Figure 3D**). Furthermore, the difference of pupil diameters between the collision and the

FIGURE 3 | Time courses of pupil diameters in the looming (A) and receding conditions (B) and their corresponding horizontal eye gaze positions (E,F). Corrected p-value of permutation test of pupil diameter discrepancy between the collision and the near-miss spheres in the two conditions (C,D), respectively. The gray area indicates the time duration ranging from 340 ms to 820 ms after the onset of the sphere. <sup>∗</sup>p < 0.05. The dotted lines represent one standard error of the mean.

near-miss spheres in the looming manner was not caused by horizontal eye gaze difference (t(6) = 0.248, p = 0.812, JZS-BF [null/alternative] = 3.598; **Figure 3E**; see **Figure 3F** for the receding manner).

# DISCUSSION

fpsyg-07-01897 November 30, 2016 Time: 12:40 # 6

Previous studies have demonstrated that consciously perceived threatening information can bias human visual perception in both spatial and temporal domains. To probe the modulation of subliminal threat on visual size perception and pupillary light reflex, we used visual looming stimuli and created two levels of threat by varying their motion trajectories in three-dimensional geometric space, such that it could either collide with the observers' heads or just nearly miss. We found that, regardless of whether or not the observers were aware of the differences between the collision and the near-miss spheres, the former was perceived as significantly larger than the latter, and the effect of size overestimation was comparable between the threat-perceived and the threatunperceived conditions. Moreover, the subliminal threat induced by impending collision elicited greater pupil constrictions. These results provide evidence that visual looming stimuli can modulate visual perception and physiological reactions in an automatic fashion and independent of whether or not the threat elicited by impending collision of looming stimuli is consciously perceived.

Converging evidence has shown that size perception can be biased by threatening information. For example, circles with a negative picture are estimated to be larger than circles with a positive or a neutral picture (van Ulzen et al., 2008). Moreover, observers with fear of heights tend to overestimate object size when looking down from a high place (Stefanucci and Proffitt, 2009), and those with spider phobia often overestimate the sizes of spiders (Vasey et al., 2012; Shiban et al., 2016). Visual looming stimulus that implies impending collision looks larger than a receding one (Whitaker et al., 1999). However, all these effects are observed when the observers are fully aware of the potential threat. The current study provides further evidence and shows that even when the threatening information is not consciously identified, it can still bias object size perception, and the amplitude of such misperception is comparable to the threat-perceived condition. Previous studies have demonstrated that threatening information, such as a visual stimulus paired with white noise (Koster et al., 2004) or a looming stimulus on a collision path (Lin et al., 2008, 2009), can automatically attract attention, and allocation of spatial attention onto a visual stimulus can affect its perceived size (Anton-Erxleben et al., 2007). Therefore, it seems possible that the effects of threat on size perception and attentional capture work in tandem to enhance the salience of the threatening information and guide the observers to make further actions.

Although the current study does not provide direct evidence on the possible mechanisms underlying the observed effects, it is postulated that a specialized subcortical visual pathway (through the superior colliculus and the pulvinar to the amygdala) is dedicated to detecting threat-related signals outside of conscious awareness (Hedger et al., 2015). This idea has been supported by converging evidence showing that subliminal threat-related visual stimuli elicit enhanced neural and physiological responses in comparison with non-threat stimuli. For example, fearful faces and spider images have been shown to enhance the amygdala activity in the absence of visual awareness across a variety of masking techniques (Whalen et al., 1998; Pasley et al., 2004; Williams et al., 2004; Jiang and He, 2006; Lipka et al., 2011), likely through the purported subcortical pathway (Tamietto and De Gelder, 2010). Visual looming stimuli are often associated with imminent collision and considered as potent perceptual indicators of threat. By using the visual looming stimuli that simulated impending collision with observers' heads to elicit threat, the current study resonates well with and extends previous findings by showing that the threatening information can be processed independent of conscious awareness, and thus suggests the existence of the subcortical visual pathway dedicated to processing threat-related signals independent of conscious perception in humans.

It is widely accepted that adjustments of pupil size occur automatically in response to variations in ambient light to optimize retinal illumination for visual perception. However, recently a growing body of evidence has shown that the pupil not only constricts to physically bright object, but also to bright illusions (Laeng and Endestad, 2012), and even to imaginary bright situations (Laeng and Sulutvedt, 2014). Moreover, pupil constrictions as well respond to a variety of visual attributes besides brightness, including the inversion effect (Conway et al., 2008; Naber and Nakayama, 2013), novelty (Naber et al., 2013), conspecific faces (Conway et al., 2008; Ebitz et al., 2014), threatening animals (Naber et al., 2012), and the sun (Binda et al., 2013; Naber and Nakayama, 2013). The current study extends the scope of stimulus properties the pupil responds to, and demonstrates that it also constricts to subliminal threatening content conveyed by impending collision. This response pattern should have adaptive value for survival, because it suggests that the brain already processes the threatening signals even before we consciously realize them, which might enable humans to act fast in fight or flight situations. More importantly, the combined evidence from previous studies and the current one suggests that pupil constrictions are not merely reflexive responses to retinal illumination, but are also mediated by the automatic visual processing of salient events in the environment. It has been suggested that the pupillary light responses are largely mediated by subcortical non-image forming system (Binda and Murray, 2015), and the retinocollicular pathway directly connecting the retina to the superior colliculus and the pulvinar is engaged in the eye movements in the absence of awareness (Spering and Carrasco, 2015), which is partially overlapped with that underlying subconscious processing of threat-related signals.

This also raises the possibility that the size overestimation and the pupil constrictions in response to the subconscious impending collision are two sides of the same coin, and their functional relationship merits further investigation.

In summary, our results demonstrate that when the observers could not discriminate any difference between the collision and the near-miss spheres, the sphere on the collision course looked larger and elicited greater pupil constrictions than that on the near-miss course. Moreover, the magnitude of size overestimation was comparable to that in the threat-perceived condition. Our findings highlight the functional dissociation between the pupillary response and the conscious perception of threat, and imply the existence of the subcortical threat-related visual pathway in the human brain.

# REFERENCES


# AUTHOR CONTRIBUTIONS

LC and YJ conceived and designed the study. LC conducted the research and analyzed the data. LC wrote the manuscript, and XY, QX, YW, and YJ provided critical revisions.

# ACKNOWLEDGMENTS

This work was supported by grants from the Strategic Priority Research Program of the Chinese Academy of Sciences (No. XDB02010003) and National Natural Science Foundation of China (No. 31100733, No. 31525011). We are grateful to two reviewers who provided helpful comments to improve this article.



**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Chen, Yuan, Xu, Wang and Jiang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Commentary: Definitely maybe: can unconscious processes perform the same functions as conscious processes?

Ariel Goldstein<sup>1</sup> and Ran R. Hassin2, 3 \*

<sup>1</sup> Cognitive Science Department, Hebrew University, Jerusalem, Israel, <sup>2</sup> Psychology Department, Hebrew University, Jerusalem, Israel, <sup>3</sup> Cognitive Science Department, Center for the Study of Rationality, Hebrew University, Jerusalem, Israel

Keywords: nonconscious, high level cognition, CFS, implicit, subliminal

#### **A commentary on**

#### **Definitely maybe: can unconscious processes perform the same functions as conscious processes?**

by Hesselmann, G., and Moors, P. (2015). Front. Psychol. 6:584. doi: 10.3389/fpsyg.2015.00584

Through critically examining Hassin's paper "Yes It Can" (YIC; 2013) Hesselmann and Moors (H&M; 2015) suggest that existing data support a more moderate, skeptical view than that suggested by Hassin. The thing we like the most about H&M's paper is the view it proposes: "Definitely Maybe." To the best of our understanding, the view that H&M suggest to those of us interested in non-conscious high-level cognitive processes—open minded skepticism—is definitely an improvement on the more traditional "No It Can't" (e.g., Newell and Shanks, 2014). Definitely Maybe leaves a door open, conceptually and empirically.

#### Edited by:

Morten Overgaard, Aarhus University, Denmark

#### Reviewed by:

Jan Brascamp, Michigan State University, United States Guido Hesselmann, Charité Universitätsmedizin Berlin, Germany

> \*Correspondence: Ran R. Hassin ran.hassin@huji.ac.il

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 08 December 2016 Accepted: 05 July 2017 Published: 04 August 2017

#### Citation:

Goldstein A and Hassin RR (2017) Commentary: Definitely maybe: can unconscious processes perform the same functions as conscious processes? Front. Psychol. 8:1230. doi: 10.3389/fpsyg.2017.01230

Frontiers in Psychology | www.frontiersin.org August 2017 | Volume 8 | Article 1230 |

The argument Hassin made in YIC is simple. He suggested that the capacity limitations of our conscious processes, as well as evolutionary considerations, make it reasonable to suspect that non-conscious processes can carry out every fundamental high-level function that conscious processes can perform. He then went on to review the literature on a number of functions that were traditionally associated with consciousness, and showed that there are already data to suggest that they can occur non-consciously (see also Hassin and Sklar, 2014). The functions that Hassin reviewed in YIC were cognitive control and executive functions, pursuing goals, and information broadcasting and reasoning. Each section reviewed multiple papers that were representative, but far from exhaustive. The function of the literature review was simple: "to illustrate [emphasis added] YIC in fundamental, high-level cognitive functions" (p. 196). The review rendered the argument for YIC more plausible simply by pointing out that many data exist to support it.

H&M offer three main criticisms of YIC that have to do with the supporting evidence reviewed in the paper. We thank them very much for making these comments, and for the way they make them. We believe science is a social endeavor that advances by exchanges of this sort. We will address each of their points below.

#### H&M'S REVIEW

In their first section H&M argue that Hassin's review of the literature was selective, and many of their points are well taken. Note, however, that much of the data that was left out of YIC was supportive: there is so much additional data out there about non-conscious high-level cognitive and motivational functions that Hassin had to limit himself in certain ways (see Bargh and Morsella, 2008; Bargh et al., 2012; for other relevant reviews and overviews see Kouider and Dehaene, 2007; Van den Bussche et al., 2009 as well as Kahneman, 2011; Dehaene, 2014).

Ironically, yet naturally, H&M's review of YIC is... selective too. In their first section, they raise concerns about a small subset of the findings Hassin reviewed. They ignore most of the papers reviewed in YIC, in most of its sections. Just to Illustrate: none of

the investigations of non-conscious cognitive control were examined or questioned (for a review of more of those see Hassin and Sklar, 2014). In the reasoning section, they selectively focus on issues of replicability and limitations of Unconscious Thought Theory (Dijksterhuis, 2006; Dijksterhuis and Nordgren, 2008; but see also the meta-analysis in Strick et al., 2011; Nieuwenstein et al., 2015), and flag priming in the U.S. (but see Ferguson et al., 2013), but do not discuss other sections such as the vast literatures on inferences, insights, and others.

Are there question marks about some of the findings, including our own? Yes, and we thank H&M for pointing out those that were erroneously left out of the original paper and those that have become public since. Will there be more question marks in the future? Definitely. After all, this is how science progresses and corrects itself. But are the challenges so far detrimental to YIC? We do not think so. In fact, as questions were being raised about a small subset of the findings in the original review, many new findings were published too.

To take a few examples, the idea that humans can nonconsciously read multiword expressions (Sklar et al., 2012) was supported by new findings and different paradigms (Armstrong and Dienes, 2013, 2014; van Gaal et al., 2014; Axelrod et al., 2015; Saxe personal communication). In these papers, multipleword expressions were presented subliminally, and behavioral and neurological evidence supported the claim that participants were able to read and understand them. New findings extend the previous literature on non-conscious executive function and implicit effects of working memory (Gayet et al., 2013; Dutta et al., 2014), as well as implicit emotion regulation (Wang and Li, 2017). Non-conscious information integration, yet another function that was once perceived as requiring consciousness, was demonstrated in new ways (Alsius and Munhall, 2013; Faivre et al., 2014; Mudrik et al., 2014; Fahrenfort et al., 2017; Hung et al., 2017; but see Moors et al., 2016); various laboratories have shown that information can be non-consciously integrated across time by demonstrating motion perception with subliminal stimuli (Kaunitz et al., 2011; Kimura et al., 2012; Faivre and Koch, 2014a,b; Salomon et al., 2016; Moors et al., 2017); it has been shown that narratives can be extracted from subliminal stimuli (Kawakami and Yoshida, 2015); and, relatedly, Moors et al. (2017) have shown that causality is inferred and used nonconsciously too (non-casual events become conscious after causal events). Lastly, priming bankers with their banker's identity leads to increased dishonesty (Cohn et al., 2014).

Again, this list is very selective, and does not begin to cover all relevant findings since the publication of YIC. Taken together, these findings, from various laboratories and using different paradigms, significantly challenge the modal view of unconscious processes vis-à-vis reading, and support the main claim of YIC.

### PRIMING IN SOCIAL AND COGNITIVE PSYCHOLOGY

As Hassin pointed out in YIC (see also Doyen et al., 2014), the processes that our minds can do with stimuli that we do not consciously perceive are a (small?) subset of the processes that our minds can do without awareness. This is the distinction between the downstream effects of subliminal perception and the effects of unconscious cognition (Bargh and Morsella, 2008). Anyone interested in the cognitive unconscious—or in human consciousness—should be interested in both types of processes.

It should not come as a surprise, then, that we do not agree with H&M's argument, according to which the shift from a focus on unconscious perception (i.e., subliminal perception) to unawareness of processes and their effects (i.e., unconscious cognition) is "problematic" (p. 2). It is anything but, and it is a necessary step if one wants to understand the contribution of unconscious processes to human existence. Studying unconscious processes solely through subliminal perception is akin to studying gender differences solely through differences in physical appearance, or studying human memory solely by examining memorization of lists. Yes, there are differences in appearances, and important insights to be gained from studying list memorization, but there is much more to gender differences and memory than that.

H&M go on to contrast the views of Doyen et al. (2014) and Hassin vis–à-vis the differences between the two traditions of studying the unconscious. We wish to note that the views are not mutually exclusive. While some of the differences are likely to be the result of the factors identified by Doyen et al., others probably have to do with motivation, practice, and ability, the factors identified by Hassin. Traditional views of automaticity confounded unconsciousness with involuntariness. Bargh (1994) suggested that these characteristics do not necessarily go hand in hand. We take this argument one step further and suggest that unconscious processes are more likely to happen with stimuli and ideas we care deeply about than with those we care less about (Hassin, 2013). The reader is invited to think about the number of times she had new insights and surprising thoughts about issues she didn't care about vs. those she really cared about.

H&M suggest that in social psychology, where some of the major advances in understanding the human unconscious have occurred in recent decades, "absence of awareness is often assumed rather than tested, and when tests are conducted, they are below the standards widely used in cognitive psychology" (p. 2). We tend to agree (noting, of course, that the qualifier OFTEN is crucial), and we definitely join H&M's call for the adoption of ever more sophisticated ways of measuring and assessing awareness—its stimuli, effects, and processes. We wish to highlight two points here, though. One, that the standards in cognitive psychology are less clear than one may want them to be. This literature is marred with disagreements.

Secondly, it is important to highlight that in both cognitive and social psychology, the role of consciousness is often assumed, and hardly ever put for to a rigorous test (Hassin and Milyavsky, 2014). In fact, even if data convincingly show that a function F cannot happen non-consciously, these data are mute vis. a vis. causality, a point that seems to be overlooked by many. More specifically, assume that we become convinced that a function F is not executed when the process is non-conscious. This is a correlational observation, and it doesn't mean that consciousness has a causal role in allowing F. There might be a factor that brings about consciousness and allows F to happen; or there might be a factor that tends to co-occur with consciousness, that allows F to happen. When we look back at the literature, for example, it is

quite clear that for many years we confounded prime duration with consciousness. Subliminal stimuli had to be very short. As CFS experiments have shown, longer duration subliminal stimuli seem to have more effects than shorter duration subliminal stimuli (but see Barbot and Kouider, 2012), so some functions we attributed to consciousness should be attributed to duration.

#### ON OPTIMISM AND CFS

CFS is a relatively new methodology, and as such there are many welcomed debates surrounding it. H&M discuss papers that suggest limitations in non-conscious processing under CFS, and how to measuring measure awareness under CFS is also a question in active debate (Gelbard-Sagiv et al., 2016). In their third section H&M build on these findings to suggest that enthusiasm about CFS as a tool for unconscious research is "premature and farfetched." It is definitely a possibility. CFS is young (only 10 years old; Tsuchiya and Koch, 2005), and only future data will tell.

But there are two reasons for cautious optimism. The format of this reply is short, but let us just note that even if it turns out that there is partial low-level awareness in CFS that does not mean that the process under consideration is conscious. Knowing that a word is written in blue (vs. red) ink, doesn't mean that you can consciously read the word, which is the question we should focus on when we examine reading (for example). Secondly, CFS has been used to show non-conscious semantic integration (Mudrik et al., 2011, 2014; but see Moors et al., 2016), nonconscious working memory (Pan et al., 2012; Gayet et al., 2013), non-conscious multisensory integration (Alsius and Munhall, 2013; Faivre et al., 2014), non-conscious emotion recognition (Capitão et al., 2014), non-conscious reading (Costello et al., 2009; Yang and Yeh, 2011; Zabelina et al., 2013), non-conscious emotion perception (Yang et al., 2007; Almeida et al., 2013; Troiani and Schultz, 2013) and non-conscious expectations (Stein and Peelen, 2015), amongst other functions. For such a young technique, these results are very encouraging.

#### BETWEEN MAYBE AND YES

Why do lay people and scientists alike have the strong intuition that there simply must be cognitive functions that are made possible only by the kind of consciousness we have? We don't know, but we suspect it has to do with the search for the one thing that makes us, human beings, special, so that we stand out from

# REFERENCES


the animal kingdom. It has to do with the desire, so beautifully identified by Dan Gilbert, "to publish a book, a chapter, or at least an article that contains this sentence: The human being is the only animal that...." (Stumbling on Happiness, p. 3) (Gilbert, 2006). And indeed, there is a correlation that is hard to miss: we (think we) are very special in the animal world and we (think we) have consciousness that no other animal has. Hence, goes the argument, there must be functions that consciousness allows for, and that make us so special.

This intuition is misleading in at least two respects. First, it is unclear whether the argument's assumptions are true. Second, and more importantly, it is logically flawed. Even if we are very special in the ways we think we are, and even if we have consciousness that no other animal has, this does not mean that the latter causes the former. Ulric Neisser, for example, had a radically different suggestion, one to which we are very sympathetic: "our hypothesis thus leads us to the radical suggestion that the critical difference between the thinking of humans and of lower animals lies not in the existence of consciousness but in the capacity for complex processes outside it" (Neisser, 1963, p. 10).

One of the implications of YIC is that the search for a holy grail—the function that only consciousness can do is the wrong way to go. Understanding what it means to be human would be better achieved by understanding the similarities and differences between conscious and unconscious pursuit of the same functions, how conscious and unconscious functions work together and orchestrate human cognition, and how unconscious processes pursue functions that only they can pursue.

# SUMMARY

We thank H&M for drawing our attention to shortcomings of our previous paper, and for the opportunity to exchange ideas, and we look forward to discovering more about consciousness and the human unconscious. While Definitely Maybe is definitely an improvement on the more traditional No It Can't, we still believe that Yes It Can is a more plausible alternative that directs our science to a fruitful agenda.

# AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.


Gilbert, D. (2006). Stumbling on Happiness. New York, NY: Knopf.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Goldstein and Hassin. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Not Merely Experiential: Unconscious Thought Can Be Rational

#### Katie E. Garrison<sup>1</sup> \* and Ian M. Handley<sup>2</sup>

<sup>1</sup> Department of Psychology, Texas A&M University, College Station, TX, United States, <sup>2</sup> Department of Psychology, Montana State University, Bozeman, MT, United States

Individuals often form more reasonable judgments from complex information after a period of distraction vs. deliberation. This phenomenon has been attributed to sophisticated unconscious thought during the distraction period that integrates and organizes the information (Unconscious Thought Theory; Dijksterhuis and Nordgren, 2006). Yet, other research suggests that experiential processes are strengthened during the distraction (relative to deliberation) period, accounting for the judgment and decision benefit. We tested between these possibilities, hypothesizing that unconscious thought is distinct from experiential processes, and independently contributes to judgments and decisions during a distraction period. Using an established paradigm, Experiment 1 (N = 319) randomly induced participants into an experiential or rational mindset, after which participants received complex information describing three roommates to then consider consciously (i.e., deliberation) or unconsciously (i.e., distraction). Results revealed superior roommate judgments (but not choices) following distraction vs. deliberation, consistent with Unconscious Thought Theory. Mindset did not have an influence on roommate judgments. However, planned tests revealed a significant advantage of distraction only within the rational-mindset condition, which is contrary to the idea that experiential processing alone facilitates complex decision-making during periods of distraction. In a second experiment (N = 136), we tested whether effects of unconscious thought manifest for a complex analytical reasoning task for which experiential processing would offer no advantage. As predicted, participants in an unconscious thought condition outperformed participants in a control condition, suggesting that unconscious thought can be analytical. In sum, the current results support the existence of unconscious thinking processes that are distinct from experiential processes, and can be rational. Thus, the experiential vs. rational nature of a process might not cleanly delineate conscious and unconscious thought.

Keywords: unconscious thought, rational and experiential systems, consciousness, problem solving, judgment and decision making

# INTRODUCTION

People can rely on a variety of processes to guide their decisions and judgments regarding complex information. For example, one might think carefully and rationally, or alternatively "feel out" information to arrive at a conclusion. Indeed, many dual-process models in psychology propose that judgments and decisions can predominantly result from either analytical or

#### Edited by:

Morten Overgaard, Aarhus University, Denmark

#### Reviewed by:

Elisabeth Norman, University of Bergen, Norway Glenn Carruthers, Charles Sturt University, Australia

> \*Correspondence: Katie E. Garrison k\_garrison@tamu.edu

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 30 December 2016 Accepted: 13 June 2017 Published: 06 July 2017

#### Citation:

Garrison KE and Handley IM (2017) Not Merely Experiential: Unconscious Thought Can Be Rational. Front. Psychol. 8:1096. doi: 10.3389/fpsyg.2017.01096

experiential/emotional processing systems (e.g., Epstein, 1994; Sloman, 1996; Kahneman, 2003; see Evans, 2008 for review). Of current focus, Cognitive Experiential Self Theory (CEST; Epstein, 1994, 1998) describes two parallel informationprocessing systems – an experiential system and a rational system – that contribute to judgments and decisions. These two systems are global aspects of personality, and within each system are psychological processes. Within the experiential system are automatic, fast, intuitive, holistic, pre-conscious and emotional processes. Within the rational system are effortful, deliberate, slow, rational and conscious processes. Furthermore, a relative reliance on one system over the other is a trait-like individual difference (Epstein et al., 1992) and can be manipulated in the laboratory (e.g., Krauss et al., 2004). For instance, an individual with a relatively dominant experiential system may make judgments and decisions based on a "vibe" or hunch, paying close attention to his or her emotions, whereas an individual with a more dominant rational system may follow rules and careful analysis to reach a decision.

The general assumption underlying CEST, and many other dual-process models, is that the rational system operates on a conscious level, whereas the experiential system operates more unconsciously (Evans, 2008). Researchers commonly use factors like the speed, or the evident rational vs. emotional basis of a decision as clues that conscious or unconscious processes produced the response. In this paper, we challenge the implicit or explicit assumptions undergirding this practice, and hope to refine the attributes distinguishing conscious and unconscious processes.

To this end, we offer that unconscious thought, as proposed by Unconscious Thought Theory (UTT; Dijksterhuis and Nordgren, 2006; Dijksterhuis and Strick, 2016) resembles both experiential and rational modes of processing, and therefore does not fit neatly within a two-system framework. As described in more detail below, unconscious thought operates outside of conscious awareness, but does so over a period of time (i.e., not quickly and automatically), and in a goal-directed manner. These features of unconscious thought resemble the rational mode, and we therefore propose that unconscious thought may be independent of the two modes of processing. The current research presents two experiments testing the hypothesis that unconscious thought is distinct from purely experiential processes and can be rational.

# THE MIS-FIT OF UNCONSCIOUS THOUGHT

Much research and theorizing support an unconscious mode of processing that does not fit neatly within a two-system framework. In particular, UTT asserts that unconscious processes can integrate, organize, and evaluate complex information (i.e., much and varied information) over several minutes to facilitate sound judgments and decisions (Dijksterhuis and Nordgren, 2006; Dijksterhuis and Strick, 2016). Dijksterhuis and Nordgren (2006, p. 96) call this process unconscious thought, defining it as "object-relevant or task-relevant cognitive or affective thought processes that occur while conscious attention is directed elsewhere." Unconscious thought is presumed to be goaldirected, occur over a period of time, and facilitate decisionmaking by actively organizing and processing information (Bos et al., 2008). It is akin to incubation (e.g., Smith and Blankenship, 1991) or "mulling over" information below the threshold of conscious awareness.

In a typical UTT experiment (e.g., Dijksterhuis, 2004), participants receive complex information about various options (e.g., apartments), are asked to form impressions from the information and ultimately evaluate the options or choose the best one. Commonly, there are 4 options associated with 12 attributes each, some positive and some negative, totaling to 48 pieces of information. Participants view each piece of information randomly and one at a time (e.g., "Apartment A is in a nice area," "Apartment B has an unfriendly landlord," etc.). The information is rigged such that one option has more positive attributes and fewer negative attributes than the others (e.g., eight positive and four negative), one has more negative attributes and fewer positive attributes than the others (e.g., eight negative and four positive), and two have an equal number of positive and negative attributes (six positive and six negative).<sup>1</sup> After information acquisition, participants are randomly assigned to thought conditions. In the unconscious-thought condition, participants engage in a distraction task for 3 or 4 min (e.g., an anagram or working memory task), during which they are unable to consciously think about the choice options. In the conscious-thought condition, participants are instructed to consciously deliberate on the information for the same duration. Further, many such experiments also include a control condition, like an immediatechoice condition in which participants make their judgments and choices immediately following information acquisition (i.e., leaving negligible time for conscious or unconscious thought; Dijksterhuis, 2004), or a mere-distraction condition in which participants are disabused of the goal to process the information just after it is presented, then engage in a distraction task (Bos et al., 2008). Finally, participants rate the options, or choose an option. Results from such designs commonly show that participants in the unconscious thought condition (relative to other thought conditions) form more favorable judgments toward the best option and less favorable judgments toward the worst option, or are more likely to choose the best option. One interpretation of this effect is that conscious capacity is limited and therefore cannot effectively process all of the relevant information (e.g., 48 attributes), whereas unconscious capacity is much larger and can handle the quantity of information to form reasonable judgments (Dijksterhuis and Nordgren, 2006).

<sup>1</sup>Research using this paradigm typically ensures that the "best" option is more desirable than the "worst" option. For example, attributes that are extremely positive or negative are excluded, or attributes across options are positive/negative versions of the same trait (e.g., "Car A gets good gas mileage" "Car B gets poor gas mileage"; see Dijksterhuis, 2004; Dijksterhuis et al., 2006; Bos et al., 2008; Ham et al., 2009; Usher et al., 2011; Manigault et al., 2015). The attribute stimuli used in the first experiment followed these parameters.

Past research has revealed that participants in unconscious thought conditions, relative to conscious thought or control conditions, integrate and evaluate considerable information (e.g., Dijksterhuis, 2004; Dijksterhuis et al., 2006; Bos et al., 2008; Lerouge, 2009; Ham and van den Bos, 2011; Manigault et al., 2015), weight information based on subjective importance (Bos et al., 2011), and recall information in a more organized fashion and clustered around similar traits (Bos et al., 2008). According to UTT, these patterns reflect a sophisticated unconscious thought process capable of integrating and organizing complex information. Of note, this pattern of results actually represents two different effects: an unconscious thought advantage (UTA; Nieuwenstein et al., 2015) relative to conscious-thought conditions, and a true unconscious thought effect (UTE) relative to control conditions such as mere distraction and immediate choice. Regardless, many interpret these patterns as evidence that unconscious thought can evaluate and process complex information in a goal-directed, slow, and seemingly analytical manner; qualities typically associated with rational (or conscious) processes in the CEST and related dual-process models.

Although many experiments and a meta-analysis reported by Strick et al. (2011) support the existence of the UTA and UTE, these effects remain somewhat controversial. Indeed, several papers also report failures to replicate, and a very recent meta-analysis and well-powered replication found no evidence for the UTA in choices using the type of information presentation described above (Nieuwenstein et al., 2015). For the present purposes, we offer that inconsistent results likely suggest unrevealed psychological moderators to the effects, and recommend well-powered experiments continue to test UTT.

# IS THE UNCONSCIOUS THOUGHT EFFECT MERELY THE RESULT OF INTUITIVE/EXPERIENTIAL PROCESSES?

Replications and meta-analyses aside, some researchers have proposed alternative explanations for the effects taken to support UTT (e.g., Lassiter et al., 2009; Usher et al., 2011; Nieuwenstein et al., 2015). Focal to the current paper, Usher et al. (2011) suggest that the decision benefit observed following a period of distraction vs. deliberation is not due to unconscious thinking per se, but to a reliance on the experiential processing system. They reasoned that "decisions performed in the distraction condition rely to a higher degree on intuitive strategies than decisions performed after deliberation" (p. 2). That is, the period of distraction relative to deliberation was thought to dampen down influences of the rational system and heighten the activation of the experiential system. This relative reliance on the experiential system should then impact how individuals process subsequent information (i.e., in a more holistic, emotional, intuitive manner). So, with a more active experiential system, individuals may "go with their gut" or use emotion or intuition to guide their choices.

In several experiments, Usher et al. (2011) use manipulations based on CEST (Epstein, 1994) to heighten the experiential system or the rational system. In one experiment, participants were induced into either an experiential mindset by focusing on and drawing their current emotional state for 3 min, or a rational mindset by solving math problems for the same duration. Next, participants made a complex decision about different car options. In line with their predictions, results revealed that participants primed with the experiential mindset chose the best car option more frequently and reported greater attitude differentiation between the best and worst cars (i.e., better evaluations overall) compared to participants primed with a rational mindset. These results support the notion that emotion-based decisions can be advantageous when the information is sufficiently complex.

In a follow-up experiment, Usher et al. (2011) added in a manipulation of thought (distraction vs. deliberation) used in typical UTT experiments to elicit unconscious vs. conscious thought, respectively. This was done to "maximize the difference between intuitive vs. analytic modes of thought" (p. 8), under the assumption that distraction would strengthen the experimental system and deliberation would strengthen the rational system. In Experiment 4, participants were induced into either a rational or experiential mindset as before, prior to receiving complex information describing three different roommate options (where one was objectively the best, one was objectively the worst, and one was neutral based on attribute qualities). The manipulation of thought (distraction vs. deliberation) was paired with the manipulation of mindset such that participants who were primed into a rational mindset additionally deliberated on the options, whereas participants who were primed into an experiential mindset were additionally distracted with anagrams. Results from this study revealed that participants in the distraction/experiential mindset condition rated the best roommate more favorably relative to participants in the deliberation/rational mindset condition, and had greater attitude differentiation scores between the best and worst roommates (i.e., overall evaluation). The authors suggested that this decision benefit resulted from processes in the experiential mode (they used the terms intuitive and affective) that were strengthened during the distraction period, rather than an active unconscious thought process independent of the two mindsets.

However, this experimental design did not allow for a clear test between the effects of unconscious thought and an experiential mindset on decision outcomes. The modes of processing (rational vs. experiential) and thought condition (conscious vs. unconscious) were confounded. In Experiment 4, only participants who were distracted from thinking about the information (i.e., unconscious thought condition) were placed in the experiential mindset, and only participants who consciously thought about the options were placed in the rational mindset (Usher et al., 2011). Logically then, it is impossible to know whether the observed difference was due to mindset, thought modality, or an additive effect due to redundant manipulations. To truly test whether the experiential processing mode is responsible for the decision benefit following distraction, it is necessary to conduct a fully crossed design in which mindset and thought modality are manipulated independently. We investigated exactly that design in our Experiment 1. As it stands,

it is possible that individuals will form better decisions when they are distracted, rather than when they think consciously, even if (or especially if) they are in an analytical mindset.

# RESEARCH QUESTION AND OVERVIEW OF EXPERIMENTS

Our research question is whether the experiential processing mode is solely responsible for the UTA and UTE observed following a period of distraction, or whether unconscious thought is orthogonal to experiential and rational modes. Ultimately, the answer to this question can help us understand whether the analytic or emotional nature of a process gives us clear insight into whether that process was conscious or unconscious.

We conducted two experiments to investigate our research question. In the first experiment, we largely replicated Usher et al.'s (2011) Experiment 4, except we randomly assigned participants to experience a rational or experiential mindset prior to receiving complex information describing three different roommates, and randomly assigned participants to deliberate on the roommate options or complete anagrams as a distraction during information acquisition.<sup>2</sup> If the UTA is merely the result of a dominant experiential system, then we should observe an advantage in roommate judgments and choices only among participants with an experiential mindset (as observed in Usher et al., 2011). Alternatively, if the UTA is driven by unconscious thought and independent from the experiential mode, we should observe the advantage in roommate judgments and choices in the unconscious-thought condition regardless of mindset.

In the second experiment we presented participants with a complex logical reasoning problem, then randomly assigned participants to adopt the goal (unconscious thought condition), or not (control condition) to solve that problem prior to a distraction task. Participants then reported solutions to this problem after the distraction task. The experiential system should offer no assistance in solving a complex logical reasoning problem (which would require the rational system). Therefore, if we observe an effect of unconscious thought (i.e., UTE) it would suggest that unconscious thought can process analytical information, and is likely independent from the experiential system.

# EXPERIMENT 1

We replicated the methods from Usher et al. (2011) and participants were induced into either a rational or experiential mindset before receiving information describing three different roommates. They were asked to form impressions of the roommates and ultimately make a decision about which one they would most like to live with. During the acquisition of the roommate information, participants either deliberated on reasons for liking or disliking each option (i.e., conscious thought) or they were distracted by solving anagrams (i.e., unconscious thought). We predicted that the experiential vs. rational mindset would lead to better judgments and decisions of the roommate options, consistent with Usher et al. (2011). We also predicted that a period of unconscious vs. conscious thought would lead to better judgments and decisions of the roommate options, consistent with UTT (Dijksterhuis and Nordgren, 2006). Critically, we predicted that unconscious thought would facilitate roommate judgments and decisions regardless of the initial mindset. That is, we predicted the emergence of a UTA independent of the mindset. This finding would support the existence of an unconscious thought process that is distinct from the experiential system.

## Method

#### Participants and Design

Three hundred nineteen undergraduate students<sup>3</sup> from Montana State University participated in the experiment for partial course credit. This experiment was approved by the Institutional Review Board of Montana State University, and participants gave their informed consent before participating. Participants were randomly assigned to the conditions of a 2 (Mindset: rational vs. experiential) × 2 (Thought: conscious vs. unconscious) betweensubjects design.

#### Materials

#### **Roommate descriptions**

The behavioral descriptions for the roommates can be found in Appendix A. They consist of the same 12 binary attributes (i.e., good/bad versions of the same trait) for each of three roommates (e.g., "Roommate A is a bit uptight" vs. "Roommate B is a relaxed and easygoing person"). Because we used the same binary attributes across all three roommates, we were somewhat able to control the relative desirability of each roommate option. The best roommate (i.e., the one presented most positively) had eight positive attributes and four negative ones, the worst (i.e., the one presented most negatively) had eight negative attributes and four positive ones, and the middle option had six positive and six negative attributes. Furthermore, the attributes were pre-tested on levels of importance to exclude extreme attributes that might unfairly bias the options (see Usher et al., 2011). In the current experiment Roommate B was the best option, Roommate C was the worst option, and Roommate A was the middle option (this was randomly determined).

#### **Mindset manipulation**

We used the same mindset manipulations as Usher et al.'s (2011) Experiment 2. Specifically, participants in the rational mindset condition solved math problems on paper for 3 min. The numeric values ranged from 2 to 3 digits (e.g., 24 × 153) and participants likely needed to work out the problems by hand. No calculators were allowed. Participants in the experiential mindset condition spent 3 min drawing a picture of their current emotional state. Instructions for this manipulation are as follows:

<sup>2</sup>We used the analytical vs. intuitive mindset manipulation from Usher et al.'s (2011) Experiment 2.

<sup>3</sup>Due to a programming error, we failed to collect demographic information.

"Feelings and attitudes can be expressed through a number of measures including creative expression. We would like you to draw a picture that describes your gut-level feelings about your emotional state right now" (Usher et al., 2011; p. 5; see Krauss et al., 2004 for the same manipulation). Instructions remained on the screen for the entire 3 min duration, and we provided participants with crayons, pens and paper. These manipulations are based on Epstein's (1994) CEST and are thought to manipulate a reliance on the emotion-based experiential system and the analytical rational system, respectively.

#### **Thought manipulation**

Most experiments examining UTT randomly assign participants to various thought conditions after they receive all the decision information. However, some experiments, like Experiment 4 of Usher et al. (2011), randomly assigned participants to receive the decision information interspersed with either a distraction task or deliberation periods (this is thought to be more ecologically valid). We elected to replicate Usher et al.'s methodology closely, and thus also interspersed the thought manipulation with the presentation of decision attributes. Specifically, participants received a cycle of 12 roommate descriptions (four descriptions per roommate; a mix of positive and negative), blocked per option, which was followed by either a period of distraction or deliberation (see **Figure 1**). Participants in the conscious thought condition deliberated on the options and were asked what they liked or disliked about each of the roommates, one roommate at a time. Participants in the unconscious thought condition were distracted and were given three anagrams to solve, one at a time (e.g., DAGERN is an anagram for GARDEN or DANGER).<sup>4</sup> The cycle of roommate descriptions and thought manipulation occurred three times, which results in approximately 3 min of distraction or deliberation time in total (see **Figure 1**).

#### **Task**

Participants received computer instructions indicating they would be presented with information describing three potential roommates, and they were to form impressions of the roommates based on the information and ultimately choose the best one to live with. The task consisted of participants reading 12 behavioral descriptions for each of the three roommates (e.g., "Roommate A has a good sense of humor") totaling to 36 pieces of information.

The presentation of the information was borrowed from the procedures of Usher et al. (2011) Experiment 4. Attributes were blocked per roommate option, such that four behavioral descriptions of the first roommate were presented on screen for 12 s followed by four behavioral descriptions of the second roommate for 12 s, followed by four behavioral descriptions of the third roommate for 12 s (see **Figure 1**). The order in which roommates were presented was randomly determined, but the order of attributes within each roommate was constant and the same for each condition. The order of attribute presentation for each roommate was as follows: Roommate B (best option) displayed 1 positive (+) and 3 negative (1) attributes in the first cycle, 4+ and 0− in the second cycle, and 3+ and 1− in the third cycle (totaling 8+ and 4−). Roommate C (worst option) displayed 2+ and 2− attributes in the first cycle, 0+ and 4− in the second cycle, and 2+ and 2− in the third cycle (totaling 8− and 4+). Roommate A (middle option) displayed 2+ and 2− attributes for each of the three cycles (totaling 6+ and 6−). Please see a complete list of attributes, in the order in which they were presented, in Appendix A.

Following the initial cycle of roommate information presentation (i.e., the first four descriptions of each roommate), participants were randomly assigned to deliberate on the roommate options (conscious thought condition) or to a distraction (unconscious thought condition). After the second round of roommate information presentation, participants again deliberated on the options or were distracted with anagrams (depending on condition). The entire task consisted of three cycles of roommate descriptions (12 attributes for each roommate; 36 attributes total) and three 1-min thought periods after each cycle of either deliberation or distraction.

#### Procedure

Up to six participants entered the laboratory per session and sat at individual computer stations. Participants first engaged in the mindset manipulation, determined via random assignment. Paper, pens, pencils, and crayons were present at each computer for participants to use; instructions for the manipulation were presented on the computer. This manipulation lasted 3 min. Next, participants received computer instructions indicating they would be presented with information describing three potential roommates, and they were to form impressions of the roommates based on the information and ultimately choose the best one to live with. At this point participants engaged in the roommate task (described above). The roommate task lasted roughly 10 min. Following this, participants were asked to rate each roommate on a scale of likeability (1 = dislike very much; 10 = like very much) and then were asked to choose one of the three roommates with whom they would most like to live. Finally, participants were thanked for their time, debriefed, and dismissed.

#### Results

#### Roommate Attitude

The individual roommate ratings were analyzed using a repeatedmeasures Analysis of Variance (ANOVA), which demonstrated the predicted likeability scores. Overall, participants rated the best option, Roommate B, most likeable (M = 6.69, SD = 1.85), the worst option, Roommate C, least likable (M = 4.45, SD = 1.77), and the middle option, Roommate A, in between [M = 5.26, SD = 1.81; F(2,636) = 118.54, p < 0.001, η 2 <sup>p</sup> = 0.272]. All three pairwise comparisons were significant (p < 0.001).

The primary dependent measure was an attitude difference score, calculated by subtracting the likeability score of the worst roommate from the likeability score of the best roommate (e.g., attitude BEST – attitude WORST). Larger values of this difference score reflect greater differentiation between the best

<sup>4</sup>Anagrams were pre-tested for an optimal word-length to ensure they were solvable in about 20 s. Participants had the option to skip ahead if they could not find the solution.

for 12 s before the next screen appeared with four descriptions of the next roommate. After three screens (and three roommates), participants engaged in the

thought manipulation [distraction vs. deliberation (think)] for 1 min. Then the cycle repeated. There were three cycles in total.

and worst options; that is, more favorable attitudes toward the best roommate option and less favorable attitudes toward the worst roommate option. This difference score in the evaluation measure is typically used in the unconscious thought literature and reflects a global evaluation of the options (e.g., Dijksterhuis, 2004).

The attitude difference score was subjected to a 2 (Mindset: rational vs. experiential) × 2 (Thought: conscious vs. unconscious) between-subjects factorial ANOVA. The analysis yielded no main effect of mindset, F(1,315) = 0.213, p = 0.644, indicating attitudes toward the best vs. worst roommates were similar in the rational (M = 2.31, SD = 2.81) and experiential (M = 2.17, SD = 2.67) mindset conditions. This is inconsistent with the findings of Usher et al. (2011). We did, however, observe a main effect of thought, F(1,315) = 6.58, p = 0.011, η 2 <sup>p</sup> = 0.02, such that participants in the unconsciousthought condition reported greater attitude differentiation (M = 2.61, SD = 2.79) than participants in the consciousthought condition (M = 1.85, SD = 2.64). This finding is consistent with UTT, such that a period of distraction that prevents conscious deliberation facilitates the processing of complex information.

The analysis did not yield a significant interaction between mindset and thought, F(1,315) = 2.243, p = 0.135, suggesting the UTA was comparable under both mindset conditions. However, we conducted simple-effect tests of thought at each level of mindset to test the competing predictions of Usher et al. (a UTA only within experiential conditions) and our prediction that the UTA is independent of the experiential mode and can occur under rational mindsets as well. Inconsistent with Usher et al.'s predictions and earlier findings, the effect of thought was non-significant in the experiential-mindset condition, F(1,315) = 0.59, p = 0.443, but was significant in the rational-mindset condition, F(1,315) = 7.98, p = 0.006, η 2 <sup>p</sup> = 0.049. Those who solved math problems at the beginning of the experiment (rational mindset) developed superior roommate judgments when they were distracted (M = 2.92, SD = 2.86) rather than when they deliberated (M = 1.68, SD = 2.63), Cohen's d = 0.45. The mean attitude difference

scores by thought condition and mindset are presented in **Figure 2**.

#### Roommate Choice

Next we assessed the choices participants made about which roommate they wanted to live with. Recall that Roommate B was the best option, Roommate C was the worst option, and Roommate A was the middle option. Overall, 71 participant chose roommate A (22.3%), 212 participants chose roommate B (66.5%), and 36 participants chose roommate C (11.3%). These numbers descriptively match what we predicted.

In order to test whether initial mindset or mode of thought influenced the roommate choices, we collapsed the frequencies of choosing the non-optimal choices (Roommates A and C) into a single incorrect choice, and the frequencies of choosing the best choice (Roommate B) into a correct choice variable. Then, we analyzed this dichotomous choice outcome (1 = correct,

0 = incorrect) in a logistic regression with the mindset condition, thought condition, and the Mindset × Thought interaction as predictors. The main effect of mindset (rational mindset as reference) was not significant, B = 0.28, SE = 0.33, Wald = 0.72, p = 0.396, odds ratio = 1.34, 95% CI [0.69, 2.55], which is reflected in the relative percentages of choosing the correct (vs. incorrect) roommate option for the experiential mindset condition (67.9%) in comparison to the rational mindset condition (64.9%). The main effect of thought (conscious thought condition as reference) was also nonsignificant, B = 0.50, SE = 0.34, Wald = 2.15, p = 0.143, odds ratio = 1.65, 95% CI [0.85, 3.21], yet the pattern of results trended toward a more frequently chosen correct option in the unconscious thought condition (70.1%) vs. the conscious thought condition (62.6%). The interaction between mindset and thought condition was also non-significant, B = −0.32, SE = 0.48, Wald = 0.45, p = 0.505, odds ratio = 0.73, 95% CI [0.29, 1.85]. In sum, although participants chose the best roommate option more frequently than the worst, neither initial mindset nor mode of thought significantly influenced this choice.

#### Discussion

The results of Experiment 1 support UTT, such that participants who were distracted from thinking about the roommate options formed attitudes that better differentiated between the best and worst options relative to participants who consciously deliberated on them (i.e., the UTA). Interestingly, and inconsistent with the results from Usher et al. (2011), we found no effect of mindset on roommate attitudes. Further, post hoc tests on our data revealed no evidence of a significant difference between the experiential/unconscious thought condition and the rational/conscious thought condition, t(315) = 1.498, p = 0.135; conditions that directly corresponded to Usher et al.'s Experiment 4.

The current study had a large sample size (N = 319, averaging 80 participants across four conditions) relative to Usher et al.'s Experiment 4 (N = 29, averaging 14–15 participants across two conditions), and thus more power. In their experiment, the difference in attitude scores between the experiential/unconscious thought and the rational/conscious thought conditions yielded an effect size of about 0.819 standard deviation units (e.g., Cohen's d), which is considerably large. Given our sample size, we had 80% power to detect an effect size of d = 0.314, which is much smaller. Therefore, we had the statistical power to detect a relatively small difference in attitudes between the experiential/unconscious thought and the rational/conscious thought conditions, yet we found none. Furthermore, by fully crossing the experimental design we were able see a more complete picture of the effects of rational and experiential modes and of unconscious/conscious thought processes on judgments of complex information. The effects of unconscious thought were orthogonal to those of the experiential and rational modes.

Thought and mindset did not interact to influence attitudes, indicating the benefit of distraction was comparable for participants in both the experiential and rational mindset conditions. However, further inspection using simple-effect tests to test competing predictions revealed that the UTA was only significant in the rational-mindset condition (see **Figure 2**). Cumulatively, then, our attitude results support the UTA, and suggest that unconscious thought is not redundant with processes in the experiential mode. Furthermore, they hint that unconscious thought can operate via the rational system. However, we found no evidence that mindset or thought condition influenced roommate choices, meaning participants' roommate attitudes did not translate into actual decisions.

One goal of the current research was to extend the research of Usher et al. (2011) and test whether unconscious thought is distinct from processes in the experiential mode. Results of Experiment 1 support this notion. Another goal of our research was to determine whether unconscious thought could be rational. There was evidence for this possibility in Experiment 1, as the UTA was largest when paired with a rational mindset. Experiment 2 provided a stricter test of this hypothesis; we gave participants a complex logical-reasoning problem to solve, and an opportunity to think unconsciously about the problem (distraction with a processing goal) or not (mere distraction without a goal). Using a control condition of distraction without a goal allowed us to test for possible effects of unconscious thought over a baseline. A benefit of unconscious thought over mere distraction in the logical reasoning problem would support the existence of a UTE and the hypothesis that unconscious thought can process information analytically. Support for this hypothesis would also strengthen the notion that unconscious thought is independent from the experiential mode (which theoretically should not guide logical reasoning).

# EXPERIMENT 2

Similar to traditional unconscious-thought experiments, participants in Experiment 2 received a sequence of complex information to consider (e.g., Dijksterhuis, 2004). However, this information is typically a set of affectively laden (good/bad) attributes describing roommates, cars, vacation destinations, etc. Instead, the current experiment used conditional statements in a logical-reasoning problem as our stimulus material, and these statements did not contain positive or negative connotations. If the emotion-based experiential system facilitates value integration and the weighting of good/bad qualities, it should offer little help in solving a logical-reasoning problem. The current logic problem was a complex deductive reasoning problem consisting of eight sentences and inherent rules to follow. We reasoned that if unconscious thought can be analytical then it should be able to integrate, organize, and evaluate information to deduce logical conclusions.

#### Method

#### Participants and Design

One hundred thirty-six undergraduate students from Montana State University participated in the experiment for partial course credit. This experiment was approved by the Institutional Review Board of Montana State University, and participants gave

their informed consent before participating. Participants were randomly assigned to one of three conditions: an unconscious thought condition (UT) and a mere distraction condition (MD) for our primary experimental comparison, and additionally a baseline condition in which participants solved the problem out right to determine base rates for solving the problem. The critical comparison to test our hypothesis was between the UT and MD conditions.

#### Procedure

Up to six participants entered the laboratory at a time and sat at individual computer stations. The entire experiment lasted approximately 30 min. Participants were informed that the purpose of the experiment was to investigate problem solving skills and that they would be exposed to a number of different types of problems. Importantly, participants were told that they might be asked to solve some problems or not, intentionally leaving some ambiguity regarding the goal of the experiment.

#### **Logical reasoning practice**

First, participants were introduced to the n-back task of working memory (Jonides et al., 1997). In this task, participants were presented with a random series of single-digit numbers in the center of the computer screen and had to press the space bar whenever a number appeared that had appeared two presentations earlier (e.g., 3, 7, 3). Participants were told that this task was just one of the many problem-solving tasks in the experiment, but it actually served as practice for the distraction task used later in the experiment. Participants engaged in the initial n-back for approximately 1 min.

Next participants were instructed to put on headphones and they subsequently learned about logical reasoning. They read instructions and listened to an audio recording over the headphones about the nature of logical reasoning. The recording explained that logical reasoning consists of drawing conclusions from given statements, and presented two simple examples (e.g., "If A is greater than B, and B is greater than C, is A greater than C?"). Following this introduction, participants were given a more complex logical-reasoning problem and were told they may or may not be asked to solve it later, and so must pay close attention.

#### **Logical reasoning main task**

The main task of the experiment was adapted from the Law School Admissions Test (LSAT) sample questions in the analytical reasoning section (Law School Admissions Council, Inc., 2012).<sup>5</sup> The problem consisted of eight conditional statements that relate to one another. Participants received the problem one sentence at a time, presented visually on the computer screen and audibly via headphones. The presentation of information was very similar to a typical unconscious-thought experiment in which participants receive pieces of information one at a time and must integrate them (e.g., Dijksterhuis, 2004). The problem involved a student needing to perform six different activities on a particular Saturday (e.g., grocery shopping, hedge trimming, jogging, kitchen cleaning, laundry, and motorbike servicing). In the logic problem, the protagonist needed to

<sup>5</sup>http://www.lsac.org/jd/lsat/prep/analytical-reasoning

perform each activity once and in a particular order. The order of the activities was conditional (e.g., laundry must occur before hedge trimming). Therefore, deductive reasoning was required to determine the correct order of activities.

After being presented with the logic problem, participants saw a screen that said, "An example of a question that can be drawn from the preceding statements is 'If jogging follows grocery shopping, then which activity must occur first'? which was followed by the six options. Importantly, participants were not asked to solve the problem, but were ostensibly shown it as an example of a possible problem. At this point, all participants had an identical opportunity to encode the information they needed to solve the problem (i.e., the conditional statements and problem itself) but did not yet have a goal to solve it.

#### **Goal manipulation**

Participants in the unconscious-thought condition (UT, n = 68) were told that they would have to solve the logical reasoning problem later in the experiment, and to keep it in the back of their mind. Participants in the mere-distraction condition (MD, n = 68) were not given a goal to solve the problem. Rather, they were asked a simple question about the information: Is it true that all activities could occur on a single day? (Yes or No). Participants in the mere distraction condition answered this question to ensure that no goal to solve the logic problem remained during the upcoming distraction period.

After information acquisition and the goal manipulation, participants in the UT and MD conditions engaged in the n-back task again as a distraction task for 3 min. Finally, participants were presented with the original question that appeared at the end of the logic problem earlier in the experiment (If jogging follows grocery shopping, then which activity must occur first?), accompanied by each of the answer options. Participants were encouraged to guess if they did not know the answer. Importantly, participants the UT condition expected to receive this problem again at some point in the experiment, but did not know when (hence they had a goal to solve it), but participants in the MD condition did not have this goal. Both groups, however, were distracted from thinking about the problem during the 3 min n-back task.

Recall that we included a baseline condition to assess the base rate for solving this problem. Participants in this condition (n = 70) received the same information as the UT and MD conditions, but were simply given the question at the end and time to solve it (they were not distracted with the n-back). This allowed us to assess the base-rates for solving the logical reasoning problem under normal conditions, and confirm that it was in fact a complex problem.

#### Results

For the dependent measure we created a dichotomous variable indicating whether the participants correctly chose the right answer (correct = 1) or chose one of the other five options (incorrect = 0) as the answer. In this particular example,

the correct answer was D, kitchen cleaning. The frequencies with which participants chose each option are presented in **Table 1**. Of note, participants in the base-rate condition chose the correct option 50% of the time, suggesting problem was challenging and difficult to solve under normal test-taking situations.

We analyzed the dichotomous dependent measure using a chi-square test of independence between the UT and MD groups. Results indicated that the frequency of choosing the correct vs. incorrect option depended significantly on whether participants were in the UT condition or the MD condition, X 2 (1, N = 136) = 3.97, p = 0.046. Participants in the UT condition correctly solved the logic problem more frequently (17 times or 25% of participants) than participants in the MD condition (8 times or 11.8% of participants). In other words, participants who held a goal to solve the problem during the distraction period were more accurate than participants who were merely distracted without such a goal.

#### N-Back Performance

To ensure that an equal amount of conscious attention was devoted to the distraction task in both the UT and MD conditions, we analyzed performance on the n-back task between the two groups. In the n-back, participants had to press a space bar whenever a digit appeared on the screen that appeared 2 trials ago (e.g., 3, 7, 3). During this task there were 14 trials that required a response (hit) and 76 trials that did not (responses here are "false alarms"). The number of hits did not differ between the UT (M = 11.91, SD = 2.49) and MD (M = 11.13, SD = 3.28) conditions, t(134) = 1.557, p = 0.122. Likewise, false alarm rates for the UT (M = 4.91, SD = 11.78) and MD (M = 3.32, SD = 4.04) conditions did not differ, t(82.56) = 1.05, p = 0.296.<sup>6</sup> Thus, participants' performance on the distraction task was comparable across the two conditions, and therefore cannot account for the performance differences on the logical reasoning problem.

## Discussion

Previous research has largely studied unconscious thought in the judgment of affectively laden (positive/negative) information such as the preference for cars, roommate, or apartments (Dijksterhuis, 2004; Dijksterhuis et al., 2006; Bos et al., 2008; Lerouge, 2009; Ham and van den Bos, 2011; Messner et al., 2011). In contrast, the current experiment used materials from a logic problem. We presented participants with a deductive reasoning problem one conditional statement at a time, and then distracted participants from thinking about the problem while they either did (UT condition) or did not (MD condition) have a goal to eventually solve the problem. Results indicated that participants who had a goal to solve the logic problem during the distraction period chose the correct answer more frequently than those who were distracted without a goal. This finding cannot be explained by the workings of an emotionbased experiential system, because the logical reasoning problem required the use of rules which is better accomplished by the analytical rational system (e.g., Epstein, 1994; Sloman, 1996; Kahneman, 2003). Rather, the current findings uncover the possibility that unconscious thought can operate via the rational system.

The current results are, of course, preliminary. Although our sample size was substantial (68 per condition), our outcome measure was simply a one-shot answer to a multiple choice question, which certainly leaves room for error. The most conservative conclusion we can draw from these results is that unconscious thought does not appear to be redundant with processes in the experiential system, and initial evidence supports the idea that unconscious thought can operate in an analytical fashion.

## GENERAL DISCUSSION

Complex judgments and decisions require careful consideration and thought. Extensive debate has surrounded the specific type of thought that is best suited for such a task, and analytical/rational and emotional/experiential modes of processing have played central roles (e.g., Epstein, 1994; Sloman, 1996; Kahneman, 2003; Evans, 2008; Mikels et al., 2011; Usher et al., 2011). Unconscious thought is another mode of processing that has been implicated in complex judgments and decisions (Dijksterhuis, 2004; Dijksterhuis and Nordgren, 2006; Dijksterhuis et al., 2006; Dijksterhuis and Strick, 2016), yet has been confounded both theoretically (e.g., dual-process theories) and empirically with fast, experiential, intuitive, and affective processes. The current research tested the hypothesis that unconscious thought is independent from processes in the experiential mode as proposed by CEST, and explored the possibility that unconscious thought can be rational. Results support the existence of a UTE and UTA in complex information processing, and also provide evidence that unconscious thought is distinct from the experiential and rational modes, and can be analytical.

# Evidence for an Unconscious Thought Effect

The results of two experiments demonstrated the existence of an effect and advantage of unconscious thought over alternative forms of thinking. In Experiment 1, participants who were distracted with anagrams made superior judgments about roommates relative to participants who thought consciously about the options. This effect did not carry over to the final roommate choice, however. Experiment 2 revealed a difference between goal-directed unconscious thought and mere distraction in solving a logical reasoning problem. Thus, two separate experiments with diverse methods revealed a benefit of unconscious thinking over deliberation (Experiment 1) and mere distraction (Experiment 2) in complex informationprocessing.

<sup>6</sup>Degrees of freedom are corrected due to a violation in the assumption of equality of variances.


TABLE 1 | Frequencies of choosing each option for the logical reasoning problem in Experiment 2, separated by thought condition: unconscious thought (UT), mere distraction (MD), and baseline (Base).

The existence of a sophisticated unconscious mode of thought has come under scrutiny and criticism (e.g., González-Vallejo et al., 2008; Lassiter et al., 2009; Newell and Shanks, 2014; Nieuwenstein et al., 2015), but past research and a meta-analysis have found evidence for an effect (see Strick et al., 2011). The current results add to the literature by providing two wellpowered experiments that reveal a positive effect of unconscious thinking (vs. conscious thought and mere distraction) on outcomes related to judgments of roommates and analytical problem solving.

We borrowed methods from previous research for Experiment 1 (Usher et al., 2011) and developed methods for Experiment 2 based on an existing UTT paradigm (e.g., Dijksterhuis, 2004). Therefore, we are confident that we manipulated mode of thought as intended and believe that the observed effects are best interpreted in light of established research paradigms and theory. In that case, the current results support the existence and advantage of a goal-directed unconscious thought process in complex judgments.

# Unconscious thought Is Not Merely Experiential

More novel was the evidence from the current research that unconscious thought is unique from experiential/emotional processing modes, and can be analytical. In Experiment 1, roommate descriptions were better integrated and processed by unconscious (vs. conscious) thought, but especially when paired with a prior rational mindset. This finding is inconsistent with the notion that unconscious thought effects are merely the result of a dominant experiential system.

Experiment 2 extended the idea that unconscious thought can operate via the rational system by testing the effect of unconscious thought in a logical deductive reasoning problem, and results revealed that unconscious thought increased the frequency of solving the reasoning problem relative to mere distraction. We take this result with caution because it is based on a single outcome and rates for solving the problem correctly were low (18.4%) among the UT and MD conditions. Yet, differences between these two conditions were significant, and in the hypothesized direction, suggesting that unconscious thought (vs. mere distraction) contributed to logical reasoning.

Components of the experiential system include affective processes and intuition, both of which have been implicated in value integration and optimal judgments of complex information (e.g., Wilson and Schooler, 1991; Bechara and Damasio, 2005; Slovic et al., 2007). Yet, the effects of unconscious thought appear to be distinct from these. Effects of unconscious thought in the current research emerged when affective processes were dampened down by a rational mindset (Experiment 1) and when the information to-be-processed was purely logical in nature (Experiment 2). Furthermore, although intuitions may arise from inaccessible and unconscious processes, they are often experienced at a conscious level as a "nudge" or "gutlevel" preference for something (e.g., Price and Norman, 2008). Based on UTT, we have no reason to believe that products of unconscious thought are felt consciously. However, we did not assess subjective feelings at the time participants made their judgments, and it is therefore possible that they experienced some sort of "nudge" toward one option over the others. Nonetheless, the current research suggests that unconscious thought is not redundant with processes in the experiential system because the effects of unconscious thought emerged in non-affective domains and (presumably) without the inklings of intuition.

# Implications and Future Research

We see at least three important implications of our research findings. First, our results support the idea that equating experiential/emotional systems to fast unconscious processes, and analytical/rational systems to slower conscious processes, is a misleading overgeneralization. The evidence from the two current experiments suggests that—at least under some contexts—slow yet unconscious processes can consider information and form judgments and decisions independent of experiential or rational systems. Thus, the speed of, or the evident logical or emotional basis of, a decision or choice might not serve as a useful clue that conscious or unconscious processes produced the response. Future research and theorizing may benefit from considering alternative, or less constraining, categories of information processing and problem solving.

Second, our results challenge the idea that unconscious thought is not rule-based. Dijksterhuis and Nordgren (2006, p. 101) offer the rule principle in UTT which "states that conscious thought can follow strict rules and is precise, whereas unconscious thought gives rough estimates." To be clear, research does indicate that conscious thought outperforms unconscious thought in gambling and betting tasks which require probabilistic rule-based solutions (Payne et al., 2008). Apparently, unconscious thought cannot do math. Yet, our second experiment demonstrates that unconscious thought can follow rules provided in an analytic problem, and use those

rules to successfully solve the problem more frequently than the negligible thought involved in control-group participants. It is also noteworthy that the problem used in Experiment 2 involved ordering a person's tasks in a day, a scenario with which undergraduate participants are likely more familiar than gambling tasks. All of this is to say that there are apparently some contexts in which unconscious thought can follow, or use, rules to increase the likelihood of correctly solving a problem. It will be important for theorists to carefully consider what type of rules unconscious thought can consider, and in what contexts, and for future research to investigate these possibilities.

Finally, we believe that our research, added to the growing body of research investigating the UTA and UTE, strongly implies psychological moderators to the effects of unconscious thought. Both the UTA and UTE are fairly small effects according to Strick et al.'s (2011) meta-analysis. Further, a recent meta-analysis only of experiments following the "multi-attribute paradigm" (e.g., Experiment 1 in which several options are described by multiple attributes), and a highly powered experiment using car options, finds no UTA using that paradigm (Nieuwenstein et al., 2015). Yet, our first experiment, powered with data from 319 participants, found a significant UTA using the multiattribute paradigm (as have other recent, and well-powered, experiments, e.g., Manigault et al., 2015). Thus, we offer that the issue might be less about whether unconscious thought outperforms conscious thought, but in what contexts, or with what information.

# Limitations

The main purpose of the current research was to test the possibility that unconscious thought can benefit decision-making in analytical contexts, and ultimately test whether unconscious thought is distinct from the emotion-based experiential system. Although we present two experiments that support this possibility, these experiments are limited in their ultimate conclusions.

One limitation in Experiment 1 is that we assumed that Roommate B was the best option and Roommate A was the worst option. Although we used stimuli from past research that were constructed to ensure that one roommate had the most "objectively" desirable traits (and one had the least), and were pre-tested on levels of importance to exclude extreme attributes (see Usher et al., 2011), we do not know the subjective importance participants assigned to each trait. For instance, personality may be more or less important than cleanliness or sense of humor to a certain individual. Nonetheless, these attributes were used in prior research (Usher et al., 2011, Experiments 3 and 4) and based on numerous other UTT experiments with multi-attribute decision paradigms (see Dijksterhuis, 2004; Bos et al., 2008; Dijksterhuis et al., 2006; Ham et al., 2009; Usher et al., 2011; Manigault et al., 2015).

On a related note, the first cycle of roommate attribute presentation could be conceptualized as a "first impression," and we did not fully balance the valence of these initial presentations. Specifically, the best option "B" initially displayed three negative and one positive attributes, the worst option "A" displayed two positive and two negative attributes, and the middle option "C" displayed two positive and two negative attributes (see Appendix A). Although the relative number of positive and negative attributes varied with each cycle, the first cycle may have had a relatively stronger influence on impressions. Even though the best option "B" was negatively skewed in this initial presentation, it was still rated most favorably overall. So, if a bias existed early on it did not appear to influence overall roommate judgments.

Additionally, we did not have a manipulation check for the mindset manipulation in Experiment 1. Although this manipulation has been used successfully in past research (Krauss et al., 2004; Usher et al., 2011), and all of the current participants complied with instructions, a measure of relative reliance on the rational vs. experiential mode would have made our conclusions stronger. Likewise, a measure of dispositional reliance on the experiential vs. rational system (i.e., Rational Experiential Inventory; Pacini and Epstein, 1999) would have allowed us to assess potential individual differences on this dimension.

Another limitation of the current experiments is that interpretations rely on behavioral outcomes far removed from the covert psychological processes involved in unconscious thought. Future research should take advantage of neuroimaging techniques (see Creswell et al., 2013, for example) or other techniques (e.g., thought probes) that can assess the neural or psychological mechanisms occurring during the distraction period, rather than relying solely on behavioral outcome measures.

# CONCLUSION

Past research has shown that judgments and decisions of complex information benefit from a period of distraction relative to deliberation or no time period. This benefit is presumed to result from goal-directed unconscious thought occurring during the distraction period (Dijksterhuis and Nordgren, 2006). Yet, it has been suggested the observed benefit from distraction is not due to unconscious thinking per se but instead to the emotionbased experiential system, which is presumably strengthened during a period of distraction (relative to deliberation). We replicated past research and systematically manipulated a reliance on the experiential vs. rational system (Experiment 1) and discovered that a period of distraction facilitated outcomes independently from these two processing modes (and to a greater degree when paired with the rational mode). Thus, a dominant experiential mode cannot solely account for the decision benefits observed following distraction. We attribute this benefit to an active unconscious thought, consistent with UTT. In a followup experiment (Experiment 2) we manipulated unconscious thought (vs. mere distraction) toward solving a logical reasoning problem and found that unconscious thought was superior at this analytical task. Taken together, these results suggest that unconscious thought can facilitate the processing of complex information, as proposed by UTT, and is not redundant with processes in the experiential system. In fact, unconscious thought may be rational.

# ETHICS STATEMENT

fpsyg-08-01096 July 4, 2017 Time: 16:2 # 12

This project was approved by the Institutional Review Board of Montana State University. Participants were healthy, college-aged students at a public university. All participants were provided detailed information about the experiments, and consented to participation before the experiment began. Participants were allowed to withdraw at any point in time (none did). Participants were compensated with course credit for participation, and there were no risks beyond the minimal risks one experiences in everyday life. Participants were debriefed fully at the end of the experiment, allowed time to ask questions, and were given contact information for future inquiries.

## REFERENCES


# AUTHOR CONTRIBUTIONS

IH was the graduate advisor of KG during the time of this work. As a mentoring team, KG and IH developed the idea and design of this work, collected and analyzed all data, and drafted and revised the manuscript.

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg. 2017.01096/full#supplementary-material


Wilson, T. D., and Schooler, J. W. (1991). Thinking too much: introspection can reduce the quality of preferences and decisions. J. Pers. Soc. Psychol. 60, 181–192.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Garrison and Handley. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# What We Talk about When We Talk about Unconscious Processing – A Plea for Best Practices

#### Marcus Rothkirch\* and Guido Hesselmann\*

Visual Perception Laboratory, Department of Psychiatry and Psychotherapy, Campus Charité Mitte, Charité – Universitätsmedizin Berlin, Berlin, Germany

#### Edited by:

Nathan Faivre, École Polytechnique Fédérale de Lausanne, Switzerland

#### Reviewed by:

Zhicheng Lin, Ohio State University at Columbus, United States Miguel A. Vadillo, Universidad Autonoma de Madrid, Spain

#### \*Correspondence:

Marcus Rothkirch marcus.rothkirch@charite.de Guido Hesselmann guido.hesselmann@charite.de

#### Specialty section:

This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology

Received: 01 February 2017 Accepted: 08 May 2017 Published: 23 May 2017

#### Citation:

Rothkirch M and Hesselmann G (2017) What We Talk about When We Talk about Unconscious Processing – A Plea for Best Practices. Front. Psychol. 8:835. doi: 10.3389/fpsyg.2017.00835 In this perspective article, we first outline the large diversity of methods, measures, statistical analyses, and concepts in the field of the experimental study of unconscious processing. We then suggest that this diversity implies that comparisons between different studies on unconscious processing are fairly limited, especially when stimulus awareness has been assessed in different ways. Furthermore, we argue that flexible choices of methods and measures will inevitably lead to an overestimation of unconscious processes. In the concluding paragraph, we briefly present solutions and strategies for future research. We make a plea for the introduction of "best practices," similar to previous attempts to constitute practicing standards for functional magnetic resonance imaging (fMRI) and electroencephalography (EEG).

#### Keywords: consciousness, awareness, unconscious processing, best practices, diversity

# INTRODUCTION

To what extent are our actions and thoughts determined by "unconscious" influences, that is, by influences that we ourselves are not aware of? Sixty years after the infamous marketing campaign launched by James Vicary in 1957, who claimed that unconsciously flashed messages can affect observers' consumption behavior (Karremans et al., 2006), this question has lost nothing of its fascination. What is more, the development of scientific methods, like neuroscientific instruments or psychophysical "blinding" techniques, has made it possible to assess unconscious processes from different perspectives and within a wide range of contexts. There is, however, a downside to this multifaceted approach to unconscious processing: It comes along with substantial diversity between scientific studies.

In the following sections, we will first provide a brief overview of aspects in consciousness research that are affected by this diversity. Afterward, we will outline the risks associated with such divergent approaches. We will conclude with a section on practical recommendations for future research.

# DIVERSITIES REGARDING DIFFERENT EXPERIMENTAL ASPECTS

#### Diversity in Suppression Techniques

Today, a wide range of psychophysical paradigms exist to experimentally manipulate sensory awareness, particularly within the visual domain (Bachmann et al., 2007). Such psychophysical "blinding" methods can evoke a transient blindness in neurologically intact observers, i.e., they can be used to render a physically presented target stimulus (or, features of it) invisible, in most cases for a limited amount of time. It is important to keep in mind that the available paradigms vary with respect to what types of stimuli can be suppressed from awareness, and how effective the suppression is, for instance in terms of duration and predictability of suppression onset and offset (Kim and Blake, 2005). But there is yet another level of diversity related to the "blinding" methods that has only recently begun to be studied systematically. In 2014, a Frontiers Research Topic ["Invisible, but how?"; edited by (Dubois and Faivre, 2014)] addressed the issue of the inherent differences in the amount of information let through by different suppression techniques. Put simply, it is not enough to know that a stimulus has been invisible, but it is crucial to also know how exactly invisibility was achieved, because it has become apparent that different levels of suppression depth are associated with different suppression methods. Recently, this notion has been formalized into a functional hierarchy of unconscious visual processing (Breitmeyer, 2014). Eventually, such a functional hierarchy will allow to formulate predictions about the level of unconscious processing that can be expected in a specific experimental setup (Breitmeyer, 2015).

In the absence of such prior assumptions on the depth of visual suppression associated with a specific paradigm, every new study reporting high-level unconscious processing is equally weighted, independently of the applied suppression method. When accumulating evidence across single studies, this might lead to the wrong conclusion that unconscious perception is unlimited. However, it is known that different suppression methods are associated with different levels of suppression. Therefore, for example, the evidence presented for unconscious processing under CFS should be stronger, as a plausible prior assumption could be that CFS leads to a relatively high level of suppression. Ideally, the functional hierarchy would be based on cumulative research strategies, and incorporate not only studies that directly compared results from different methods with each other (Izatt et al., 2014; Peremen and Lamy, 2014a), but also take data from neuroimaging studies into account (Fogelson et al., 2014; Ludwig et al., 2016).

### Diversity in Awareness Measures

The question of how to optimally measure awareness in experiments on unconscious processing has been a long-standing one, and the issue is still much debated (e.g., Merikle and Reingold, 1990; Kunimoto et al., 2001; Schmidt and Vorberg, 2006; Sandberg et al., 2010). While criteria for valid awareness measures have been formulated (Shanks and John, 1994; De Houwer et al., 2009; Newell and Shanks, 2014), there is no accepted "gold standard." We will briefly outline the diversity of frequently applied measures of awareness, but this overview is not meant to be exhaustive. One crucial differentiation is the distinction between objective and subjective measures of awareness. According to the objective awareness criterion, forced-choice detection or discrimination performance above chance level indicates stimulus awareness, while performance at chance level indicates its absence. Subjective measures of awareness, on the other hand, are based on participants' metacognitive judgements on their own mental states (Lau and Rosenthal, 2011). Usually participants perform such judgments either on their own experience of the stimulus, or on their accuracy in a discrimination task. In the first case, participants are required to rate the visibility of the stimulus, either on a larger (Sergent and Dehaene, 2004) or smaller scale (Ramsøy and Overgaard, 2004). In the second case, participants have to evaluate how confident they felt with their response in a previously performed discrimination task (e.g., Rothkirch et al., 2012). The pros and cons of objective and subjective awareness measures have been summarized elsewhere (e.g., Hesselmann, 2013). A further distinction can be made between filtering measures and aggregate measures of awareness. If behavioral reports are provided on a trial-by-trial basis during the main experiment ("online"), filtering allows the experimenter to postselect subsets of trials for further analysis (e.g., "seen" vs. "not seen"). By contrast, aggregate measures allow inferences only about blocks of trials and thus depend on more than a single trial (e.g., percent correct, d'). Behavioral reports may be recorded either after a block of trials or in a separate control experiment ("offline," or "two-task design"). If a pre-defined criterion, such as performance at chance level, is satisfied, the experimenter infers that stimulus awareness was absent during the main experiment (or, in a given block of trials). Importantly, diversity does not end here, and the devil is in the details. For example, it has been shown that the exact composition of a block of trials (e.g., blocks with weakly and fully visible stimuli, vs. blocks with only weakly visible stimuli) can influence the experienced level of awareness [(Lin and Murray, 2014); also see (Pratte and Rouder, 2009)]. Levels of stimulus awareness may also change in longer experiments, e.g., awareness can increase across trials due to perceptual learning (Schwiedrzik et al., 2011; Ludwig et al., 2013). Finally, the presence or absence of trial-by-trial awareness reports can in turn influence the effect of interest, e.g., response priming (Peremen and Lamy, 2014b).

# Diversity in Statistical Analysis

We will briefly illustrate the diversity in the statistical analysis of awareness test data by an example. It is not uncommon that participants perform a two-alternative forced-choice (2AFC) discrimination task in a control experiment, either in addition to a subjective awareness measure (e.g., Hesselmann et al., 2016), or without any further awareness tests during the main experiment (e.g., Koechlin et al., 1999; Mattler and Palmer, 2012). To show that participants did not perceive the presented stimuli, researchers then often apply null hypothesis significance testing (NHST) on the 2AFC data, either at the subject level or at the

group level. Alternatively, the researcher may choose to apply a cutoff, e.g., consider 2AFC performance below 60% as indicative of an absence of stimulus awareness. It is well understood that when the NHST procedure is underpowered (or, the power is unknown), the experimenter risks to falsely accept the null hypothesis, i.e., conclude that there is no deviation from chance level when there actually is a deviation (Vadillo et al., 2016b). For example, the number of trials included in the awareness test may determine whether participants are categorized as aware or unaware, simply because tests comprising more trials also yield more sensitive measures. There have been suggestions to avoid the problem of type II errors by using equivalence tests and equivalence confidence intervals (Overgaard et al., 2013). A challenge related to such equivalence tests, however, is the a priori definition of a boundary around chance level that is still considered acceptable to indicate unawareness (Lin and Murray, 2015). Recently, Bayesian statistics have more frequently been applied to establish chance performance (e.g., Dienes, 2015; Sand and Nilsson, 2016). The details of this approach are beyond the scope of this perspective article, but it is important to keep in mind that Bayesian statistics are inherently diverse too. For example, when using Bayes factors to estimate the evidence in favor of the null model, there are different options for specifying the predictions of the alternative model (i.e., the case where participants saw the stimulus). One particularly relevant feature of the Bayesian approach seems to be sequential sampling, i.e., the strategy to collect more data until the evidence in favor of one model is considered as conclusive. Beyond the question of NHST or Bayesian statistics, post hoc selection of data has frequently been used, for example when residual stimulus visibility greatly varies between participants in experiments using interocular suppression (e.g., Sklar et al., 2012). The idea is to perform the main statistical analysis on a post hoc selection of the recorded data, e.g., by excluding participants whose subjective or objective behavioral reports indicated stimulus awareness, or by exclusively analyzing "not seen" trials. It has, however, been criticized to interpret "not seen" trials purely in isolation (Schmidt, 2015). Instead, these trials should be contrasted against seen trials, i.e., trials in which the suppression of the critical stimulus was not successful (cf. Madipakkam et al., 2015). Furthermore, especially in cases in which a large number of trials or participants are excluded, data analysis can resemble some form of extreme group analysis. Due to regression to the mean such an analysis strategy can yield a statistical bias in the selected sample and let the researcher erroneously assume unawareness in the selected participants (Shanks, 2016). The magnitude of regression to the mean in a data set is indicated by the correlation between the awareness measure and the measured effect. In this context, a model taking only into account regression to the mean can function as a null model against which the experimental effect can be tested. It has to be noted, however, that the extent to which the experimental effect differs from such a null model neither conclusively demonstrates nor precludes an unconscious effect.

#### Diversity in Experimental Setting

In a typical psychological experiment, stimuli are presented to participants who are asked to elicit a particular response to these stimuli. If such an experiment is intended to investigate unconscious processes, the unconscious dimension could be related to different aspects of the experimental setup. Firstly, participants could be unconscious of the presented stimulus, which is usually achieved by some form of masking (Kim and Blake, 2005). The second category is constituted by studies in which the putatively unconscious process pertains to the relationship between the stimulus and participants' behavior. This is, for instance, typically the case in studies on social priming (Bargh, 2016). Finally, the behavioral response or sequence of responses elicited by the participants might be unconscious, as in studies focusing on learning processes (Destrebecqz and Cleeremans, 2001).

While in cognitive psychology the focus lies primarily on the unawareness of the stimuli, in social psychology the relation between stimuli and observers' behavior is often the primary target (Doyen et al., 2014). Especially in the latter case, however, it proves difficult to determine whether the process of interest is indeed unconscious. As pointed out by (Stafford, 2014), for instance, the degree to which observers lack knowledge about associations between stimuli and their behavior that is required to be considered 'unconscious' is conceptually elusive. An ostensible remedy often chosen in the face of this predicament is then to mask the stimulus such that it cannot be consciously perceived anymore. This demonstrates that the different approaches to study unconscious processes are often used as if they were interchangeable, although they might target distinct processes.

Remarkably, it is widely accepted, on the one hand, that consciousness is not a unitary process, but that there are instead distinct modes of 'consciousness,' which, for instance, implies that observers might have conscious knowledge about one aspect of a particular stimulus while lacking conscious access to other aspects of that same stimulus (Zeki and Bartels, 1998; Navajas et al., 2014). In contrast, however, unconsciousness is sometimes treated as a singular process that does not require a further differentiation, although the term "unconscious" is not unitary either (Moors and De Houwer, 2006).

# CONSEQUENCES OF THE OUTLINED DIVERSITIES

As summarized above, researchers have several options to study and identify unconscious processes, ranging from the experimental design to the statistical analysis of the acquired data. While this patchwork of approaches could be deemed beneficial in the sense that it entails a more extensive overview of the phenomenon of unconscious processing, we argue that it more likely has the opposite effect. It should be noted that the diversities detailed above have distinct implications. On the one hand, they can have direct consequences on the interpretability of findings from single studies, which can, in severe cases, imply that participants were actually not unaware of the presumed unconscious process. Such consequences are especially related to the diversities discussed in section "Diversity in Awareness Measures and Diversity in Statistical Analysis." On the other hand, the diversities regarding the experimental setting

(see section Diversity in Experimental Setting) and the suppression techniques (see section Diversity in Suppression Techniques) have less impact on single studies, but play a more important role at the conceptual level, that is, when the findings across several studies are intended to be integrated into a general framework or model of (un)consciousness.

Specifically, studies differ with respect to the way awareness or unawareness, respectively, is assessed and statistically analyzed. This means that different criteria are set to determine whether participants are aware or unaware of a particular stimulus or process. For a given study, it is thus not sufficient to simply know whether participants were unaware, but instead according to which criterion participants were unaware, since it cannot be precluded that other criteria might indicate participants' awareness of the critical stimulus or process. This, however, limits the comparability between different studies, especially when awareness has been assessed in different ways. For example, two functional magnetic resonance imaging (fMRI) studies have investigated neural activity in visual areas when images of objects were rendered invisible with CFS (Fang and He, 2005; Hesselmann and Malach, 2011). In one study (Fang and He, 2005), participants were asked to report whether they perceived any shape or object in the preceding block of trials. In the other case (Hesselmann and Malach, 2011), participants provided trial-by-trial visibility ratings on a 3-point scale. It could be argued that only a limited comparison of the effect of interest is possible between the two studies, since evidence for participants' unawareness was based on different types of responses.

Another critical aspect that especially pertains to the assessment of observers' awareness is the flexibility in data analysis based on the vast amount of awareness definitions. Simply put, if researchers are interested in demonstrating that observers were unaware they could choose to report a particular analysis that indeed speaks in favor of observers' unawareness, while neglecting that other criteria might have indicated awareness. For example, when faced with the divergence of subjective and objectives measures in some participants, researchers may argue in terms of "blindsight" in normal observers (Hesselmann et al., 2011), criticize the underlying assumption of different objective and subjective thresholds (Peters and Lau, 2015), or choose one or the other measure to demonstrate observers' unawareness. Such a flexible choice could overstretch the concept of unawareness and consequently lead to an overestimation of unconscious processes, as the researcher has the "freedom" to report only the particular analysis that confirms their expectation (cf. Simmons et al., 2011).

On a more conceptual level, the vagueness of the terms "awareness" and "consciousness" allows one to operate with them in different contexts (Moors and De Houwer, 2006). If two different studies make reference to "unconscious processes," the reader may likely assume that similar processes have been studied, although one study may, say, use visually masked stimuli while the other one may focus on "unconscious processes" in response to visible stimuli. Such an inflationary reference to "awareness" and "consciousness" bears the risk of counteracting a differentiation between qualitatively distinct phenomena and may eventually – perhaps even inevitably – lead to the claim that unconscious processes can carry out every fundamental highlevel cognitive function that conscious processes can perform [(Hassin, 2013); for a reply (Hesselmann and Moors, 2015)], and thus impede deeper insights into the nature of unconscious processes.

Finally, one can speculate that the outlined diversities also contribute to the current crisis of confidence in psychological research. Some of the findings on unconscious processes, including, among others, effects of social priming and unconscious perception, turned out to be difficult to replicate (Doyen et al., 2012; Hesselmann and Knops, 2014; Moors et al., 2016), show clear signs of bias (Shanks et al., 2015; Vadillo et al., 2016a), or are open to alternative explanations (Stein et al., 2016; Street and Vadillo, 2016).

# SOLUTIONS AND STRATEGIES FOR FUTURE RESEARCH

One obvious strategy to handle the described diversity is the formulation of explicit guidelines for the assessment and statistical analysis of awareness. Such guidelines may not only help to unify the fragmented approaches that exist to study unconscious processing, but also may serve as a guide for researchers who are not yet acquainted with the peculiarities and pitfalls related to the assessment of observers' level of awareness. They should, however, not be understood as a discrete set of binding rules. Instead, we propose that the whole field should strive for the establishment of 'best practices,' similar to previous attempts to constitute practicing standards for fMRI (Poldrack et al., 2017) and EEG research (Picton et al., 2000). For example, the "21-word-solution" (Simmons et al., 2012) that is intended to make the post hoc exclusion of conditions and/or participants transparent, could easily be adopted by the field, given the fact that post hoc exclusions may indeed cause serious problems in studies on unconscious processing (Shanks, 2016). Ideally, full transparency of methods should be the norm. Notably, the current diversity between studies will not be completely abolished by such an intervention, but only reduced. The definition of an overarching 'gold standard' would not be feasible, since, for instance, the possibilities to assess awareness are often limited by the particular experimental design.

Moreover, it should be acknowledged that qualitatively distinct phenomena are subsumed under the umbrella terms "consciousness" and "unconsciousness." To increase the understanding of the scope and limits of unconscious processes, however, such a deficient differentiation seems counterproductive. Thus, to reflect and accentuate that research on unconscious processing stems from a variety of experimental settings, a more fine-grained taxonomy of unconscious phenomena seems expedient. For instance, studies in which the processing of visually masked stimuli is investigated could consistently label "unconscious processing" as "subliminal processing" (Dehaene et al., 2006). In contrast, if supraliminal information is processed unbeknownst to the observer in an incidental manner, as in the case of "unconscious learning processes," one could label this as "implicit processing."

The choice of such terms should also be tailored to their underlying assumptions, as in the case of "subliminal" for instance, which could suggest a high-threshold model. Thus, other labels may be more suitable. Especially to keep up with the development of new techniques, methods, and experimental designs, however, such a differentiation is generally advisable.

Finally, we believe that a great service to this field will be done by an adherence to openly available data, so that the results can be reproduced, and the conclusions tested against alternative statistical analyses.

## REFERENCES


# AUTHOR CONTRIBUTIONS

MR and GH: Conception of the manuscript, Preparation of a first draft, Contribution of critical input and finalization of the manuscript.

# FUNDING

MR and GH were supported by the German Research Foundation (grants RO 4836/2-1 & HE 6244/1-2).


congruency during continuous flash suppression. Psychol. Sci. 27, 945–956. doi: 10.1177/0956797616642525


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Rothkirch and Hesselmann. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.