Antonino Vallesi1,2*- 1Department of Neuroscience, Università degli Studi di Padova, Padova, Italy
- 2Padova Neuroscience Center, Università degli Studi di Padova, Padova, Italy
1 Introduction: replicability crisis
In this Editorial Grand Challenge, I will first summarize some of the multiple (intrinsic and extrinsic) reasons why cognitive psychology, along with many other scientific fields, has experienced a replicability crisis. I will then illustrate some of the proposed remedies that are leading to a credibility revolution. I will highlight that the proposed best practices for complex epistemological problems, while necessary, also need to be critically evaluated to ward off potentially new concerns. Finally, I will advocate for a new balance to ensure methodologically sound and credible science while also safeguarding scientific sustainability, creativity and enthusiasm.
In the history of psychology, periods of vibrant discovery and exciting research have occasionally been perturbed by troubling waves of replicability crisis, the most recent one of which is still ongoing (Lakens, 2023; Open Science Collaboration, 2015). While cognitive psychology appears to be slightly less affected by the replicability crisis than social psychology (Open Science Collaboration, 2015), it has certainly not been immune to similar concerns.
First, publications with small sample sizes and inadequate statistical power often led to findings that could not always be reliably reproduced (Button et al., 2013). Second, the pressure to publish novel and positive results, coupled with distorting incentives, has fostered questionable research practices (or even fraud, in the worst cases), such as for instance selective reporting and p-hacking, in which some researchers may, more or less deliberately, manipulate or select data to achieve statistically significant outcomes (e.g., Lilienfeld, 2017; Nosek et al., 2012). Additionally, complex experimental designs, researchers' degrees of freedom and variability in methodological practices have contributed to inconsistencies across studies. The prevalence of a confirmation bias, that is, the tendency of researchers to favor data that support their hypotheses, has further exacerbated this issue. Moreover, a lack of transparency in data sharing and reporting of methodological details has hindered efforts to accurately replicate findings.
These challenges are currently prompting a massive cultural paradigm shift that would imply entirely rethinking the research process. It is not surprising that a growing movement, largely and meritoriously driven by the new generation of researchers, is advocating for more rigorous and transparent scientific practices. This shift is a virtuous reaction to the replicability crisis (e.g., Lindsay, 2020). These practices include for instance, (i) registering reports or at least pre-registering studies to minimize questionable practices such as Harking (Kerr, 1998), cherry-picking and confirmation bias (e.g., Clark et al., 2022a), (ii) implementing open data initiatives to increase transparency and maximizing collective scientific efforts (e.g., Mayernik, 2017; Rasti et al., 2025), and (iii) conducting replication efforts through coordinated multi-laboratory projects to achieve the large sample sizes necessary for sufficient statistical power. Therefore, awareness of the replicability crisis is currently driving positive and praiseworthy changes in the methods, procedures and, importantly, policies that will eventually and ideally benefit the credibility of the scientific community as a whole (e.g., Korbmacher et al., 2023). However, as I will argue throughout this contribution, one would be naïve to think that the proposed solutions to these complex epistemological problems are always straightforward and without their own drawbacks. Any new remedy needs to be critically evaluated and a new balance sought.
2 Multiple contributions to the replicability crisis
As a first consideration, it is undeniable that the replicability crisis can be, at least partially, attributed to a series of questionable research practices that have been commonly used, tolerated, or even favored by standard practices in the field and by ill-conceived academic incentives. However, we should not overlook other potential factors that may also have contributed to the replicability crisis in our specific field. Indeed, replicability issues may also stem, in part, from deeper epistemological and methodological problems.
The social sciences, and cognitive psychology is no exception, are concerned with objects of investigation that cannot be observed directly but are captured primarily through latent variables and constructs. These constructs require careful, and often challenging, operationalization in order to be scientifically attacked through some measurable proxies. Because cognitive processes are inherently “fuzzier” and more ambiguous compared to more basic biological processes, there is typically no single best operationalization of a given construct to test, and ontological definitions require additional effort. To take an example from cognitive psychology used by Poldrack and Yarkoni (2016), performance on N-back tasks and OSPAN tasks shows weak, and often non-significant, correlations (Kane et al., 2007; Oberauer, 2005; Roberts and Gibson, 2002). This is unsettling as both tasks are designed to assess the same cognitive construct, that is, working memory. Such discrepancies suggest issues such as ambiguous operational definitions and task impurity (Kane et al., 2007), which contribute to the entropy in the literature. Some promising remedies have been proposed. These include the Cognitive Atlas Project (Poldrack and Yarkoni, 2016), a collaborative project to develop a knowledge base that represents the current state of cognitive science, or the Cognitive Paradigm Ontology (Turner and Laird, 2012), a systematic collection of cognitive paradigms for functional neuroimaging applications. While these tools are commendable, there is a risk that only global inferences can be made. This would happen if the tasks included in these databases were too general and lacked subtask-specific manipulations. Indeed, despite their methodological idiosyncrasies, subtle experimental manipulations are often more theoretically meaningful than the common features across tasks.
Additionally, the findings obtained need to be interpreted, with more than one account often being equally plausible at the same time, thus lowering internal validity. This produces uncertainty about which interpretation of the findings might be the most accurate, increasing the likelihood that biased conclusions will be preferred instead. A related consequent issue is the so-called motivated research, which occurs when factors unrelated to accuracy influence how scientists engage with existing data, generate new hypotheses, collect, analyze and interpret data, and communicate research outcomes to their peers and the general audience (e.g., Clark et al., 2022b; Kunda, 1990). These factors include confirmation biases and the need for social and moral approval of specific research topics and conclusions.
Furthermore, delving deeper into the core of the issue, the inherent variability of the object of investigation is another major challenge in the social sciences, which also include cognitive psychology. The manifestation of cognitive processes can indeed vary considerably, not only between individuals but also within individuals (e.g., Dhawale et al., 2017; Fiske and Rice, 1955; Kahana et al., 2018; MacPherson et al., 2019), with variability being a key aspect of cognition that cannot be dismissed (or at least not exclusively) as a measurement error (Judd et al., 2024). Indeed, variability is not only spuriously caused by ill-defined operationalizations of cognitive constructs, imprecise measurements of their proxies (e.g., behavior), violations of statistical assumptions, power issues, and more generally, poor experimental control and other methodological flaws (e.g., Clark and Watson, 2019; Podsakoff et al., 2012). There are also other fundamental sources of variability linked to the very nature of mental processes.
Natural fluctuations characterize the manifestations of mental functions such as perception, emotion, memory, executive functions, and decision-making, to name a few. This variability arises from numerous factors, which include psychobiological variables, genetic differences, developmental stages, but also situational contexts and environmental influences (e.g., epigenetics).
Specifically, cognitive performance can be affected by an individual's mood (Chepenik et al., 2007), motivation (Braver et al., 2014), stress (Steinhauser et al., 2007), sleep deprivation (Killgore, 2010), and prior experience (Pastukhov and Braun, 2011). Moreover, the brain's complex neural architecture and the dynamic nature of its networks contribute to this variability, as its connectome is constantly shaped and reshaped by new experiences, providing the neural foundation for an ever-changing cognitive life (e.g., Dosenbach et al., 2010; Kolb and Gibb, 2011; Seung, 2012; Tost et al., 2015). Acknowledging, and ideally accounting for, this intrinsic variability and lability of the object of investigation in cognitive psychology (i.e., mental processes) is essential for developing robust, or at least plausible, theories and applications in cognitive psychology and neuroscience.
Additionally, many findings cannot be easily generalized across different contexts (e.g., cultural, spatial, temporal) or experimental conditions (Stroebe and Strack, 2014). Indeed, the environmental or experimental context can affect both how and to what extent a cognitive process is activated and expressed. An example is provided in the work by Hommel et al. (2012). Their study showed that cognitive control can be, somewhat counterintuitively, more effective when exercised in the context of the acoustic noise typical of fMRI sequences than in standard settings. This specific example raises more general considerations about the potential applicability of Heisenberg's uncertainty principle in cognitive neuroscience, where the object of investigation (the mind) is being transformed by the very tools used to study it. A more classic example of this issue is the Hawthorne effect, in which individuals alter their behavior because they are aware of being observed and measured (e.g., Adair, 1984). Although, ironically enough, this famous phenomenon is probably not fully reliable, or at least not in its initial reports (Letrud and Hernes, 2019; Levitt and List, 2011), it represents just one example of the broader (but often neglected) modulatory role of participants' motivation and demand characteristics in task performance (e.g., Orne, 1962; Weber and Cook, 1972).
To summarize this section, while variability can be seen as an obstacle to scientific progress, as it can lead to average null effects, uncovering its intrinsic (but also exogenous) sources can also have a leveraging effect on theory development. Indeed, variability could be exploited to better understand inter-individual differences in cognition and to advance cognitive theories (e.g., Langerock et al., 2025; Miller and Schwarz, 2017; Wang et al., 2012; cf., Rowe and Healy, 2014).
3 Remedies for the replicability crisis and new concerns of the credibility revolution
One potential concern with the proposed remedies for the replicability crisis is that, while highly desirable in principle, some of them conceal other risks that should at least be acknowledged, if not actively countered. As the Latin writer Horace once wrote: “Est modus in rebus”, which reminds us to pursue balance in life by avoiding deleterious extremes. As will become clear from the following, the coveted solutions proposed to overcome the replicability crisis are also subject to this caveat. Therefore, we need to avoid applying them uncritically.
3.1 Data-driven approaches in cognitive psychology: potential risks and antidotes
While the widespread availability of large open datasets promotes reproducibility and collective efficiency, it also inevitably encourages data mining and exploratory approaches. Far be it from me to want to unleash or incite a witch hunt, but such a data-driven mindset (which could bring to a so-called fishing expedition attitude) can sometimes temptingly lead to the post hoc generation of hypotheses (HARKing) and, even worse, to a lack of theoretical depth. A data-driven attitude is also encouraged by the unprecedented, exponential expansion of Machine Learning and other Artificial Intelligence (AI) algorithms. AI can undoubtedly enhance the efficient exploration and analysis of complex scientific data in a number of ways, such as identifying patterns and trends, allowing predictive modeling, integrating multidimensional data and so on, accelerating data-driven discoveries including in the field of cognitive psychology. However, would powerful but opaque AI algorithms always help enhance our theoretical understanding of underlying cognitive mechanisms? And, in case this is achievable, could this AI-based understanding then generalize to new datasets? Although the debate in the field is currently heated with these and other open questions, it would definitely benefit from a more extensive and deeper epistemological reflection, hopefully with contributions also coming from cognitive psychology.
One proposed solution to the risks of a lenient data-driven attitude is the pre-registration of studies, which forces researchers to truly specify their a priori hypotheses in advance (and to clearly label as exploratory all the other hypotheses that are generated a posteriori with respect to the pre-registration). However, these and other credibility-enhancing practices are still adopted on a voluntary basis and are not yet routinely applied as gold standards. Additionally, pre-registration alone (as a mere bureaucratic formality) is not sufficient. It would be equally important to address the deeper issue of the theoretical crisis: the hypotheses that are pre-registered could still be purely empirical in nature or, even worse, poorly formulated, unjustified and without strong theoretical foundations. Thus, pre-registering hypotheses helps to reassure the scientific community that our thinking remains unbiased but it does not in itself substitute theoretical and logic-based reasoning. Simply pre-registering a hypothesis, as a sort of fig leaf, without substantiating it with a solid theoretical rationale serves little purpose and does not in itself enhance the credibility of the hypothesis (Oberauer and Lewandowsky, 2019). Moreover, it would be fine to candidly admit that a given study was exploratory, especially if at the beginning of a new line of research, as long as the data obtained could inspire theory development, new hypotheses, and further research to test those hypotheses and refine the theory.
An excessive focus on trying to empirically reproduce or replicate published effects or phenomena, while extremely valuable to reshape the trust and the foundation of what we already know about human cognition, if an end in itself, can divert attention away from developing robust theories (Oberauer and Lewandowsky, 2019). Proposing strong theories would allow scholars to formulate credible and specific hypotheses, which would benefit scientific progress (Clark et al., 2022b). Developing accurate and comprehensive theories or at least models of cognitive processes is inherently challenging due to their fuzzy nature and complexity. Thus, theories and models need to consider a broad spectrum of variables and interactions, which necessitates advanced computational approaches, biological plausibility and extensive experimental testing (e.g., Busemeyer and Diederich, 2010; Turner et al., 2017). This process could benefit from collective, multidisciplinary scientific coordination toward common, challenging goals (Brignol et al., 2024; Rasti et al., 2025).
One special type of scientific coordination that has been proposed as a remedy for the lack of a theory-oriented mindset is represented by adversarial collaborations (sometimes referred to as “coopetition”; Clark et al., 2022a; Ellemers et al., 2020). Although challenging, engaging with those who propose alternative theories can be scientifically rewarding in the long run (see Cowan et al., 2020; Mellers et al., 2001, for some examples). These collaborations require clear, testable hypotheses, mutual understanding of differing viewpoints, and methods that can distinguish and potentially falsify both hypotheses. This process limits researchers' biases and beliefs, enhances the integrity of methodological approaches, and effectively advances debates. By committing to methods before testing, even better when coupled with best practices such as pre-registering, open data and replication, scholars reduce the risk of post-hoc criticism or hiding unwanted results. This would also make the findings more credible to the scientific community. More plausible hypotheses will prevail, and incorrect ones will be falsified sooner, preventing wasted time and resources and increasing the reliability and credibility of scientific findings.
Relying on increasingly sophisticated methods to analyze rich and complex data poses similar risks. The field of neuroimaging can be used as an example. While advances in neuroimaging data analysis have provided valuable insights into brain activity, understanding how these findings are related to cognitive processes is still difficult. Explaining how complex mental functions arise from brain regions and neural networks has traditionally been a major challenge (e.g., Coltheart, 2006; Hommel and Colzato, 2017; Niv, 2021). More specifically, while recent advances in the field of connectomics in cognitive neuroscience (e.g., graph theory) have significantly enhanced our ability to explain the complexity of brain organization, at least in principle, clear relationships between topological indices of network integration and segregation and cognitive processes have yet to be fully established (Litwińczuk et al., 2023). These relationships would be crucial for assessing the biological plausibility and behavioral relevance of these highly derived, network-based measures.
Complicating matters, it is acknowledged in the neuroimaging literature that behavioral data are not always characterized with sufficient fidelity to support robust and reliable brain-behavior correlations (Tiego and Fornito, 2022). Additionally, some common interpretations of cognitive neuroscience findings are unjustified because they rely on flawed reasoning or other unproven assumptions. A well-known example of flawed reasoning is reverse inference, that is, deducing the involvement of particular mental processes solely based on observed patterns of brain activation, without the constraints of elegant experimental manipulations that would enhance internal validity (Kriegeskorte et al., 2009; Poldrack, 2006). As another example, many compensatory cognitive interpretations of brain patterns in older adults are not fully justified unless data are analyzed longitudinally (e.g., Pudas et al., 2013).
Another potential drawback of widely available open data is the inherent risk of salami slicing, in which data from a research study that should form a single publication are divided into several smaller publishable units. Salami slicing, which has long been considered a questionable and unethical research practice in various fields (Karlsson and Beaufils, 2013; Siegel and Baveye, 2010; Spielmans et al., 2010) remains a significant concern, and even more so in the era of open science. This practice can lead to various issues, such as consuming excessive resources during the editorial and review process and overwhelming science readers. It can also lead to an inflated number of articles cluttering the literature, increasing the risk of publication bias, and distorting effect sizes in meta-analyses. To overcome this issue, the same or overlapping data samples should always be properly acknowledged as such in different publications (e.g., Hilgard et al., 2019; Urbanowicz and Reinke, 2018).
Furthermore, it is also worth mentioning that the risk of wasting resources and producing flawed science is amplified if the tasks used to collect behavioral data in large open datasets were not carefully designed and suffer from some error.
3.2 Potential side effects of the credibility revolution and excessive methodological control
A well-known trade-off exists between experimental control and generalizability. On the one hand, more attention to well-designed experimental paradigms that account for confounding factors when trying to fully characterize a psychological effect is desirable to increase internal validity (e.g., Verbruggen et al., 2019; Viviani et al., 2024), and hopefully also to enhance scientific credibility and replicability. On the other hand, excessive methodological control in research poses a significant risk, as it can lead to a decrease in generalizability, that is, external and ecological validity (Shadish et al., 2002). In other words, the more tightly controlled experiments become, the less accurately their findings reflect real-world conditions, making it difficult to apply the results to broader contexts or diverse populations. This undermines the relevance and applicability of the research, limiting its potential impact outside the highly controlled environment in which it was conducted.
Another potential risk of placing too much emphasis on the remedies for the replicability crisis is a reduction in the enthusiasm for exploration, creativity and discovery, which are also essential aspects of the scientific process that complement methodological rigor. Focusing too much on strict standards, while being undeniably vital for overcoming the pressing credibility crisis, may also lead to excessive caution, hindering divergent thinking and the serendipitous attitude that drives new discoveries (Kaufman and Glǎveanu, 2018; Ness, 2015). In principle, the majority of researchers would support recommendations, best practices, and, more generally, to virtuous organized skepticism, in the hope that the scientific community would universally adopt them (Anderson et al., 2007). However, when these begin to become imposed norms, a looming risk is that they may lead to additional paperwork in fulfilling the already hypertrophic and self-referential accountability and evaluation processes. The latter require additional time and mental energy, limited resources that researchers would prefer to devote, with much more motivation and enthusiasm, to their actual research activities. If new practices become excessively demanding due to an increased bureaucratic burden, they risk reducing the time and resources available for creative activities (Kaufman and Glǎveanu, 2018). In other words, with excessive constraints, creativity could also be suppressed (Medeiros et al., 2014). This risk within the scientific field should be considered in addition to the already observed decline in creativity in the general population (Kim, 2011).
In other words, while best practices in science are indispensable and non-negotiable, the burdensome bureaucracy and other possible downsides that might accompany their implementation should be kept strictly in check, and new incentives should be envisioned that could promote slower but more accurate science and allow for more quality time. Scientists are therefore compelled to closely monitor the tendency toward hyper-bureaucratization of the scientific process and to continually seek a new balance between the apparently conflicting, yet equally essential needs of methodological rigor and scientific creativity (Ness, 2015; Scheffer, 2014; Simonton, 2004), while moving away from purely quantity-based institutional evaluation processes and incentives.
4 Concluding remarks
The awareness of a massive replicability crisis has been a slap in the face for the scientific community. It is undeniable that we can no longer continue with business as usual and that remedies to address this crisis must be implemented. However, as I have tried to argue here, this cannot be done naively or uncritically, as any new practice could also bring its own drawbacks, including changes in how we value scientific research. It has been shown, for instance, that awareness of replication failures and criticism of questionable research practices can diminish trust in past research (Anvari and Lakens, 2018). Worse, being informed about potential solutions, such as increased transparency, may be, somewhat paradoxically, associated with reduced trust in future research (Anvari and Lakens, 2018). This highlights a significant issue with the trustworthiness of research, which is exacerbated by a lack of enthusiasm.
In conclusion, as discussed in this contribution, the proposed remedies aimed to foster the credibility revolution are highly desirable and non-negotiable. However, these remedies (along with others yet to be envisaged) cannot, sic et simpliciter, be seen as a panacea for the replicability crisis without first being critically evaluated for their own hidden risks or costs. To avoid throwing the baby out with the bathwater, the scientific community desperately needs to find a new balance in the scientific process between imperative methodological rigor and best practices on the one hand and enthusiastic theory-advancing creativity and sustainability on the other. Both types of needs require more quality time for researchers, free from ill-conceived (and only apparently objective) institutional academic incentives and excessive bureaucratic burdens. We need to fine-tune best practices that ensure methodological thoroughness, without renouncing greasing the wheels of scientific creativity, while minimizing the overproduction of non-replicable results. Although I am certainly not the first to propose this sort of call to action, I hope that our journal, Frontiers in Psychology: Cognition, can serve (through specific Research Topics or other formats) as an inspiring receptacle for new ways of conceiving the entire scientific process in the pursuit of this new balance, free of any chauvinistic preconceptions.
Author contributions
AV: Conceptualization, Writing – original draft, Writing – review & editing.
Funding
The author(s) declare that financial support was received for the research and/or publication of this article. AV was partially supported by the Italian Ministry of Health (project code: GR-2018-12367927, FINAGE), by the Italian Ministry of University and Research (MUR), and by the European Union in the framework of PNRR NextGenerationEU with projects: MNESYS (CUP #B83C22004960002; PE0000006) and ORCA (CUP #J93C24000190007; PE0000013).
Acknowledgments
Part of this work was inspired by fruitful discussions with AV and other esteemed colleagues.
Conflict of interest
The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.
Generative AI statement
The author(s) declare that Gen AI was used in the creation of this manuscript. ChatGPT (OpenAI, 2024) was occasionally used for language editing purposes.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Adair, J. G. (1984). The Hawthorne effect: a reconsideration of the methodological artifact. J. Appl. Psychol. 69, 334–345. doi: 10.1037//0021-9010.69.2.334
Anderson, M. S., Martinson, B. C., and De Vries, R. (2007). Normative dissonance in science: results from a national survey of U.S. Scientists. J. Empir. Res. Human Res. Ethics 2, 3–14. doi: 10.1525/jer.2007.2.4.3
Anvari, F., and Lakens, D. (2018). The replicability crisis and public trust in psychological science. Compreh. Results Soc. Psychol. 3, 266–286. doi: 10.1080/23743603.2019.1684822
Braver, T. S., Krug, M. K., Chiew, K. S., Kool, W., Westbrook, J. A., Clement, N. J., et al. (2014). Mechanisms of motivation–cognition interaction: challenges and opportunities. Cognit. Affect. Behav. Neurosci. 14, 443–472. doi: 10.3758/s13415-014-0300-0
Brignol, A., Paas, A., Sotelo-Castro, L., St-Onge, D., Beltrame, G., and Coffey, E. B. J. (2024). Overcoming boundaries: interdisciplinary challenges and opportunities in cognitive neuroscience. Neuropsychologia 200, 108903. doi: 10.1016/j.neuropsychologia.2024.108903
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376. doi: 10.1038/nrn3475
Chepenik, L. G., Cornew, L. A., and Farah, M. J. (2007). The influence of sad mood on cognition. Emotion 7, 802–811. doi: 10.1037/1528-3542.7.4.802
Clark, C. J., Costello, T., Mitchell, G., and Tetlock, P. E. (2022a). Keep your enemies close: adversarial collaborations will improve behavioral science. J. Appl. Res. Mem. Cogn. 11, 1–18. doi: 10.1037/mac0000004
Clark, C. J., Honeycutt, N., and Jussim, L. (2022b). “Replicability and the psychology of science,” in Avoiding Questionable Research Practices in Applied Psychology (Cham: Springer Nature Switzerland AG), 45–71
Clark, L. A., and Watson, D. (2019). Constructing validity: New developments in creating objective measuring instruments. Psychol. Assess. 31, 1412–1427. doi: 10.1037/pas0000626
Coltheart, M. (2006). What has functional neuroimaging told us about the mind (so far)? Cortex 42, 323–331. doi: 10.1016/S0010-9452(08)70358-7
Cowan, N., Belletier, C., Doherty, J. M., Jaroslawska, A. J., Rhodes, S., Forsberg, A., et al. (2020). How do scientific views change? Notes from an extended adversarial collaboration. Perspect. Psychol. Sci. 15, 1011–1025. doi: 10.1177/1745691620906415
Dhawale, A. K., Smith, M. A., and Ölveczky, B. P. (2017). The role of variability in motor learning. Annual Rev. Neurosci. 40, 479–498. doi: 10.1146/annurev-neuro-072116-031548
Dosenbach, N. U. F., Nardos, B., Cohen, A. L., Fair, D. A., Power, J. D., Church, J. A., et al. (2010). Prediction of individual brain maturity using fMRI. Science 329, 1358–1361. doi: 10.1126/science.1194144
Ellemers, N., Fiske, S. T., Abele, A. E., Koch, A., and Yzerbyt, V. (2020). Adversarial alignment enables competing models to engage in cooperative theory building toward cumulative science. Proc. Nat. Acad. Sci. 117, 7561–7567. doi: 10.1073/pnas.1906720117
Fiske, D. W., and Rice, L. (1955). Intra-individual response variability. Psychol. Bull. 52, 217–250. doi: 10.1037/h0045276
Hilgard, J., Sala, G., Boot, W. R., and Simons, D. J. (2019). Overestimation of action-game training effects: publication bias and salami slicing. Collabra: Psychol. 5:1. doi: 10.1525/collabra.231
Hommel, B., and Colzato, L. S. (2017). The grand challenge: integrating nomothetic and ideographic approaches to human cognition. Front. Psychol. 8:100. doi: 10.3389/fpsyg.2017.00100
Hommel, B., Fischer, R., Colzato, L. S., van den Wildenberg, W. P. M., and Cellini, C. (2012). The effect of fMRI (noise) on cognitive control. J. Exp. Psychol. Hum. Percept. Perform. 38, 290–301. doi: 10.1037/a0026353
Judd, N., Aristodemou, M., Klingberg, T., and Kievit, R. (2024). Interindividual differences in cognitive variability are ubiquitous and distinct from mean performance in a battery of eleven tasks. J. Cognit. 7:371. doi: 10.5334/joc.371
Kahana, M. J., Aggarwal, E. V., and Phan, T. D. (2018). The variability puzzle in human memory. J. Exp. Psychol. Learn. Mem. Cogn. 44, 1857–1863. doi: 10.1037/xlm0000553
Kane, M. J., Conway, A. R. A., Miura, T. K., and Colflesh, G. J. H. (2007). Working memory, attention control, and the N-back task: a question of construct validity. J. Exp. Psychol. Learn. Mem. Cogn. 33, 615–622. doi: 10.1037/0278-7393.33.3.615
Karlsson, J., and Beaufils, P. (2013). Legitimate division of large data sets, salami slicing and dual publication, where does a fraud begin? Knee Surg Sports Traumatol. Arthrosc. 21, 751–752. doi: 10.1007/s00167-013-2413-3
Kaufman, J. C., and Glǎveanu, V. P. (2018). The road to uncreative science is paved with good intentions: ideas, implementations, and uneasy balances. Perspect. Psychol. Sci. 13, 457–465. doi: 10.1177/1745691617753947
Kerr, N. L. (1998). HARKing: hypothesizing after the results are known. Personal. Soc. Psychol. Rev. 2, 196–217. doi: 10.1207/s15327957pspr0203_4
Killgore, W. D. S. (2010). “Effects of sleep deprivation on cognition,” in Progress in Brain Research, eds. G. A. Kerkhof & H. P. A. van Dongen (London: Elsevier), 105–129.
Kim, K. H. (2011). The creativity crisis: the decrease in creative thinking scores on the torrance tests of creative thinking. Creat. Res. J. 23, 285–295. doi: 10.1080/10400419.2011.627805
Kolb, B., and Gibb, R. (2011). Brain plasticity and behaviour in the developing brain. J. Canadian Acad. Child Adolescent Psychiat. 20, 265–276.
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., et al. (2023). The replication crisis has led to positive structural, procedural, and community changes. Commun. Psychol. 1, 1–13. doi: 10.1038/s44271-023-00003-2
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S., and Baker, C. I. (2009). Circular analysis in systems neuroscience – the dangers of double dipping. Nat. Neurosci. 12, 535–540. doi: 10.1038/nn.2303
Kunda, Z. (1990). The case for motivated reasoning. Psychol. Bull. 108, 480–498. doi: 10.1037//0033-2909.108.3.480
Lakens, D. (2023). Concerns about Replicability, Theorizing, Applicability, Generalizability, and Methodology across Two Crises in Social Psychology. Dover, PA: OSF.
Langerock, N., Oberauer, K., Throm, E., and Vergauwe, E. (2025). The cognitive load effect in working memory: refreshing the empirical landscape, removing outdated explanations. J. Mem. Lang. 140:104558. doi: 10.1016/j.jml.2024.104558
Letrud, K., and Hernes, S. (2019). Affirmative citation bias in scientific myth debunking: a three-in-one case study. PLoS ONE 14:e0222213. doi: 10.1371/journal.pone.0222213
Levitt, S. D., and List, J. A. (2011). Was there really a hawthorne effect at the hawthorne plant? An analysis of the original illumination experiments. Am. Econ. J. 3, 224–238. doi: 10.1257/app.3.1.224
Lilienfeld, S. O. (2017). Psychology's replication crisis and the grant culture: righting the ship. Persp. Psychol. Sci. 12, 660–664. doi: 10.1177/1745691616687745
Lindsay, D. S. (2020). Seven steps toward transparency and replicability in psychological science. Can. Psychol. 61, 310–317. doi: 10.1037/cap0000222
Litwińczuk, M. C., Muhlert, N., Trujillo-Barreto, N., and Woollams, A. (2023). Using graph theory as a common language to combine neural structure and function in models of healthy cognitive performance. Hum. Brain Mapp. 44, 3007–3022. doi: 10.1002/hbm.26258
MacPherson, S. E., Gillebert, C. R., Robinson, G. A., and Vallesi, A. (2019). Editorial: Intra- and inter-individual variability of executive functions: determinant and modulating factors in healthy and pathological conditions. Front. Psychol. 10:432. doi: 10.3389/fpsyg.2019.00432
Mayernik, M. S. (2017). Open data: accountability and transparency. Big Data Soc. 4:2. doi: 10.1177/2053951717718853
Medeiros, K. E., Partlow, P. J., and Mumford, M. D. (2014). Not too much, not too little: The influence of constraints on creative problem solving. Psychol. Aesthet. Creat. Arts 8, 198–210. doi: 10.1037/a0036210
Mellers, B., Hertwig, R., and Kahneman, D. (2001). Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration. Psychol. Sci. 12, 269–275. doi: 10.1111/1467-9280.00350
Miller, J., and Schwarz, W. (2017). Implications of individual differences in on-average null effects. J. Exp. Psychol. General 147, 377–397. doi: 10.1037/xge0000367
Ness, R. B. (2015). The Creativity Crisis: Reinventing Science to Unleash Possibility. Oxford: Oxford University Press.
Niv, Y. (2021). The primacy of behavioral research for understanding the brain. Behav. Neurosci. 135, 601–609. doi: 10.1037/bne0000471
Nosek, B. A., Spies, J. R., and Motyl, M. (2012). Scientific Utopia: II. restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615–631. doi: 10.1177/1745691612459058
Oberauer, K. (2005). Binding and inhibition in working memory: individual and age differences in short-term recognition. J. Exp. Psychol. 134, 368–387. doi: 10.1037/0096-3445.134.3.368
Oberauer, K., and Lewandowsky, S. (2019). Addressing the theory crisis in psychology. Psychon. Bull. Rev. 26, 1596–1618. doi: 10.3758/s13423-019-01645-2
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science 349:aac4716. doi: 10.1126/science.aac4716
Orne, M. T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. Am. Psychol. 17, 776–783. doi: 10.1037/h0043424
Pastukhov, A., and Braun, J. (2011). Cumulative history quantifies the role of neural adaptation in multistable perception. J. Vis. 11, 12. doi: 10.1167/11.10.12
Podsakoff, P. M., MacKenzie, S. B., and Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annu. Rev. Psychol. 63, 539–569. doi: 10.1146/annurev-psych-120710-100452
Poldrack, R. A. (2006). Can cognitive processes be inferred from neuroimaging data? Trends Cogn. Sci. 10, 59–63. doi: 10.1016/j.tics.2005.12.004
Poldrack, R. A., and Yarkoni, T. (2016). From brain maps to cognitive ontologies: informatics and the search for mental structure. Annu. Rev. Psychol. 67, 587–612. doi: 10.1146/annurev-psych-122414-033729
Pudas, S., Persson, J., Josefsson, M., de Luna, X., Nilsson, L.-G., and Nyberg, L. (2013). Brain characteristics of individuals resisting age-related cognitive decline over two decades. J. Neurosci. 33, 8668–8677. doi: 10.1523/JNEUROSCI.2900-12.2013
Rasti, S., Vaesen, K., and Lakens, D. (2025). A Framework for Describing the Levels of Scientific Coordination. Dover, PA: OSF.
Roberts, R., and Gibson, E. (2002). Individual differences in sentence memory. J. Psycholinguist. Res. 31, 573–598. doi: 10.1023/A:1021213004302
Rowe, C., and Healy, S. D. (2014). Measuring variation in cognition. Behav. Ecol. 25, 1287–1292. doi: 10.1093/beheco/aru090
Scheffer, M. (2014). The forgotten half of scientific thinking. Proc. Nat. Acad. Sci. 111, 6119–6119. doi: 10.1073/pnas.1404649111
Seung, S. (2012). Connectome: How the Brain's Wiring Makes Us who We are. Boston, MA: Houghton Mifflin Harcourt.
Shadish, W. R., Cook, T. D., and Campbell, D. T. (2002). Experimental and Quasi-experimental Designs for Generalized Causal Inference. Boston, MA: Houghton Mifflin.
Siegel, D., and Baveye, P. (2010). Battling the paper glut. Science 329:1466. doi: 10.1126/science.329.5998.1466-a
Simonton, D. K. (2004). Creativity in Science: Chance, Logic, Genius, and Zeitgeist. Cambridge: Cambridge University Press.
Spielmans, G. I., Biehn, T. L., and Sawrey, D. L. (2010). A case study of salami slicing: pooled analyses of duloxetine for depression. Psychother. Psychosom. 79, 97–106. doi: 10.1159/000270917
Steinhauser, M., Maier, M., and Hübner, R. (2007). Cognitive control under stress: how stress affects strategies of task-set reconfiguration. Psychol. Sci. 18, 540–545. doi: 10.1111/j.1467-9280.2007.01935.x
Stroebe, W., and Strack, F. (2014). The alleged crisis and the illusion of exact replication. Persp. Psychol. Sci. 9, 59–71. doi: 10.1177/1745691613514450
Tiego, J., and Fornito, A. (2022). “Putting behaviour back into brain–behaviour correlation analyses,” in Aperture Neuro, BWAS Editorials (Roseville, MN: Organization for Human Brain Mapping), 1–4.
Tost, H., Champagne, F. A., and Meyer-Lindenberg, A. (2015). Environmental influence in the brain, human welfare and mental health. Nat. Neurosci. 18, 1421–1431. doi: 10.1038/nn.4108
Turner, B. M., Forstmann, B. U., Love, B. C., Palmeri, T. J., and Van Maanen, L. (2017). Approaches to analysis in model-based cognitive neuroscience. J. Math. Psychol. 76, 65–79. doi: 10.1016/j.jmp.2016.01.001
Turner, J. A., and Laird, A. R. (2012). The cognitive paradigm ontology: design and application. Neuroinformatics 10, 57–66. doi: 10.1007/s12021-011-9126-x
Urbanowicz, C., and Reinke, B. A. (2018). Publication overlap: building an academic house with salami shingles. Bullet. Ecol. Soc. Am. 99:e01425. doi: 10.1002/bes2.1425
Verbruggen, F., Aron, A. R., Band, G. P., Beste, C., Bissett, P. G., Brockett, A. T., et al. (2019). A consensus guide to capturing the ability to inhibit actions and impulsive behaviors in the stop-signal task. Elife 8:e46323. doi: 10.7554/eLife.46323
Viviani, G., Visalli, A., Montefinese, M., Vallesi, A., and Ambrosini, E. (2024). The Stroop legacy: A cautionary tale on methodological issues and a proposed spatial solution. Behav. Res. Methods 56, 4758–4785. doi: 10.3758/s13428-023-02215-0
Wang, L., Hamaker, E., and Bergeman, C. S. (2012). Investigating inter-individual differences in short-term intra-individual variability. Psychol. Methods 17, 567–581. doi: 10.1037/a0029317
Keywords: replicability crisis, credibility revolution, scientific discovery, creativity, psychological theories, open science
Citation: Vallesi A (2025) Credibility revolution: pursuing a balanced and sustainable approach, without dogmas, without magic elixirs. Front. Psychol. 16:1581160. doi: 10.3389/fpsyg.2025.1581160
Received: 21 February 2025; Accepted: 28 March 2025;
Published: 16 April 2025.
Edited and reviewed by: Axel Cleeremans, Université libre de Bruxelles, Belgium
Copyright © 2025 Vallesi. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Antonino Vallesi, YW50b25pbm8udmFsbGVzaUB1bmlwZC5pdA==