Commentary: Acetaminophen Enhances the Reflective Learning Process

Pearson et al. (2018) very interesting article “Acetaminophen enhances the reflective learning process.” reports an effect of acetaminophen on reflective processes. By claiming to demonstrate that acetaminophen increases reflective processing, this article brings to mind exciting sci-fiesque musings about “smart pills,” and even more realistic excitement about an applied use of acetaminophen for pilots, surgeons, and students (or perhaps politicians!). This article is promising and investigates important questions about the connection between physical and cognitive processes which entails even more important implications. Unfortunately, I think this article suffers from several flaws and that Pearson et al. have greatly overstated the findings and the implications. I detail five critiques below.


THE HYPOTHESES ARE NOT STATISTICALLY SUPPORTED
The two hypotheses Pearson et al. test (overall accuracy: p = 0.07; trials to criterion: p = 0.06) are not below conventional levels of statistical significance (e.g., p < 0.05). Recent research has demonstrated that p-values close to the arbitrary 0.05 threshold are unlikely to occur repeatedly and, thus, are less likely to replicate (Simonsohn et al., 2014a,b;Benjamin et al., 2018;Camerer et al., 2018). Instead of seeking stronger evidence, the authors go on to examine interactions, and conduct exploratory tests which do not provide much stronger evidence for the claims.
While I commend the authors for clearly distinguishing between exploratory and confirmatory tests, these exploratory tests should be followed up with a clear replication of the effects in an independent dataset (Forstmeier et al., 2017). To be sure, there are ways to overcome unlikely pvalues and "small" effect sizes-for example, using a well-powered sample, preregistering the initial study, or including a preregistered study directly replicating the effect-but this research does not demonstrate those qualities.
Finally, a particularly astute reviewer also pointed out an additional concern regarding the statistical reporting: the effect sizes are reported incorrectly. Indeed, eta-squared can be calculated from the information reported in the article using the following formula (Cohen, 1965):

THE FOLLOW-UP TESTS ARE UNWARRANTED
Despite the non-significance of the key interactions, the authors conduct follow-up tests of simple effects which appear to be the tests on which they rest their main claims. This is inappropriate. Further, it appears that the first test on "influence of acetaminophen on reaching criterion" (p. 1,032) showed no significant results, so the authors recoded the criterion variable dichotomously (yes/no) and reconducted the analysis, interpreting it as supporting their hypothesis (p = 0.049).

THE SAMPLE SIZE IS UNJUSTIFIABLY SMALL
There is no discussion of power or expected effect size.
Recommendations have been made to increase sample sizes to at least 100 per cell (Simmons et al., 2018) to reduce false positives. Given the recent literature on issues of replicability in the social sciences (e.g., Open Science Collaboration, 2015; Forstmeier et al., 2017;Camerer et al., 2018), the authors should justify such a small sample in a between-subjects design. Better yet, a direct replication of these effects with a larger sample would yield more convincing evidence.

THE AUTHORS MAKE USE OF MANY RESEARCHER DEGREES OF FREEDOM WHICH UNDERMINE THE CREDIBILITY OF THEIR RESULTS
The authors utilize at least 11 researcher degrees of freedom (Wicherts et al., 2016), which are arbitrary decisions made by researchers during various phases of research, all of which increase the likelihood of false positives. These instances are outlined in Table 1. For example, participants were excluded for multiple reasons, including for performance on the main DV. We are left to wonder whether including these participants influences the results in any way. Further, the DVs were recoded in multiple ways, including dichotomizing a nonsignificant continuous score (trials to criterion) and testing an arbitrary subset of trials (i.e., accuracy prior to the first rule switch), further increasing the likelihood of a false positive result (Simmons et al., 2011).

THE IMPLICATIONS OF THE RESEARCH ARE SEVERELY OVERSTATED
Given the above limitations, the implications of these findings are greatly overstated. For example, on page 1,033 in the first full paragraph, the authors state "To the degree that people who take acetaminophen are trying to make decisions with clear, logical rules, their performance may improve." In the same paragraph, the authors also state ". . . acetaminophen could potentially help people make difficult decisions by reducing emotional responses to affective contexts while at the same time facilitating more deliberative, effortful information processing . . . " The present results do not show that acetaminophen facilitates deliberative processing, nor does the study investigate emotions in any way. In the final paragraph on page 1,033, the authors conclude "We found that reflective-optimal decision-making can be enhanced by acetaminophen."

FINAL COMMENTS
In summary, while the results seem exciting at first glance, there are several limitations of these results and the implications are overstated. This is an interesting and very important area of research to be sure, but this is precisely the reason why research on this topic needs to be rigorous and accurate. The average reader or lay-person may read this article and leave believing some effect to be true when, in fact, the article offers very weak evidence for this effect. It would be extremely unfortunate if a person began taking acetaminophen believing it to improve learning. Indeed, as a reviewer pointed out, consumers may reason that, because taking the amount described in the article (1,000 mg) makes one somewhat smarter, then taking 10 times that amount (10 g) may make one ten times as smart. This is a large possible ethical consequence which seems to have not been considered. In short, caution is urged when interpreting these research findings, especially if they are to be used to inform interventions or as a basis for future research.

AUTHOR CONTRIBUTIONS
The author confirms being the sole contributor of this work and has approved it for publication.