ChatGPT-simulated sentence-plausibility in event contexts, with teens, younger and older adults, in fiction and newspaper texts

Pia Knoeferle^*

• Faculty of Language, Literature and Humanities, Humboldt University of Berlin, Berlin, Germany

In 4 pilot studies and two main experiments, Chat-GPT4o / 5 plausibility ratings were simulated from the graphical user interface using written prompts, factorial designs, Latin-Square counterbalanced lists, and N=200 simulated participants per between-participant factor level. In this way, an experiment setup much like for in-laboratory experiments with human participants was simulated. At issue was to what extent the large language model would produce simulations that are close enough to human-based world knowledge to serve as pilot data for human experimentation: LLMs are developing rapidly and if they were sufficiently accurate databases of human world knowledge, this would open up interesting opportunities for empirical research; with their advent we may have the opportunity of accessing a very comprehensive model of world knowledge. This claim was assessed via simulating human plausibility ratings and their variation depending on (i) the presence versus absence of an event description, and (ii) the age of LLM-simulated participants (Pilot 1, Pilot 2, and Experiment 1a), and (iii) LLM-simulated participant expectations of distinct text sources / genres (Experiment 1b). As a baseline, plausibility ratings generated via LLM chat interface were compared against human plausibility ratings reported in prior research. Overall, ChatGPT produced simulated ratings which on average were higher for plausible than implausible sentences and higher when an event description supported the event conveyed by the target sentence. The model also revealed fine-grained differences depending on simulated participant age and context-sentence relations, and genre. These can guide the formulation of testable hypotheses for future research with human participants.