Statistics is not measurement: The inbuilt semantics of psychometric scales and language-based models obscures crucial epistemic differences

Jana Uher^*

• School of Human Sciences, University of Greenwich, London, United Kingdom

This article provides a comprehensive critique of psychology’s overreliance on statistical modelling at the expense of epistemologically grounded measurement processes. It highlights that statistics deals with structural relations in data regardless of what these data represent, whereas measurement establishes traceable empirical relations between the phenomena studied and the data representing information about them. These crucial epistemic differences are elaborated using Rosen’s general model of measurement, involving the coherent modelling of the 1) objects of research, 2) data generation (encoding), 3) formal manipulation (e.g. statistical analysis) and 4) result interpretation regarding the objects studied (decoding). This system of interrelated modelling relations is shown to underlie metrologists’ approaches for tackling the problem of epistemic circularity in physical measurement, illustrated in the special cases of measurement coordination and calibration. The article then explicates psychology’s challenges for establishing genuine analogues of measurement, which arise from the peculiarities of its study phenomena (e.g., higher-order complexity, non-ergodicity) and language-based methods (e.g., inbuilt semantics). It demonstrates that psychometrics cannot establish coordinated and calibrated modelling relations, thus generating only pragmatic quantifications with predictive power but precluding epistemically justified inferences on the phenomena studied. This epistemic gap is often overlooked, however, because many psychologists mistake their methods’ inbuilt semantics—thus, descriptions of their study phenomena (e.g., in rating scales, item variables, statistical models)—for the phenomena described. This blurs the epistemically necessary distinction between the phenomena studied and those used as means of investigation, thereby confusing ontological with epistemological concepts—psychologists’ cardinal error. Therefore, many mistake judgements of verbal statements for measurements of the phenomena described and overlook that statistics can neither establish nor analyze a model’s relations to the phenomena explored. The article elaborates epistemological and methodological foundations to establish coherent modelling relations between real and formal study system and to distinguish the epistemic components involved, considering psychology’s peculiarities. It shows that epistemically justified inferences necessitate methods for analyzing individuals’ unrestricted verbal responses, now advanced through artificial intelligence systems modelling natural language (e.g., NLP-algorithms, LLMs). Their increasing use to generate standardised descriptions of study phenomena for rating scales and constructs, by contrast, will only perpetuate psychologists’ cardinal error—and thus, psychology’s crisis.