Statistics for Evaluating Pre-post Change: Relation Between Change in the Distribution Center and Change in the Individual Scores

In a number of scientific fields, researchers need to assess whether a variable has changed between two time points. Average-based change statistics (ABC) such as Cohen's d or Hays' ω2 evaluate the change in the distributions' center, whereas Individual-based change statistics (IBC) such as the Standardized Individual Difference or the Reliable Change Index evaluate whether each case in the sample experienced a reliable change. Through an extensive simulation study we show that, contrary to what previous studies have speculated, ABC and IBC statistics are closely related. The relation can be assumed to be linear, and was found regardless of sample size, pre-post correlation, and shape of the scores' distribution, both in single group designs and in experimental designs with a control group. We encourage other researchers to use IBC statistics to evaluate their effect sizes because: (a) they allow the identification of cases that changed reliably; (b) they facilitate the interpretation and communication of results; and (c) they provide a straightforward evaluation of the magnitude of empirical effects while avoiding the problems of arbitrary general cutoffs.


APPENDIX 1 Alternative average-based effect sizes and their relation to individual-based statistics
A1.1. Single group pre-post design: Computation of Cohen's d using the standard deviation of the pre scores as standardizer For a pre-post design, effect size is usually estimated as the standardized mean of the pre-post differences (Cohen, 1988) but standardization can be computed in two different ways: a) the mean of the pre-post differences (dif) is divided by the standard deviation of pre scores (pre), or b) it is divided by the standard deviation of pre-post differences (dif).
Sometimes, researchers prefer to use pre because such variability of the pre scores is a natural reference for thinking about original scores, as opposed to the variability of the differences (dif) (Cohen, 1988;Cumming & Finch, 2001). Importantly, using a different standardizer for δ does not alter the findings in this study: the strong relation between ABC and IBC holds, regardless of whether we use δ1 = dif /dif or δ2 = dif /pre as the ABC.
The main consequence of using δ2 is the fact that it leads to different slope coefficients (B1) for different values of pre-post correlation -or standard deviation of the differences. In other words, the regression equation describing the relation -and its predictions-vary as a function of the pre-post correlation. This is represented in Figure   For this design, we have used ω 2 as our effect size estimate. This is an "r-type" statistic based on the percentage of variance explained by the interaction effect between group and time. However, a pre-post design with a control group also allows the computation of "d-type" effect size measures, based on the standardization of the mean differences. Although r-type and d-type effect estimations are often interchangeable, d-type statistics are more frequent in some research fields.
In a control group pre-post design, the treatment effect can be estimated by a version where M is the mean and Spre is the pooled standard deviation in the pre-test, which can be In Table A.1 we report the coefficient of determination (R 2 ) for the linear, quadratic, cubic and logistic functions when dPPC is used as the independent variable and the net percentage of individual changes as the dependent variable, calculated with SID index and n = 25. As with 2  , the four functions achieve a very good fit.  In Table A  To illustrate this idea, in Table A.3 we report the correspondence between dPPC and the net percentage of individual changes for some values of ρpre-post. Therefore, when using dPPC for estimating the net percentage of changes, it is necessary to correct the estimation incorporating the relationship between pre and post scores. One way to do this is by using 2  to compute the estimates, after transforming dPPC into 2  by: Relationship between dPPC and net benefit based on SID (n = 25 and normal distribution).

Computation of the net percentage of changes using data from a published study
Lowrie, Logan & Ramful (2017) applied a 10 week visual-spatial intervention program for improving spatial reasoning and mathematics performance in 120 students of ages 10-12. They compared the pre-and post-scores with those of 66 control students.
According to their APPENDIX 3

Effect of measurement error on the relation between ABC and IBC
In almost all real scenarios, measurement error is expected in any observed variable.
However, we have not considered measurement error in our study for a simple reason: it has no effect on the relation between ABC and IBC. Figure A.2 illustrates this idea. It represents the relation between ABC and IBC for three different degrees of measurement error. In the left panel, the pre and post variables are measured without error -as is the case in our simulations. In the center and right panels, the percentage of variance in pre and post due to measurement error is 20% and 40% respectively.