Commentary: Studying a Possible Placebo Effect of an Imaginary Low-Calorie Diet

While Panayotov reported very strong effects in their study of a “placebo” diet (1), the statistical analysis was performed incorrectly. To evaluate the effect of a treatment in a randomized controlled trial (RCT), the difference between groups should be tested directly rather than basing conclusions on the changes within each group separately. Conclusions in Panayotov’s manuscript are based on what has been called a DINS error or difference in nominal significance (2). A significant change from baseline within the treatment group and a non-significant change from baseline within the control group does not necessarily mean that the treatment effect was significant, i.e. that the treatment group improved significantly more than the control group. Allison and colleagues have identified DINS errors in many papers in the field of nutrition calling for their corrections (3–5). Panayotov’s RCT tests the differential effects of participants believing they are on a low-calorie diet compared to those who know they are on an isocaloric diet (n=7 per group). Outcomes measured were body mass (kg), fat mass (%), and BMI (kg/m) in the treatment and control group at the beginning of the experiment and after 8 weeks of both groups being on an isocaloric diet, where the treatment group were told they were receiving a low calorie diet. The results of the study are based on paired t-tests comparing the outcomes at baseline and 8 weeks, separately within each group. This can make treatment effects appear to be significant where they otherwise would not be. We commend the author for sharing his data at our request, and we re-analyze the data focusing on the between-group differences to isolate the treatment effect. When we conduct t-tests on the differences (post-pre) between groups, conclusions change for one of the three outcomes (Table 1). Panayotov reported a p-value of < .001 for decreased fat mass in the treatment group and nonsignificant change (p=.21) in the control group to conclude that the treatment is better than the control. However, when we compare the decrease in the treatment group to the decrease in the control group with an independent samples t-test, we obtain a non-significant p-value of p=.105.

While Panayotov reported very strong effects in their study of a "placebo" diet (1), the statistical analysis was performed incorrectly. To evaluate the effect of a treatment in a randomized controlled trial (RCT), the difference between groups should be tested directly rather than basing conclusions on the changes within each group separately. Conclusions in Panayotov's manuscript are based on what has been called a DINS error or difference in nominal significance (2). A significant change from baseline within the treatment group and a non-significant change from baseline within the control group does not necessarily mean that the treatment effect was significant, i.e. that the treatment group improved significantly more than the control group. Allison and colleagues have identified DINS errors in many papers in the field of nutrition calling for their corrections (3-5).
Panayotov's RCT tests the differential effects of participants believing they are on a low-calorie diet compared to those who know they are on an isocaloric diet (n=7 per group). Outcomes measured were body mass (kg), fat mass (%), and BMI (kg/m 2 ) in the treatment and control group at the beginning of the experiment and after 8 weeks of both groups being on an isocaloric diet, where the treatment group were told they were receiving a low calorie diet. The results of the study are based on paired t-tests comparing the outcomes at baseline and 8 weeks, separately within each group. This can make treatment effects appear to be significant where they otherwise would not be.
We commend the author for sharing his data at our request, and we re-analyze the data focusing on the between-group differences to isolate the treatment effect. When we conduct t-tests on the differences (post-pre) between groups, conclusions change for one of the three outcomes ( Table 1). Panayotov reported a p-value of < .001 for decreased fat mass in the treatment group and nonsignificant change (p=.21) in the control group to conclude that the treatment is better than the control. However, when we compare the decrease in the treatment group to the decrease in the control group with an independent samples t-test, we obtain a non-significant p-value of p=.105.
When analyzing the % change [(post-pre)/pre], the p-value for the treatment effect on fat mass is also not significant p=.156. Treatment effects on body mass and BMI remained significant in re-analysis.
Incidentally, in our re-analysis, we found that we were unable to replicate the results of the paired t-tests reported in Panayotov's Table 2 for the experimental group. We obtained more significant results than those reported with p=.002 and p=.001 instead of the reported p=.02 and p=.01 reported for body mass (kg) and BMI respectively.
Panayotov should re-analyze the data from his study with statistical comparisons of outcomes between groups rather than within-groups and publish corrected results.

AUTHOR CONTRIBUTIONS
DA: identified the error in the Panayotov paper and co-wrote the paper. SD and GF: re-analyzed the data with the correct analyses and co-wrote the paper.

FUNDING
This work was funded in part by the National Institutes of Health (NIH): R25HL124208. The opinions expressed are those of the authors and not necessarily of the NIH or any other organization.