While Panayotov reported very strong effects in their study of a “placebo” diet (1), the statistical analysis was performed incorrectly. To evaluate the effect of a treatment in a randomized controlled trial (RCT), the difference between groups should be tested directly rather than basing conclusions on the changes within each group separately. Conclusions in Panayotov’s manuscript are based on what has been called a DINS error or difference in nominal significance (2). A significant change from baseline within the treatment group and a non-significant change from baseline within the control group does not necessarily mean that the treatment effect was significant, i.e. that the treatment group improved significantly more than the control group. Allison and colleagues have identified DINS errors in many papers in the field of nutrition calling for their corrections (3–5).
Panayotov’s RCT tests the differential effects of participants believing they are on a low-calorie diet compared to those who know they are on an isocaloric diet (n=7 per group). Outcomes measured were body mass (kg), fat mass (%), and BMI (kg/m2) in the treatment and control group at the beginning of the experiment and after 8 weeks of both groups being on an isocaloric diet, where the treatment group were told they were receiving a low calorie diet. The results of the study are based on paired t-tests comparing the outcomes at baseline and 8 weeks, separately within each group. This can make treatment effects appear to be significant where they otherwise would not be.
We commend the author for sharing his data at our request, and we re-analyze the data focusing on the between-group differences to isolate the treatment effect. When we conduct t-tests on the differences (post-pre) between groups, conclusions change for one of the three outcomes (Table 1). Panayotov reported a p-value of < .001 for decreased fat mass in the treatment group and non-significant change (p=.21) in the control group to conclude that the treatment is better than the control. However, when we compare the decrease in the treatment group to the decrease in the control group with an independent samples t-test, we obtain a non-significant p-value of p=.105. When analyzing the % change [(post-pre)/pre], the p-value for the treatment effect on fat mass is also not significant p=.156. Treatment effects on body mass and BMI remained significant in re-analysis.
Table 1
| Trt Group | Ctrl Group | Effect size Cohen’s d | T-test | ANCOVA* | |
|---|---|---|---|---|---|
| M (SD) (n=7) | M (SD) (n=7) | p-value | p-value | ||
| Change score (Post-Pre) | |||||
| Body Mass (kg) | -9.26 (4.80) | -2.26 (4.43) | 1.52 | .015 | .025 |
| Fat (%) | -3.40 (0.89) | -1.43 (2.68) | 1.00 | .105 | .096 |
| BMI | -2.89 (1.36) | -0.99 (1.20) | 1.48 | .017 | .023 |
| % Change (Post-Pre)/Pre | |||||
| Body Mass (kg) | -0.08 (0.04) | -0.02 (.04) | 1.63 | .011 | .019 |
| %Change in Fat (%) | -0.09 (0.03) | -0.04 (.08) | 0.84 | .156 | .142 |
| %Change in BMI | -0.08 (0.04) | -0.03 (.04) | 1.45 | .020 | .025 |
Re-analysis of intervention effects.
*ANCOVA includes a covariate for baseline values of each outcome.
Incidentally, in our re-analysis, we found that we were unable to replicate the results of the paired t-tests reported in Panayotov's Table 2 for the experimental group. We obtained more significant results than those reported with p=.002 and p=.001 instead of the reported p=.02 and p=.01 reported for body mass (kg) and BMI respectively.
Panayotov should re-analyze the data from his study with statistical comparisons of outcomes between groups rather than within-groups and publish corrected results.
Funding
This work was funded in part by the National Institutes of Health (NIH): R25HL124208. The opinions expressed are those of the authors and not necessarily of the NIH or any other organization.
Statements
Author contributions
DA: identified the error in the Panayotov paper and co-wrote the paper. SD and GF: re-analyzed the data with the correct analyses and co-wrote the paper.
Conflict of interest
In the last 12 months, DA has received personal payments or promises for same from: American Society for Nutrition; American Statistical Association; Biofortis; California Walnut Commission; Columbia University; Fish & Richardson, P.C.; Frontiers Publishing; Henry Stewart Talks; IKEA; Indiana University; Laura and John Arnold Foundation; Johns Hopkins University; Law Offices of Ronald Marron; MD Anderson Cancer Center; Medical College of Wisconsin; National Institutes of Health (NIH); Sage Publishing; The Obesity Society; Tomasik, Kotin & Kasserman LLC; University of Alabama at Birmingham; University of Miami; Nestle; WW (formerly Weight Watchers International, LLC). Donations to a foundation have been made on his behalf by the Northarvest Bean Growers Association. DA is an unpaid member of the International Life Sciences Institute North America Board of Trustees. The institution of DA, SD, and GF, Indiana University, has received funds to support their research or educational activities from: NIH; Alliance for Potato Research and Education; American Federation for Aging Research; Dairy Management Inc; Herbalife; Laura and John Arnold Foundation; National Cattlemen’s Beef Association, Oxford University Press, the Sloan Foundation, The Gordan and Betty Moore Foundation, and numerous other for-profit and non-profit organizations to support the work of the School of Public Health and the university more broadly. DA’s prior institution, the University of Alabama at Birmingham, received gifts, contracts, and grants from other organizations including the Coca-Cola Company, Pepsi, and Pepper/Snapple.
References
1
PanayotovVS. Studying a Possible Placebo Effect of an Imaginary Low-Calorie Diet. Front Psychiatry (2019) 10:550. doi: 10.3389/fpsyt.2019.00550
2
AllisonDBBrownAWGeorgeBJKaiserKA. Reproducibility: A tragedy of errors. Nat News (2016) 530(7588):27. doi: 10.1038/530027a
3
AllisonDBAntoineLHGeorgeBJ. Incorrect statistical method in parallel-groups RCT led to unsubstantiated conclusions. Lipids Health Dis (2016) 15(1):77. doi: 10.1186/s12944-016-0242-3
4
CassaniRSLFassiniPGSilvahJHLimaCMMMarchiniJS. Retraction Note: Impact of weight loss diet associated with flaxseed on inflammatory markers in men with cardiovascular risk factors: a clinical study. Nutr J (2015) 15(1):59. doi: 10.1186/s12937-016-0165-x
5
KroegerCMBrownAWAllisonDB. Differences in Nominal Significance (DINS) Error leads to invalid conclusions: Letter regarding,”Diet enriched with fresh coconut decreases blood glucose levels and body weight in normal adults”. J Complementary Integr Med (2019) 16(2):20180037. doi: 10.1515/jcim-2018-0037
Summary
Keywords
difference in nominal significance, reanalyzed data, error, correction, data analysis
Citation
Dickinson SL, Foote G and Allison DB (2020) Commentary: Studying a Possible Placebo Effect of an Imaginary Low-Calorie Diet. Front. Psychiatry 11:329. doi: 10.3389/fpsyt.2020.00329
Received
11 January 2020
Accepted
01 April 2020
Published
29 April 2020
Volume
11 - 2020
Edited by
Stephan Zipfel, University of Tübingen, Germany
Reviewed by
Paul Enck, University of Tübingen, Germany
Updates
Copyright
© 2020 Dickinson, Foote and Allison.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Stephanie L. Dickinson, sd3@indiana.edu
This article was submitted to Psychosomatic Medicine, a section of the journal Frontiers in Psychiatry
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.