A long journey into reproducible computational neuroscience
Meropi Topalidou
Arthur Leblois
Thomas Boraud
Nicolas P. Rougier- 1INRIA Bordeaux Sud-Ouest, Bordeaux, France
- 2Institut des Maladies Neurodégénératives, Université de Bordeaux, UMR 5293, Bordeaux, France
- 3Institut des Maladies Neurodégénératives, Centre National de la Recherche Scientifique, UMR 5293, Bordeaux, France
- 4LaBRI, Université de Bordeaux, Institut Polytechnique de Bordeaux, Centre National de la Recherche Scientifique, UMR 5800, Talence, France
- 5Centre de Neurophysique, Physiologie et Pathologies, Université Paris Descartes, Centre National de la Recherche Scientifique, UMR 8119, Paris, France
A commentary on
Interaction between cognitive and motor cortico-basal ganglia loops during decision making: a computational study
by Guthrie, M., Leblois, A., Garenne, A., and Boraud, T. (2013). J. Neurophysiol. 109, 3025–3040. doi: 10.1152/jn.00026.2013
Computational neuroscience is a powerful ally in our quest to understand the brain. Even the most simple model can shed light on the role of this or that structure and propose new hypothesis concerning the overall brain organization. However, any model in Science is doomed to be proved wrong or incomplete and replaced by a more accurate one. In the meantime, for such replacement to happen, we have first to make sure that models are actually reproducible such that they can be tested, evaluated, criticized and ultimately modified, replaced or even rejected. This is where the shoe pinches. If we cannot reproduce a model in the first place, we're doomed to re-invent the wheel again and again, preventing us from building an incremental computational knowledge of the brain.
We have been recently confronted to the problem when we tried to reproduce a model of the literature (Guthrie et al., 2013) concerning a computational model of the basal ganglia. This model was based on a previous modeling study by Leblois et al. (2006) where authors proposed an action selection mechanism based on a competition between the positive feedback, direct pathway through the striatum and the negative feedback, hyperdirect pathway through the subthalamic nucleus. Guthrie et al. (2013) further investigated how multiple level action selection could be performed, and the model has been extended in a manner consistent with known anatomy and electrophysiology of the basal ganglia in the monkey. The model is quite complex, but such is the basal ganglia. We asked authors for the sources of the model only to realize it has been implemented using 6000 lines of Delphi (Pascal language). We were unable to compile it (due to missing packages that we couldn't locate in any repository) and we thus decided to recode it from scratch. Unfortunately, the information provided in the article was not sufficient to allow for the direct reproduction of the model, mainly because there were factual errors in the manuscript and some information was ambiguous or missing. Ultimately, we were able to reach two of the original authors, T. Boraud, and A. Leblois (who are also authors of this commentary) in order to ask them about the details of the model. We joined efforts and proceeded with a complete rewrite, using the Python language (Perkel, 2015), a dedicated library (DANA), a versioning system (git), a public repository (github) and the IPython notebook, merely following the principles of reproducible computational science as proposed in Peng (2011), Sandve et al. (2013), Stodden et al. (2014). We also established the tabular description of the model as proposed in Nordlie et al. (2009). We claim this revamped model now allows any researcher in computational neuroscience to run it and to obtain the exact same results as the ones described in the original article (see https://hal.inria.fr/hal-01109483 for complete description and https://github.com/rougier/Neurosciences/tree/master/basal-ganglia/guthrie-et-al-2013 for source code). Furthermore, the new description, as well as the new figures, allow anyone to rewrite the model using a different language, tools or software.
However, the whole process took us approximately 3 months. This is hardly acceptable for the reproduction of a computational model that should be straightforward.
Unfortunately, this is not an isolated case. If computer science offers a large set of tools for prototyping, writing, running, testing, validating, sharing, and reproducing results, computational neuroscience still lags behind. In the best case, authors may provide the sources of their model as a compressed archive and feel confident their model is reproducible. But this is not exactly true. Buckheit and Donoho (1995) explained almost 20 years ago that, “an article about computational result is advertising, not scholarship. The actual scholarship is the full software environment, code and data that produced the result.” The computational part in computational sciences implies the use of computers, operating systems, tools, frameworks, libraries, and data. This leads to such a large number of combinations (taking into account the version for each components) that the chances to have the exact same configuration as one of your colleague are nearly zero. This draws consequences in our respective computational approaches in order to make sure models and simulations can be actually and faithfully reproduced. We have to enforce the rules proposed in the literature and editors have to make sure this actually happens. From a broader perspective, this singular experience raises also some questions about the whole publication process. If articles remain the best media to publish a research and to introduce a model, why can't we have associated resources for the actual code just like we can have Supplementary Material as a separate document? For example, it is quite surprising that there is still no official code repository associated with journals. Even a dedicated public account on github (or any similar website) would really help on this matter.
But more importantly, given the quality of the new tools available today, it may be time to envisage new formats for communicating computational researches. For example, interactive documents could allow to replay a simulation and to modify parameters while reading the description of a model or a simulation. The IPython notebook is a serious candidate in that direction and could soon become a new way to exchange knowledge. It has been recently highlighted on Nature (Shen, 2014) and it is already widely used for teaching and writing books. Such new formats would definitely help authors, reviewers, readers and ultimately, Science.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
Buckheit, J., and Donoho, D. L. (1995). “Wavelab and reproducible research,” in Wavelets and Statistics, Vol. 103, ed A. Antoniadis (New York, NY: Springer-Verlag), 55–81. doi: 10.1007/978-1-4612-2544-7_5
Guthrie, M., Leblois, A., Garenne, A., and Boraud, T. (2013). Interaction between cognitive and motor cortico-basal ganglia loops during decision making: a computational study. J. Neurophysiol. 109, 3025–3040. doi: 10.1152/jn.00026.2013
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Leblois, A., Boraud, T., Meissner, W., Bergman, H., and Hansel, D. (2006). Competition between feedback loops underlies normal and pathological dynamics in the basal ganglia. J. Neurosci. 26, 3567–3583. doi: 10.1523/JNEUROSCI.5050-05.2006
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Nordlie, E., Gewaltig, M., and Plesser, H. E. (2009). Towards reproducible descriptions of neuronal network models. PLoS Comput. Biol. 5:e1000456. doi: 10.1371/journal.pcbi.1000456
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Peng, R. D. (2011). Reproducible research in computational science. Science 334, 1226–1227. doi: 10.1126/science.1213847
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Perkel, J. M. (2015). Programming: pick-up python. Nature 518, 125–126. doi: 10.1038/518125a
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Sandve, G. K., Nekrutenko, A., Taylor, J., and Hovig, E. (2013). Ten simple rules for reproducible computational research. PLoS Comput. Biol. 9:e1003285. doi: 10.1371/journal.pcbi.1003285
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Shen, H. (2014). Interactive notebooks: sharing the code. Nature 515, 151–152. doi: 10.1038/515151a
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Stodden, V., Leisch, F., and Peng, R. D. (eds.). (2014). Implementing Reproducible Research. Chapman & Hall/CRC. Available online at: http://www.crcnetbase.com/doi/book/10.1201/b16868
Keywords: reproducible science, publication process, computational models, python language, version control, public repository, notebook
Citation: Topalidou M, Leblois A, Boraud T and Rougier NP (2015) A long journey into reproducible computational neuroscience. Front. Comput. Neurosci. 9:30. doi: 10.3389/fncom.2015.00030
Received: 30 January 2015; Accepted: 19 February 2015;
Published: 05 March 2015.
Edited by:
Florentin Wörgötter, University Goettingen, GermanyReviewed by:
Florentin Wörgötter, University Goettingen, GermanyTimo Nachstedt, University of Goettingen, Germany
Copyright © 2015 Topalidou, Leblois, Boraud and Rougier. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Nicolas P. Rougier, nicolas.rougier@inria.fr