NeuroUnit: Validation Tests for Neuroscience Models
-
1
Carnegie Mellon University, Dept. of Biological Sciences, United States
-
2
Carnegie Mellon University, Dept. of Computer Science, United States
Rigorous validation of a scientific model’s explanatory power requires comparing the model’s predictions against all relevant experimental data. Model validation is an ongoing process -- models must not simply be validated against data available when the model is initially peer reviewed, but also to data gathered after it has been published. Today, model validation remains an informal and incomplete process. We argue, by drawing an analogy between scientific model validation and software testing, that precise validation criteria that allow model validation to be partially automated are essential for assessing model validity and scope. While modern journal articles tell us clearly how a model works, they provide only incomplete and quickly dated outlines for telling us which observations it aims to explain and how well it achieves them. To overcome this problem, we propose formalizing the model validation process by creating a collection of software tools and associated cyberinfrastructure dedicated to scientific model validation. This validation framework will exist in parallel to the publication system: publications can focus on answering how and refer to the testing framework for answering how well.
In software engineering, a unit test is a function that validates a single component of a computer program against a single correctness criterion. The core of our proposal is the analogous concept of a validation test -- an executable function that validates a provided model implementation against a single empirical observation to produce a score that indicates agreement between the model and a single piece of data. Suites of these tests will be developed via a distributed cyberinfrastructure that enables 1) collaborative construction and curation of tests by members of a scientific community and 2) the execution of tests and the reporting of their results continuously as new data is gathered and new models are developed. We describe a core pythonic framework, SciUnit, that begins to fulfill this vision, and NeuroUnit, a library of SciUnit tests and associated standards for the neurosciences. Visual summaries of aggregate NeuroUnit test results will provide neuroscientists with an up-to-date report of progress in neuroscience modeling, illustrating the merits and deficiencies of competing models, benefiting both ongoing efforts and informing new theoretical and experimental directions. To illustrate the generality of this approach, we consider the result of subjecting historical models of planetary motion to tests derived from empirical observations throughout history (Table 1).
Keywords:
unit test,
Validation,
python,
neuroscience methods,
Software Validation
Conference:
Neuroinformatics 2013, Stockholm, Sweden, 27 Aug - 29 Aug, 2013.
Presentation Type:
Poster
Topic:
Computational neuroscience
Citation:
Gerkin
RC and
Omar
C
(2013). NeuroUnit: Validation Tests for Neuroscience Models.
Front. Neuroinform.
Conference Abstract:
Neuroinformatics 2013.
doi: 10.3389/conf.fninf.2013.09.00013
Copyright:
The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers.
They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.
The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.
Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.
For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.
Received:
08 Apr 2013;
Published Online:
11 Jul 2013.
*
Correspondence:
Dr. Richard C Gerkin, Carnegie Mellon University, Dept. of Biological Sciences, Pittsburgh, PA, 15218, United States, rgerkin@asu.edu