NDF: A Data Standard for the CARMEN system
-
1
University of York, United Kingdom
Introduction
CARMEN (Code Analysis, Repository and Modelling for e-Neuroscience, Flecher et al.) is an environment for sharing neurophysiological experimental data and algorithms through an internet portal. There is a wide range of incoming data types used in electrophysiology as well as derived data from the output of system applications/services. They are often unreadable without the original software or knowledge of the format encoding. This presents a real barrier for data sharing and reuse of algorithms and software in neuroscience system. This paper describes Neurophysiology Data translation Format (NDF) developed for CARMEN that overcomes some of these problems.
The NDF data format
NDF provides a standard for sharing data, specifically for data transfer between the various analysis applications/services on the CARMEN system. NDF can be also a useful data format on a researcher’s desktop.
An NDF dataset consists of a configuration file in XML format which contains metadata and references to the associated host data files. Using a separated header allows services or users to extract the necessary information about the data set from a small size of header file without need to download the full data set over the network. This also provides means for the system to display detailed meta-data of a remote NDF data without need to access the binary data file remotely. The NDF specifies a set of the most commonly used experimental data entities as "NDF internal data types". NDF can include images and image series data as a basic NDF data type. NDF also supports annotation/marker event data in XML format as a special experimantal event data type.
Data processing services can output data types that may not be represented by formats used for primary data. NDF provides two extandable "semi-defined" data types for applications to create a new data type as its output. The configuration file provides seamless access to these different representations based on the applications used to read the data.
A special XML data element, History, is designed within the header file for data processing history recording. This element contains the full histroy record chain of the privious processing and provides important information for both users and machines to understand a particular experiment, validation of algorithms and for other researchers to accurately repeat an experiment reported.
The NDF API and tools
The NDF API has been implemented as a C library. The NDF API translates the XML tree/node into C style data structures and insulates the data structures of the binary data file from the clients. The API also provides a standard way for data structure memory management for NDF data application programming. Currently, the NDF API has been applied to a data analysis/visualization tool (the SDE) and the CARMEN web based data processing services.
A MATLAT toolbox for NDF has also been implemented. This toolbox has been used as the data input/output pre-processor for the CARMEN services. It can also be used a set of convenient tools on a researcher’s desktop for data I/O, independent of CARMEN.
The system currently uses the NeuroShare library as the data input module and will be extended to other input modules.
Conclusions.
It has been shown that a standard data format is crucial in a system like CARMEN where data, tools and algorithms are shared. A single data format standard greatly reduces the work load on system and tool implementation. The efforts of neuroscientists can therefore be concentrated on research work rather than being consumed with the intricacies of transferring data from one format to the others.
Conference:
Neuroinformatics 2010 , Kobe, Japan, 30 Aug - 1 Sep, 2010.
Presentation Type:
Poster Presentation
Topic:
Infrastructural and portal services
Citation:
Liang
B,
Jackson
T and
Austin
J
(2010). NDF: A Data Standard for the CARMEN system.
Front. Neurosci.
Conference Abstract:
Neuroinformatics 2010 .
doi: 10.3389/conf.fnins.2010.13.00118
Copyright:
The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers.
They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.
The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.
Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.
For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.
Received:
15 Jun 2010;
Published Online:
15 Jun 2010.
*
Correspondence:
Bojian Liang, University of York, York, United Kingdom, austin@cs.york.ac.uk