Effective benchmarking of high-density MEA spike-sorters
-
1
ETH Zürich, D-BSSE, Switzerland
-
2
Friedrich Miescher Institute for Biomedical Research, Switzerland
MOTIVATION
State of the art high-density microelectrode arrays (HD-MEAs) feature several hundreds to thousands of parallel readout channels and are used for a variety of applications in neuroscience (Obien et al., 2015). These HD-MEAs create a demand for fast and reliable spike-sorters to reveal single-neuron activity from extracellular signals and a variety of algorithms are being developed.
The performance of these algorithms depends strongly on the experimental conditions and the parameters used. Therefore, it is important to perform effective benchmarking in each situation. This is not a trivial task due to a lack of reliable 'ground-truth' data, i.e., datasets in which the exact spike times of some or all neurons are known.
Here we present a simple method to generate benchmarking datasets for high-density microelectrode array recordings. The method generates surrogate datasets based on real recordings, and thus works for a variety of experimental conditions. We demonstrate how the method can be used to evaluate the accuracy of spike-sorting algorithms. In the example we present, we took the characteristic spike waveforms of recorded retinal ganglion cells, displaced them in space and inserted them back into the original data at random time-points. We then spike-sorted these datasets and compared the detected units to the inserted ground-truth neurons. This allowed us to estimate the performance of the spike-sorter under the given experimental conditions.
The data sets and the corresponding analysis scripts are publicly available.
MATERIALS AND METHODS
We used data recorded from murine retinae using a high-density CMOS MEA (Müller et al., 2015) with a total of 26400 electrodes at a density of 3300 el./mm2. The MEA featured 1024 readout channels with a 10 bit, 20 kHz analog-to-digital converter (ADC) each. The signals were digitally band-pass filtered between 300 and 7000 Hz in a post-processing step.
A reprogrammable switch-matrix in the MEA allowed us to connect two arbitrarily located disjoint blocks of 23 x 23 adjacent electrodes to the readout channels with only few electrodes per block that remained unconnected. We created these high-density blocks so that they were translation-symmetric, which means that the unconnected electrodes were located at the same relative positions within each block.
We used an automatic spike-sorter to obtain single-unit activity of neurons in the recordings and we called them 'original units'. From each of the two high-density blocks, we randomly selected 10 original units that had at least 4000 spikes.
After cutting out the voltage traces of each spike on all recording electrodes, we averaged them to get the characteristic spike waveforms of each unit. To ascertain zero on- and offset, we multiplied these waveforms with a Tukey-window.
Based on each characteristic waveform, we created a new 'artificial unit' simply by swapping the location of the two high-density blocks. This resulted in a new characteristic waveform that had the same spatial extent as its precursor but at a different location on the HD-MEA and without the need for spatial upsampling of a model-based modification of the waveform.
We inserted these new waveforms as spikes into the original recordings by adding them onto the recorded voltage traces. The amplitudes of individual artificial spikes were randomly scaled so that their distribution matched the amplitude distributions of units in the recording. We also created temporal jitter by up-sampling the waveforms and randomly shifting them within the up-sampling interval before down-sampling them again. This procedure ensured that the inserted spikes were not always perfectly aligned with the sampling intervals. We chose the time points at which spikes were inserted into the data using an independent Poisson process with a refractory period of 1.5 ms.
To evaluate the spike-sorting results, we compared the spike trains of 'detected units' and 'artificial units', matched the pairs that produced the smallest number of errors and assessed the sorting accuracy for each unit.
RESULTS
Based on six 20-minute recordings of murine retinae with an estimated total number of 4000 neurons, we created 12 datasets with a total of 240 artificial units. In half of the datasets we kept the firing rate constant, in the other half the spike amplitude. With these datasets we could assess the performance of HD-MEA spike-sorters as a function of amplitude and spiking rate of neurons.
DISCUSSION
It is crucial that experimenters can estimate the performance of a spike-sorter in their specific use cases. There are alternative methods to obtain test datasets for spike-sorters: One method would be to use patch-clamp recordings to obtain the ground-truth of specific neurons in real cell populations. This approach is strongly limited by the fact that only one or two cells can be patched at the same time. A different method would be to generate simulated data based on a computer model of cell populations. This would provide ground-truth of the entire population, but to obtain realistic noise properties and spike shapes, a complex model would be required and many parameters would have to be fitted to accommodate the respective situation.
Our method represents a trade-off between having a large number of ground-truth neurons and having realistic noise conditions, and furthermore, it is simple and fast to implement. The method is suitable for all HD-MEAs where the recording electrodes are arranged in a way that contiguous groups can be swapped.
CONCLUSION
The presented method is a simple way for experimenters to evaluate spike-sorters under the respective experimental conditions. It allows for generating test datasets that have the same noise profiles and contain ground-truth neurons with similar spatiotemporal characteristics as the recorded data. Moreover, users can evaluate the sorting performance as a function of the spiking amplitude or rate.
Acknowledgements
Financial support through the ERC Advanced Grant 694829 "neuroXscales" and the Swiss National Science Foundation Sinergia Projects CRSII3_141801 and CRSII5_173728 are acknowledged, as well as individual support for R. Diggelmann through a Swiss SystemsX IPhD grant and for F. Franke through a Swiss National Science Foundation Ambizione Grant PZ00P3_167989.
References
Müller, J., Ballini, M., Livi, P., Chen, Y., Radivojevic, M., Shadmani, A., Viswam, V., Jones, I.L., Fiscella, M., Diggelmann, R., et al. (2015). High-resolution CMOS MEA platform to study neurons at subcellular, cellular, and network levels. Lab Chip 15, 2767-2780.
Obien, M.E.J., Deligkaris, K., Bullmann, T., Bakkum, D.J., and Frey, U. (2015). Revealing neuronal function through microelectrode array recordings. Front. Neurosci. 8.
Keywords:
Spike-sorting,
high-density MEAs,
Benchmarking,
Ground-truth,
Retinal Ganglion Cells
Conference:
MEA Meeting 2018 | 11th International Meeting on Substrate Integrated Microelectrode Arrays, Reutlingen, Germany, 4 Jul - 6 Jul, 2018.
Presentation Type:
Poster Presentation
Topic:
Neural Networks
Citation:
Diggelmann
R,
Hierlemann
AR and
Franke
F
(2019). Effective benchmarking of high-density MEA spike-sorters.
Conference Abstract:
MEA Meeting 2018 | 11th International Meeting on Substrate Integrated Microelectrode Arrays.
doi: 10.3389/conf.fncel.2018.38.00092
Copyright:
The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers.
They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.
The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.
Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.
For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.
Received:
15 Mar 2018;
Published Online:
17 Jan 2019.
*
Correspondence:
Mr. Roland Diggelmann, ETH Zürich, D-BSSE, Zurich, Switzerland, roland.diggelmann@bsse.ethz.ch