AUTHOR=Qiao Kai , Zhang Chi , Wang Linyuan , Chen Jian , Zeng Lei , Tong Li , Yan Bin 

TITLE=Accurate Reconstruction of Image Stimuli From Human Functional Magnetic Resonance Imaging Based on the Decoding Model With Capsule Network Architecture

JOURNAL=Frontiers in Neuroinformatics

VOLUME=Volume 12 - 2018

YEAR=2018

URL=https://www.frontiersin.org/journals/neuroinformatics/articles/10.3389/fninf.2018.00062

DOI=10.3389/fninf.2018.00062

ISSN=1662-5196

ABSTRACT=In neuroscience, all kinds of computation models were designed to answer the open question of how sensory stimuli are encoded by neurons and conversely, how sensory stimuli can be decoded from neuronal activities. Especially, functional Magnetic Resonance Imaging (fMRI) studies have made many great achievements with the rapid development of the deep network computation. However, comparing with the goal of decoding orientation, position and object category from activities in visual cortex, accurate reconstruction of image stimuli from human fMRI is a still challenging work. Inspired by the structure of cortical mini column including several hundred neurons in primates, the capsule means containing a group of neurons to perform better organization of feature structure and representation. Especially, the high-level capsule’s features in the capsule network (CapsNet) includes various features of image stimuli such as semantic class, orientation, location, scale and so on, and these features can better represent the processed information inherited in the fMRI data collected in visual cortex. In this paper, a novel CapsNet architecture based visual reconstruction (CNAVR) computation model is developed to reconstruct image stimuli from human fMRI. The CNAVR is composed by linear encoding computation from capsule’s features to fMRI data and inverse reconstruction computation. In the first part, we employed the CapsNet model to train the non-linear mapping from image stimuli to high-level capsule’s features, and from high-level capsule’s features to image stimuli again in an end-to-end manner. In the second part, we trained the non-linear mapping from the fMRI data of selected voxels to high-level capsule’s features. For a new image stimuli, we can use the method to predict the corresponding high-level capsule’s features from fMRI data, and reconstruct image stimuli with the trained reconstruction part in the CapsNet. As a result, we evaluated the proposed CNAVR method on the open dataset of handwritten digital images, and achieved overwhelming superiority and exceeded about 10% than the accuracy of all existing state-of-the-art methods on the structural similarity index (SSIM).