Introduction
In recent years, brain imaging technology has become a hot topic within the field of neuroscience. Such technology includes functional Magnetic Resonance Imaging (fMRI) (Liu et al., ; Wang et al., ), electroencephalography (EEG) (Zhang et al., ), Magnetoencephalography (MEG) (Alotaiby et al., ), and functional Near Infrared Spectroscopy (fNIRS) (Vergotte et al., ). In particular, because of its space resolving power, fMRI, combined with deep learning methods, has been widely used in the study of neuroscience. For example, Radial Basis Function Networks (RBFN) were used to identify Autism Spectrum Disorder (ASD) (Vigneshwaran et al., ), Convolutional Neural Networks (CNN) were used to diagnose amnestic Mild Cognitive Impairment (aMCI) (Meszlényi et al., ) and Feedforward Neural Networks (FNN) were used to classify Schizophrenia (Patel et al., ). When we studied this topic, we mainly searched the literature from the database of Web of Science, and reviewed references to the application of deep learning methods in the field of fMRI from 2004 to the present. This paper first summarized the analyses of the applications, advantages and disadvantages of a variety of deep learning methods in processing fMRI data, and the names, target problem, cohort size, and accuracy of these methods were presented in Table 1. Then, considering the importance of studying cognitive disorders within the field of neuroscience, this paper reviewed the trend in the development of deep learning methods to process fMRI data for the diagnosis of cognitive disorders. Lastly, recommendations for future work in developing deep learning methods to process fMRI data for the diagnosis of cognitive disorders were stated. It is expected that the analysis may provide useful suggestions to the field of the deep learning methods of fMRI data with cognitive impairment.
Table 1
| Name of method | Target problem | Cohort size | Accuracy | References |
|---|---|---|---|---|
| CNN | Classification of AD | AD 20, NC 20 | – | Huang et al., |
| AD 28, NC 15 | 96.86% | Sarraf and Tofighi, | ||
| Classification of aMCI | aMCI 25, NC 2 | 71.9% | Meszlényi et al., | |
| Classification of functional networks | HCP 68 | 94.61% | Zhao et al., | |
| Classification of mTBI | mTBI 16, HC 24 | – | Zhao et al., | |
| Classification of the data when watching image | Sample 5275 | 85% | Horikawa and Kamitani, | |
| Classification of the data when watching video | Training set 12 Test set 9 | – | Güçlü and Van Gerven, | |
| Sample 258 | 72% | Wen et al., | ||
| Classification of the data when dreaming | Sample 1600 | 82% | Horikawa and Kamitani, | |
| Classification of ADHD | ADHD 274, TDC 456 | 69.15% | Zou et al., | |
| Prediction of Glioma patient survival time | Glioma patient 69 | 89.9% | Nie et al., | |
| Classification of the memory about word | Sample 480 | 73.3% | Firat et al., | |
| FNN | Identify the data in simulated war environment | Subject 5 | 93% | Floren et al., |
| Classification of the data when watching image | Subject 3 | 72.3% | Zafar et al., | |
| Classification of HAND | HIV+ Subjects 5, HC 5 | – | Abidin et al., | |
| Classification of ASD | ASD 443, HC 435 | 70~80% | Vigneshwaran et al., | |
| ASD 505, TC 530 | 70% | Heinsfeld et al., | ||
| ASD 55, TC 55 | 86.36% | Guo et al., | ||
| Classification of aMCI | aMCI 10, HC 8 | 93.75% | Jin et al., | |
| Classification of MCI | MCI 48, NC 52 | 87.5% | Hu et al., | |
| Classification of Schizophrenia | SZ 72, HC 75 | 85.8% | Kim et al., | |
| SZ 72, HC 74 | 92% | Patel et al., | ||
| Classification of MCI | MCI 31, NC 31 MCI 12, NC 25 | ADNI2 data 72.58% Own data 81.08% | Suk et al., | |
| Other methods | Classification of ADHD | Sample 263 Sample 73 Sample 113 | NYU 37.41% Neuro 44.4% OHSU 80.88% | Kuang and He, |
| Classification of the data when execute the action | Task-fMRI data 12 | 94.2% | Jang et al., | |
| Classification of the data when execute the action | Subject 20 | – | Huang et al., | |
| Classification of the data when watching video | Sample 3660 | 94.15% | Han et al., | |
| Classification of the data watch different object | Sample 100 | True data 100% Model data 97% | Avesani et al., | |
| Classification of the data when read words | Sample 2160 | 87.5% | Kasabov et al., | |
| Classification of ASD | ASD 539, TC 573 | 68.5% | Dvornek et al., | |
| Classification of the data when touch object | Subject 4 | Mean square error 0 | Dixit and Mosier, |
The names, target problem, cohort size, and accuracy of deep learning methods to process fMRI data.
NC, normal control; TDC, typically developing children; SZ, Schizophrenia; TC, typical controls; HC, healthy controls; HIV, Human immunodeficiency virus.
Deep learning methods in fMRI data analysis
CNN
In recent years, CNN has drawn much attention and has been widely used in fMRI data processing, because valid features can be automatically extracted (Sarraf and Tofighi, ; Horikawa and Kamitani, ; Meszlényi et al., ). For example, CNN was used to effectively classify fMRI data (Horikawa and Kamitani, ) and to classify and diagnose Alzheimer's disease (AD) (Sarraf and Tofighi, ) and aMCI (Meszlényi et al., ). The CNN method consists of the following three applications: feature extraction, construction of auto-encoder and construction of 3-Dimensional Convolutional Neural Networks (3D-CNN) method.
Feature extraction
To predict the lifetime of patients with brain tumors, CNN was used to extract features from fMRI data with the Support Vector Machine used for classification (Nie et al., ). Indeed, the study can use CNN to extract valid features, but whether the resulting features are redundant or not remains to be tested. In addition, the extracted features correspond to what features of the brain have yet to be explored.
Auto-encoder
Feature maps can be obtained layer by layer using an auto-encoder (Huang et al., ). Auto-decoders had also drawn attention when interpreting fMRI data related to vision (Wen et al., ). In this study, unlike in the previous studies, Deconvolutional Neural Network was constructed to achieve the operation approximate to inverse convolution. However, for this method, it is still unknown for this method to explain all the region of vision. Generally, CNN is used mainly for two-dimensional data to build an auto-encoder, but there was a 3D-Convolutional Auto Encoder built for recognition of mild Traumatic Brain Injury (mTBI) (Zhao et al., ), despite more samples being needed to validate the results of the study. In order to comprehend the changes in brain state following memory words, a sparse encoder was constructed and used in conjunction with CNN for classification (Firat et al., ).
3D-CNN
The 3D-CNN can also be used to process fMRI data. The data from Human Connectome project (HCP) was utilized to reconstruct the brain network with a higher accuracy (Zhao et al., ). The methods of combining fMRI with structural magnetic resonance imaging had also been used for the classification of Attention deficit hyperactivity disorder (ADHD) (Zou et al., ). Although CNN can share the filter in the convolving layer and the number of parameters can be reduced in the pooling layer, the large amount of data required by the 3D network still demands extended training time.
FNN
FNN has been applied to analyze human brain changes in a simulated war environment (Floren et al., ). However, the FNN method is inefficient in training based on fMRI data due to the large input eigenvector. To reduce the dimension of the data and solve this problem, the fMRI data could be passed to the input layer through the encoder (Hu et al., ; Kim et al., ). However, the method proposed by Kim et al. is not suitable for the analysis of fMRI data on patients with different diseases. Thus, its generalization on different diseases needs to be strengthened. Based on the original encoder, a Deep Auto Encoder was used to classify fMRI data with MCI (Suk et al., ). But this study did not consider the dynamic relationship between different regions of the brain. In another approach, four hidden layers were added in between the encoder and the decoder in the FNN method (Patel et al., ). Apart from adding hidden layers, multiple encoders can also be stacked. For example, when diagnosing ASD, two denoising self-encoder were constructed (Heinsfeld et al., ). Multiple sparse auto-encoders had also been used (Guo et al., ). Yet, further experiments will need to be conducted with patients of different ages to validate the clinical value of these methods.
RBFN is a FNN, which uses radial basis functions in the hidden layer to non-linearly transform the input space. RBFN had been used to study HIV Associated Neurocognitive Disorder (HAND) (Abidin et al., ). The RBFN could be used to extract features from fMRI data (Vigneshwaran et al., ). This method can be extended to studies on women of different ages. Besides the basic feature selection, a feature selection method combining region of interest and voxel was designed to determine brain states under the different stimuli (Zafar et al., ). With regards to cognitive diseases, RBFN had also been used to classify different regions of the brain network impaired by aMCI (Jin et al., ). However, the sample size in this study is limited and so the impact of larger sample sizes needs to be investigated. Meanwhile, more careful regularization methods need to be developed to address the issue of over-fitting.
Other deep learning methods
In addition to the widely used methods described above, Restricted Boltzmann Machine (RBM) and Deep Boltzmann Machine (DBM) are also used for fMRI data analysis. The performance of the RBM had been shown to be better than independent component analysis (Huang et al., ), but the functional relationship between different regions of the brain was not examined. The DBM was proposed to identify areas of the brain that were activated while the subject was watching videos (Han et al., ). In the future, emotion-oriented data may be considered for classification. In general, the greater the number of training samples available to the deep learning method is, the better the classification performance of the method will be. Recently, three data sets (OHSU, NYU, and Neuro) used in the diagnosis of ADHD were classified with an accuracy of 80.88, 37.41, and 44.4%, respectively (Kuang and He, ), and the quantitative difference indicates that the generalization ability of this method is limited. When the amount of data is insufficient, the synthesis of analog data can be considered. Avesani et al. used Liquid State Machine to train and classify both simulated and real data (Avesani et al., ), but the results of the simulation data do not reflect the true performance of the classification method.
There are also two time-dependent deep learning methods for fMRI data analysis. The first method, Spiking Neuron Networks, converted fMRI data of the brain during reading into pulse signals (Kasabov et al., ). This method performed well in analyzing the temporal characteristics of the data, but failed to analyse the spatial characteristics. The second method, Long Short Term Memory (LSTM), was used for the study of ASD. In this method, the training samples were obtained from the average time series extracted from fMRI data after preprocessing (Dvornek et al., ). The extraction of the average time series in this study affects the accuracy of the method classification.
To reduce the number of features extracted, a deep belief network (DBN) was used to initialize the parameters and sparse pre-training was performed on weight (Jang et al., ). This method reduced the number of parameters and decreased the degree of over-fitting, but it takes more time for calculations.
The Hopfield Neural Network was used many years ago to classify fMRI data on subjects touching different objects (Dixit and Mosier, ). However, due to the small amount of data collected, the performance of the method still needs to be fully verified.
The main problem currently facing the development of deep learning methods for fMRI data analysis is that the training methods used a smaller amount of data. Therefore, the studies of small sample may lead to obvious over-fitting problems. The training methods may take more time when number of sample increase, and more training time makes it more difficult to conduct in-depth and meticulous parameter adjustment work, resulting in hindered improvement of method performance.
Development of deep learning methods for fMRI data analysis in cognitive impairment
Cognitive disorders, which is a general category including conditions such as Alzheimer's disease, mild cognitive impairment, and subjective cognitive decline, usually encompass the disorder of memory, language, visual space, execution, calculation, and understanding, and involve multiple regions of the brain and the associated abnormalities between these brain regions. This makes the diagnosis of cognitive disorders complicated. Therefore, the diagnosis of cognitive disorders has also become a very important issue in the field of neuroscience. In the following sections, the literature analyzing deep learning methods for classification of fMRI data in cognitive impairment studies will be reviewed. Recommendations and our opinion for the development of these methods are presented.
The measure to improve the performance of deep learning method in cognitive impairment
Firstly, for improving the model itself, as the number of fMRI data samples related to cognitive disorders is insufficient, we recommend to use a more refined regularization method, which includes the other hyperparameters of the deep learning methods. Secondly, the time cost of deep learning method training is higher, as the learning rate has an impact on the time taken for method training, the learning rate can be automatically adjusted according to the learning progress in future work. Thirdly, as the current method is aimed at studying a specific stage of the disease, a deep learning method to study all stages of the disease can be explored in the future.
From another perspective, we can apply methods for studying other diseases to cognitive impairment. In other words, deep learning methods that perform well in studying other diseases can be applied to analyzing fMRI data related to cognitive disorders. For example, the existing CNN method could be used to analyze fMRI data while the subject was watching videos (Güçlü and Van Gerven, ). As the method does not have to restart training, the training time is also greatly reduced.
Exploring deep learning methods specifically for fMRI data analysis of cognitive impairment
The functioning of the brain has now been shown to involve interactions and connection among multiple functional areas, and is a phenomenon also known as brain function network dynamics (Kim et al., ), in which the synchronous and coupling activity will occur between or among different areas of the brain when the brain performs a task. Cognitive disorders are due to variations in the dynamic relationships between pairs of specific brain regions and the network dynamics of multiple brain regions. Therefore, as current deep learning methods disregard dynamic considerations, when designing a deep learning method for fMRI data classification of cognitive impairment, it is suggested that the training of deep learning methods could be performed after calculating the coupling or synchronizing intensity between specific brain regions.
Conclusion
In conclusion, this study reviewed the recent literature of the deep learning used in fMRI data. Although these studies obtained promising results, more work needs to be done before conclusions can be confidently drawn in the field of deep learning used in fMRI. It must be noted that there are some limitations that occur both for deep learning and classical machine learning methods, even if in different proportions. For example, a small sample size is a source of overfitting in both cases, even if the number of samples required for training a classical machine learning method is extremely lower than the one needed for deep learning techniques (Salvatore et al., ). In addition, brain dynamics have not been taken seriously in the two ways. Compared with the classical machine learning methods, deep learning methods may have poorer performance under the condition of small sample size and training time cost (Nie et al., ), but we can make full use of the automatically-extracted features to improve accuracy of deep learning methods. In the future, we suggest collecting more subjects of cognitive impairment to test and revise the existing methods, and considering adequately the dynamic relationships among different brain regions.
Statements
Author contributions
DW: designed the study and wrote this paper; ZW and YZ: analyzed literatures and wrote this paper; GL and XZ: revised this paper; WH: designed the study.
Acknowledgments
This research was funded by National Natural Science Foundation of China (61503326), Natural Science Foundation of Hebei Province in China (F2016203343), China Postdoctoral Science Foundation (2015M581317) and Doctorial Foundation of Yanshan University in China (B900).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
1
AbidinA. Z.D'souzaA. M.NagarajanM. B. (2016). Investigating changes in brain network properties in HIV-Associated Neurocognitive Disease (HAND) using Mutual Connectivity Analysis (MCA), in Proceedings of SPIE–the International Society for Optical Engineering (San Diego, CA), 9788.
2
AlotaibyT. N.AlrshoudS.AlshebeiliS. A. (2017). Automatic meg data classification to epileptic and healthy subject using statistical features, in 32nd International Epilepsy Congress SEP (Barcelona) 5, S145–S145.
3
AvesaniP.HazanH.KoilisE. (2015). Non-parametric temporal modeling of the hemodynamic response function via a liquid state machine. Neural Netw.70, 61–73. 10.1016/j.neunet.2015.04.009
4
DixitS.MosierK. (2004). Analysis of cortical connectivity using Hopfield neural network. Neurocomputing58, 1163–1170. 10.1016/j.neucom.2004.01.181
5
DvornekN. C.VentolaP.PelphreyK. A. (2017). Identifying autism from resting-state fMRI using long short-term memory networks, in Machine Learning in Medical Imaging, MLMI (Workshop) (Quebec City, QC), 10541, 362–370.
6
FiratO.AksanE.OztekinI. (2015). Learning deep temporal representations for fMRI brain decoding, in 1st International Workshop on Medical Learning Meets Medical Imaging (MLMMI) (Lille).
7
FlorenA.NaylorB.MiikkulainenR. (2015). Accurately decoding visual information from fMRI data obtained in a realistic virtual environment. Front. Hum. Neurosci.9:327. 10.3389/fnhum.2015.00327
8
GüçlüU.Van GervenM. A. J. (2017). Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. Neuroimage145, 329–336. 10.1016/j.neuroimage.2015.12.036
9
GuoX.DominickK. C.MinaiA. A. (2017). Diagnosing autism spectrum disorder from brain resting-state functional connectivity patterns using a deep neural network with a novel feature selection method. Front. Neurosci.11:460. 10.3389/fnins.2017.00460
10
HanJ.JiX.HuX. (2015). Arousal recognition using audio-visual features and FMRI-based brain response. IEEE Trans. Affect. Comp.6, 337–347. 10.1109/TAFFC.2015.2411280
11
HeinsfeldA. S.FrancoA. R.CraddockR. C. (2018). Identification of autism spectrum disorder using deep learning and the ABIDE dataset. Neuroimage17, 16–23. 10.1016/j.nicl.2017.08.017
12
HorikawaT.KamitaniY. (2017a). Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun.8:15037. 10.1038/ncomms15037
13
HorikawaT.KamitaniY. (2017b). Hierarchical neural representation of dreamed objects revealed by brain decoding with deep neural network features. Front. Comput. Neurosci.11:4. 10.3389/fncom.2017.00004
14
HuC.JuR.ShenY. (2016). Clinical decision support for Alzheimer's disease based on deep learning and brain network, in IEEE International Conference on Communications (ICC) (Kuala Lumpur).
15
HuangH.HuX.HanJ. (2016). Latent source mining in fMRI data via deep neural network, in IEEE 13th International Symposium on Biomedical Imaging (ISBI) (Prague).
16
HuangH.HuX.ZhaoY. (2017). Modeling task fMRI data via deep convolutional autoencoder. IEEE Trans. Med. Imaging. [Epub ahead of print]. 10.1109/TMI.2017.2715285
17
JangH.PlisS. M.CalhounV. D. (2017). Task-specific feature extraction and classification of fMRI volumes using a deep neural network initialized with a deep belief network: evaluation using sensorimotor tasks. Neuroimage145, 314–328. 10.1016/j.neuroimage.2016.04.003
18
JinM.CurranT.CordesD. (2014). Classification of amnestic mild cognitive impairment using fMRI, in 11th IEEE International Symposium on Biomedical Imaging (ISBI) (Beijing).
19
KasabovN. K.DoborjehM. G.DoborjehZ. G. (2017). Mapping, learning, visualization, classification, and understanding of fMRI data in the NeuCube evolving spatiotemporal data machine of spiking neural networks. IEEE Trans. Neural Netw. Learn. Syst.28, 887–899. 10.1109/TNNLS.2016.2612890
20
KimJ.CalhounV. D.ShimE. (2016). Deep neural network with weight sparsity control and pre-training extracts hierarchical features and enhances classification performance: evidence from whole-brain resting-state functional connectivity patterns of schizophrenia. Neuroimage124, 127–146. 10.1016/j.neuroimage.2015.05.018
21
KimJ.CriaudM.ChoS. S. (2017). Abnormal intrinsic brain functional network dynamics in Parkinson's disease. 140, 2955–2967. 10.1093/brain/awx233
22
KuangD.HeL. (2014). Classification on ADHD with deep learning, in International Conference on Cloud Computing and Big Data (CCBD) (Wuhan).
23
LiuF.GuoW.LiuL.LongZ.MaC.XueZ.et al. (2013). Abnormal amplitude low-frequency oscillations in medication-naive, first-episode patients with major depressive disorder: a resting-state fMRI study. J Affect. Disord. 146, 401–406. 10.1016/j.jad.2012.10.001
24
MeszlényiR. J.BuzaK.VidnyánszkyZ. (2017). Resting state fMRI functional connectivity-based classification using a convolutional neural network architecture. Front. Neuroinformatics11:61. 10.3389/fninf.2017.00061
25
NieD.ZhangH.AdeliE. (2016). 3D deep learning for multi-modal imaging-guided survival time prediction of brain tumor patients, in International Conference on Medical Image Computing and Computer-Assisted Intervention (Istanbul) 9901, 212–220.
26
PatelP.AggarwalP.GuptaA. (2016). Classification of schizophrenia versus normal subjects using deep learning, in 10th Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP) (Guwahati).
27
SalvatoreC.BattistaP.CastiglioniI. (2016). Frontiers for the early diagnosis of AD by means of MRI brain imaging and support vector machines. Curr. Alzheimer Res.13, 509–533. 10.2174/1567205013666151116141705
28
SarrafS.TofighiG. (2016). Deep learning-based pipeline to recognize Alzheimer's disease using fMRI data, in Future Technologies Conference (FTC) (San Francisco, CA).
29
SukH.-I.WeeC.-Y.LeeS.-W. (2016). State-space model with deep learning for functional dynamics estimation in resting-state fMRI. Neuroimage129, 292–307. 10.1016/j.neuroimage.2016.01.005
30
VergotteG.TorreK.ChirumamillaV. C. (2017). Dynamics of the human brain network revealed by time-frequency effective connectivity in fNIRS. Biomed. Opt. Express8, 5326–5341. 10.1364/BOE.8.005326
31
VigneshwaranS.MahanandB. S.SureshS. (2015). Using regional homogeneity from functional MRI for diagnosis of ASD among males, in International Joint Conference on Neural Networks (IJCNN) (Killarney).
32
WangS.ZhanY.ZhangY. (2018). Abnormal long- and short-range functional connectivity in adolescent-onset schizophrenia patients: a resting-state fMRI study. Prog. Neuropsychopharmacol. Biol. Psychiatry81, 445–451. 10.1016/j.pnpbp.2017.08.012
33
WenH.ShiJ.ZhangY. (2017). Neural encoding and decoding with deep learning for dynamic natural vision. Cereb. Cortex [Epub ahead of print]. 1–25. 10.1093/cercor/bhx268
34
ZafarR.KamelN.NaufalM. (2017). Decoding of visual activity patterns from fMRI responses using multivariate pattern analyses and convolutional neural network. J. Integr. Neurosci.16, 275–289. 10.3233/JIN-170016
35
ZafarR.MalikA. S.KamelN. (2016). Role of voxel selection and ROI in fMRI data analysis, in IEEE International Symposium on Medical Measurements and Applications (MeMeA) (Benevento).
36
ZhangH.SuJ.WangQ. (2018). Predicting seizure by modeling synaptic plasticity based on EEG signals - a case study of inherited epilepsy. Commun. Nonlinear Sci. Num. Simul.56, 330–343. 10.1016/j.cnsns.2017.08.020
37
ZhaoY.DongQ.ChenH. (2017a). Constructing fine-granularity functional brain network atlases via deep convolutional autoencoder. Med. Image Anal.42, 200–211. 10.1016/j.media.2017.08.005
38
ZhaoY.DongQ.ZhangS. (2017b). Automatic recognition of fMRI-derived functional networks using 3D convolutional neural networks. IEEE Trans. Biomed. Eng. [Epub ahead of print]. 10.1109/TBME.2017.2715281
39
ZouL.ZhengJ.MiaC. (2017). 3D CNN Based Automatic diagnosis of attention deficit hyperactivity disorder using functional and structural MRI. IEEE Access5, 23626–23636. 10.1109/ACCESS.2017.2762703
Summary
Keywords
deep learning, fMRI, convolutional neural network, deep neural network, radial basis function network, cognitive impairment
Citation
Wen D, Wei Z, Zhou Y, Li G, Zhang X and Han W (2018) Deep Learning Methods to Process fMRI Data and Their Application in the Diagnosis of Cognitive Impairment: A Brief Overview and Our Opinion. Front. Neuroinform. 12:23. doi: 10.3389/fninf.2018.00023
Received
20 February 2018
Accepted
11 April 2018
Published
26 April 2018
Volume
12 - 2018
Edited by
Christian Salvatore, Istituto di Bioimmagini e Fisiologia Molecolare (CNR), Italy
Reviewed by
Dante R. Chialvo, Center for Complex Systems & Brain Sciences (CEMSC3), Argentina; Feng Liu, Tianjin Medical University General Hospital, China
Updates
Copyright
© 2018 Wen, Wei, Zhou, Li, Zhang and Han.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Wei Han 251541719@qq.com
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.