Edited by: Veronika Schöpf, Medical University Vienna, Austria
Reviewed by: Robert C. Welsh, University of Michigan, USA; Georg Langs, Medical University of Vienna, Austria
†Jing Sui and Hao He are co-first authors for this paper.
This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and subject to any copyright notices concerning any third-party graphics etc.
Multimodal brain imaging data have shown increasing utility in answering both scientifically interesting and clinically relevant questions. Each brain imaging technique provides a different view of brain function or structure, while multimodal fusion capitalizes on the strength of each and may uncover hidden relationships that can merge findings from separate neuroimaging studies. However, most current approaches have focused on pair-wise fusion and there is still relatively little work on
Multimodal brain imaging techniques are playing increasingly important roles in elucidating structural and functional properties in normal and diseased brains, as well as providing the conceptual glue to bind together data from multiple types or levels of analysis. The related computational methods are also valuable for clinical research on the mechanisms of disease progression. The goal of multimodal fusion is to capitalize on the strength of each imaging modality as well as their inter-relationships in a joint analysis, rather than to analyze separately.
Each imaging modality provides a different view of brain function or structure, and data fusion capitalizes on the strengths of each imaging modality/task and their inter-relationships in a joint analysis, creating an important tool to help unravel the black box of psychotic disorders, such as schizophrenia (SZ) (Calhoun et al.,
However, most current approaches have focused on pair-wise fusion and there is still relatively little work on
To our knowledge, there have been only a few reports combining three or more types of brain imaging data to investigate brain disorders (e.g., Correa et al.,
In this project, we applied the
Existing multivariate fusion methods have different optimization priorities and limitations: some enable common as well as distinct levels of connection among modalities, such as mCCA (Correa et al.,
The basic strategy of mCCA + jICA is shown in Figure
We assume that the multimodal dataset
Finally,
Multi-set canonical correlation analysis + jICA was compared with its alternatives in simulation in Sui et al. (
Multi-set canonical correlation analysis + jICA was applied to DTI, resting state fMRI, and sMRI data of 63 subjects recruited as part of a multimodal SZ center for biomedical research excellence (COBRE) study at the Mind Research Network
Num | Age | Gender | Ethnicity | |
---|---|---|---|---|
HC | 28 | 39 ± 15 | 21M/7F | 21 Whites |
SZ | 35 | 36 ± 12 | 26M/9F | 22 Whites |
0.36 | 0.99 | 0.58 |
All the data were collected on a 3-T Siemens Trio scanner with a 12-channel radio frequency coil at the Mind Research Network. The imaging parameters were as follows:
Resting-state scans were a minimum of 5 min, 4 s in duration (152 volumes). Subjects were instructed to keep their eyes open during the scan and stare passively at a foveally presented fixation cross, as this is suggested to facilitate network delineation compared to eyes-closed conditions and helps ensure that subjects are awake.
SPM8 software package
DTI data were preprocessed by FMRIB Software Library (FSL)
sMRI data were also preprocessed using the SPM8 software package which was used to segment the brain into white-matter (WM), GM, and cerebral spinal fluid with unmodulated normalized parameters via the unified segmentation method (Ashburner and Friston,
After feature extraction (preprocessing), the 3D brain images of each subject were reshaped into a one-dimensional vector and stacked, forming a matrix with dimensions of 63 × number of voxels for each of the three modalities. These three feature matrices were then normalized to have the same average sum-of-squares (computed across all subjects and all voxels/locus for each modality) to ensure all modalities had the same ranges. Following normalization, the relative scaling (a normalization factor) within a given data type was preserved (i.e., 1.08, 0.24, 0.39 for ALFF, FA, GM respectively), but the normalized input units have the same voxel-wise mean square variance for all modalities. Next, the data was processed via the pipeline shown in Figure
After applying the mCCA + jICA to the human brain data, independent component
Two-sample
We also looked into the column-wise correlations between A1, A2, and A3 pair wisely. It is likely that the joint group-discriminative components have a strong inter-modality correlation between their mixing coefficents, which indicates the interaction and correspondence among modalities.
The derived mixing coefficients also provide a way to investigate the relationships between the identified components and subjects’ clinical data, e.g., the correlation between mixing coefficients of patients for each component and antipsychotic medication doses [standardized as olanzapine equivalents (Gardner et al.,
To test the potential use of the identified group-discriminative components (i.e., corresponding rows of
For each modality, we transferred the group-discriminating components (for ALFF and GM, we use only two ICs with minimum
Two-sample
There was no significant correlation regarding the antipsychotic medication doses. However, two ICs: FA_IC4 (anterior thalamic radiation, ATR and superior longitudinal fasciculus, SLF) and GM_IC4 (subregions of the default mode) were significantly correlated with positive PANSS scores, while there was no significant correlation with negative PANSS score. The scatter plots and linear trends are shown in Figure
The specific identified regions of the components of interest and their abbreviations are summarized in Table
Area | Brodmann area | Vol. (cm3) | |
---|---|---|---|
Positive | |||
Superior temporal gyrus | 13, 22, 38, 39, 41 | 4.4/3.4 | 3.6 (−48, −40, 8)/4.6 (48, −38, 7) |
Middle temporal gyrus | 21, 22, 37, 39 | 5.4/1.3 | 4.5 (−48, −35, 2)/3.5 (48, −32, 2) |
Middle frontal gyrus | 6, 8, 9, 46 | 3.4/1.5 | 3.7 (−50, 16, 32)/3.0 (50, 19, 32) |
Inferior frontal gyrus | 9, 44, 45, 47 | 3.8/0.1 | 3.1 (−50, 10, 33)/2.1 (42, 30, 12) |
Negative | |||
Middle temporal gyrus | 21 | 0.7/0.3 | 3.1 (−45, −55, 6)/2.6 (42, −52, 8) |
Parahippocampal gyrus | 30 | 0.3/0.2 | 3.0 (−24, −46, 5)/2.6 (27, −46, 5) |
Positive | |||
Superior temporal gyrus | 21, 22, 39 | 1.0/2.0 | 2.9 (−48, −40, 8)/3.6 (50, −26, −1) |
Middle temporal gyrus | 19, 20, 21, 22, 39 | 1.8/2.9 | 3.2 (−48, −32, 2)/3.5 (48, −26, −4) |
Inferior frontal gyrus | 13, 46 | 1.2/1.6 | 2.7 (−39, 30, 12)/3.1 (39, 35, 9) |
Parahippocampal gyrus | 28, 36 | 1.3/1.0 | 2.8 (−27, −12, −15)/2.4 (30, −7, −17) |
Fusiform gyrus | 37 | 0.8/0.4 | 2.8 (−48, −47, −13)/2.5 (48, −47, −13) |
Negative | |||
Precentral gyrus | 4, 6 | 6.1/6.0 | 4.3 (−24, −23, 65)/3.3 (15, −23, 67) |
Lingual gyrus | 18 | 0.6/1.0 | 4.0 (3, −73, −6)/4.2 (12, −82, −14) |
Paracentral lobule | 4, 5, 6, 31 | 2.6/2.5 | 4.2 (0, −29, 51)/3.9 (3, −32, 51) |
Postcentral gyrus | 1, 2, 3, 5, 7, 40 | 4.3/3.3 | 4.1 (−21, −26, 65)/3.0 (50, −29, 51) |
Medial frontal gyrus | 6, 8, 32 | 3.0/4.2 | 4.1 (0, −23, 56)/3.6 (3, −20, 56) |
Posterior cingulate | 29 | 0.3/0.4 | 3.2 (−3, −58, 6)/3.6 (3, −58, 6) |
Superior frontal gyrus | 6, 8 | 3.4/3.2 | 3.3 (0, 5, 49)/3.2 (21, −8, 67) |
Precuneus | 7, 39 | 1.4/4.9 | 3.3 (−30, −62, 34)/3.2 (9, −74, 42) |
Inferior parietal lobule | 40 | 1.6/2.0 | 3.3 (−42, −35, 54)/3.3 (48, −32, 54) |
Positive | |||
Middle temporal gyrus | 19, 21, 22, 37, 39 | 6.2/2.2 | 3.7 (−42, −69, 15)/2.9 (53, −58, 11) |
Superior temporal gyrus | 13, 22, 38, 39, 41, 42 | 5.2/2.6 | 3.5 (−53, −57, 19)/3.0 (50, −52, 14) |
Supramarginal gyrus | 40 | 2.9/2.4 | 3.4 (−53, −54, 22)/2.8 (53, −45, 30) |
Precuneus | 7, 19, 23, 31, 39 | 3.2/6.0 | 3.2 (0, −51, 36)/3.3 (3, −36, 43) |
Parahippocampal gyrus | 19, 28, 34 | 2.3/0.9 | 3.2 (−24, −38, 5)/2.7 (24, −41, 5) |
Cingulate gyrus | 24, 31, 32 | 2.0/2.1 | 3.1 (0, −42, 35)/3.2 (3, −33, 40) |
Anterior cingulate | 25 | 0.6/0.3 | 3.1 (0, 5, −8)/2.7 (3, 5, −10) |
Postcentral gyrus | 2, 40 | 2.0/0.2 | 3.1 (−50, −33, 49)/2.1 (50, −32, 51) |
Positive | |||
Precuneus | 7, 19, 39 | 2.9/1.5 | 4.0 (−24, −65, 36)/4.6 (30, −59, 36) |
Cerebellum | 8.8/7.8 | 3.7 (0, −47, −8)/3.5 (3, −50, −8) | |
Middle frontal gyrus | 6, 10 | 1.0/0.7 | 3.6 (−33, 39, 20)/2.9 (33, 47, 6) |
Thalamus | 1.8/1.0 | 3.5 (−6, −23, 12)/2.7 (3, −14, 12) | |
Middle temporal gyrus | 19, 21, 22, 37, 39 | 1.8/0.9 | 3.1 (−48, −38, 5)/2.9 (48, −35, 2) |
Negative | |||
Superior temporal gyrus | 21, 38 | 1.5/0.6 | 3.1 (−30, 16, −24)/2.4 (45, 20, −16) |
Positive | |||
Superior temporal gyrus | 22, 38 | 1.4/2.5 | 3.1 (−45, 11, −11)/3.7 (48, 11, −6) |
Cuneus | 7, 17, 18, 23, 30 | 2.6/0.7 | 3.5 (−12, −93, 5)/2.4 (18, −96, 8) |
Superior frontal gyrus | 6, 8, 9, 10 | 4.0/3.1 | 3.3 (−24, 48, 31)/3.1 (21, 11, 49) |
Middle frontal gyrus | 6, 8, 9, 10 | 5.3/2.6 | 3.1 (−33, 58, 3)/2.7 (27, 3, 52) |
Precuneus | 7, 19, 31 | 1.5/0.6 | 3.1 (−27, −62, 34)/2.9 (30, −62, 36) |
Medial frontal gyrus | 6, 8, 10, 32 | 1.3/1.1 | 3.1 (0, 11, 44)/3.0 (21, 5, 49) |
Negative | |||
Middle temporal gyrus | 19, 22, 39 | 1.8/1.5 | 3.9 (−48, −43, 5)/5.0 (42, −57, 22) |
Positive | |||
Angular gyrus | 39 | 0.6/0.4 | 3.7 (−33, −54, 36)/3.8 (36, −56, 36) |
Precuneus | 7, 19, 39 | 1.5/0.6 | 3.7 (−30, −62, 36)/3.1 (36, −62, 36) |
Supramarginal gyrus | 40 | 0.4/0.4 | 3.1 (−36, −51, 36)/3.2 (36, −51, 36) |
Middle frontal gyrus | 6, 8, 9, 10 | 1.0/2.6 | 3.0 (−33, 16, 27)/2.9 (33, 19, 27) |
Lingual gyrus | 17 | 1.7/0.5 | 3.0 (−12, −87, 2)/2.6 (18, −87, 4) |
Negative | |||
Inferior frontal gyrus | 9, 44, 45, 47 | 2.7/2.1 | 3.7 (−48, 14, −3)/3.7 (48, 17, −6) |
Superior temporal gyrus | 22, 38, 42 | 4.2/1.7 | 3.7 (−48, 11, −6)/3.2 (50, 14, −6) |
Insula | 13 | 1.6/0.1 | 3.5 (−45, 8, −5)/2.2 (45, 8, −5) |
Abbreviation | WM tracts | Vol. (cm3) | % | |
---|---|---|---|---|
Positive | ||||
ATR | Anterior thalamic radiation | 2.3/7.2 | 5/14 | 4.7 (26, 31, 13)/5.2 (28, 25, 6) |
CST | Corticospinal tract | 2.1/2.3 | 6/7 | 5(25, 33, 7)/5.1(31, 34, 14) |
CG | Cingulum | 0.5/0.7 | 2/2 | 2.9(18, 21, 18)/3.1(28, 14, 31) |
FM | Forceps minor/Forceps major | 1.7/3.4 | 3/7 | 3.9(27, 47, 21)/5(27, 26, 22) |
IFO | Inferior fronto-occipital fasciculus | 1.1/2 | 2/5 | 3.9(16, 11, 22)/3.7(35, 45, 21) |
ILF | Inferior longitudinal fasciculus | 1.7/3.1 | 4/7 | 3.9(12, 19, 17)/5.3(41, 31, 15) |
SLF | Superior longitudinal fasciculus | 5.6/4.6 | 5/4 | 4.8(6, 25, 15)/5.4(44, 27, 15) |
UF | Uncinate fasciculus | 0.3/0.5 | 3/4 | 3.8(22, 51, 13)/2.9(40, 37, 10) |
Negative | ||||
ATR | Anterior thalamic radiation | 1.1/0.9 | 2/2 | 3.3(20, 38, 27)/3.4(27, 27, 4) |
CST | Corticospinal tract | 1.9/1.4 | 5/4 | 3.5(25, 27, 7)/4.6(29, 31, 8) |
SLF | Superior longitudinal fasciculus | 3.2/4.1 | 3/4 | 5.2(12, 39, 29)/6(46, 30, 11) |
Positive | ||||
ATR | Anterior thalamic radiation | 0.8/4.2 | 2/8 | 7.8(27, 26, 2)/7.4(28, 24, 1) |
CST | Corticospinal tract | 2.7/1.9 | 7/6 | 8.5(26, 26, 1)/9.3(27, 26, 1) |
ILF | Inferior longitudinal fasciculus | 0.7/2.2 | 2/5 | 2.9(11, 32, 12)/4.2(44, 30, 12) |
SLF | Superior longitudinal fasciculus | 1.6/3.0 | 2/3 | 5.6(4, 26, 17)/5.3(48, 29, 10) |
Negative | ||||
ATR | Anterior thalamic radiation | 2.3/1.2 | 6/4 | 4.2(24, 24, 8)/4.3(28, 31, 11) |
IFO | Inferior fronto-occipital fasciculus | 2.1/1.7 | 4/4 | 3.6(19, 9,23)/3.7(40, 15, 25) |
ILF | Inferior longitudinal fasciculus | 2.1/1.4 | 5/3 | 3.4(13, 15, 18)/3.3(45, 32, 13) |
SLF | Superior longitudinal fasciculus | 4.4/6.3 | 5/6 | 5(7, 27, 15)/5.1(48, 29, 14) |
Area | Brodmann area | Vol. (cm3) | |
---|---|---|---|
Positive | |||
Superior frontal gyrus | 8, 9, 10, 11 | 3.8/4.8 | 9.5 (−30, 43, −15)/9.5 (21, 43, −17) |
Middle frontal gyrus | 6, 10, 11, 46, 47 | 6.5/5.8 | 7.9 (−30, 40, −17)/7.8 (30, 40, −17) |
Inferior frontal gyrus | 11, 46, 47 | 2.4/3.3 | 7.4 (−24, 31, −19)/6.0 (15, 31, −17) |
Medial frontal gyrus | 10, 11, 25 | 5.8/6.8 | 6.0 (−12, 28, −17)/6.1 (9, 43, −17) |
Superior temporal gyrus | 22, 38 | 0.4/0.4 | 3.5 (−56, 11, −6)/2.8 (59, 11, −6) |
Anterior cingulate | 10, 25, 32 | 1.0/0.3 | 3.5 (−12, 49, −5)/2.3 (15, 46, −5) |
Thalamus | 0.3/0.2 | 3.0 (−6, −11, 14)/3.0 (6, −5, 11) | |
Positive | |||
Superior frontal gyrus | 6, 10, 11 | 0.8/0.3 | 5.4 (−18, 64, 8)/3.4 (9, 67, 8) |
Superior temporal gyrus | 22, 38 | 5.4/0.1 | 4.9 (−33, 13, −28)/2.3 (30, 10, −31) |
Medial frontal gyrus | 10 | 0.9/0.0 | 4.3 (−6, 64, 5)/−999.0 (0, 0, 0) |
Inferior frontal gyrus | 44, 45, 46, 47 | 2.0/0.0 | 4.0 (−53, 20, −9)/−999.0 (0, 0, 0) |
Middle frontal gyrus | 10, 11 | 1.3/0.3 | 3.4 (−42, 52, −10)/3.7 (30, 62, 19) |
Negative | |||
Cingulate gyrus | 23, 24, 32 | 2.2/2.8 | 3.5 (−9, 4, 27)/4.1 (9, 4, 27) |
Anterior cingulate | 24, 33 | 0.5/0.8 | 3.5 (−6, 10, 24)/4.0 (12, 13, 24) |
Superior frontal gyrus | 8, 10, 11 | 1.1/1.7 | 3.7 (−30, 32, 51)/3.9 (18, 43, −15) |
Middle temporal gyrus | 21, 38, 39 | 0.1/0.8 | 3.0 (−56, −66, 28)/2.6 (62, −35, −8) |
Positive | |||
Superior frontal gyrus | 6, 8, 9, 10, 11 | 14.3/14.1 | 6.6 (−21, 57, 28)/6.5 (18, 65, 16) |
Middle frontal gyrus | 6, 8, 9, 10, 11, 46 | 12.5/11.6 | 5.0 (−27, 59, 19)/5.6 (24, 62, 19) |
Medial frontal gyrus | 6, 8, 9, 10 | 4.9/4.5 | 4.7 (−3, 49, 42)/4.7 (3, 49, 42) |
Inferior frontal gyrus | 9, 10, 45, 46, 47 | 2.6/2.0 | 3.9 (−42, 55, 0)/2.9 (56, 10, 33) |
Superior temporal gyrus | 38 | 0.6/0.1 | 3.1 (−42, 19, −26)/2.2 (39, 22, −26) |
After transferring the group-discriminating components into
Each individual was assigned one of two class memberships (SZ versus HC) and we have seven modal combinations (three single, three pair-wise, one three-way) as shown in Figure
In this paper we applied the mCCA + jICA model to three-way fusion of resting state fMRI, sMRI, and DTI data. The aim of the method is to identify precise correspondence among
IC 7 significantly differentiated SZ from HC in all three modalities, suggesting the following abnormalities in SZ: (a) prefrontal cortex and left superior temporal gyrus (STG) (rest fMRI); (b) ATR, corticospinal tract (CSF), and forceps major (FMAJ; WM, DTI); and (c) regions of the motor cortex, medial/superior frontal cortex, and temporal gyrus (GM density). Furthermore, these identified affected regions may share some underlying relationship in SZ. The FA changes in ATR, CST, and FMAJ were previously associated with disconnectivity of brain networks in SZ in separate studies (Schlosser et al.,
Furthermore, GM-ALFF IC6 is another joint group-discriminative component, with middle/medial frontal cortex and thalamus (Woodward et al.,
We also identified ICs of interest showing significance only in one modality, such as GM_IC 5, 9, 10 and ALFF_IC3 (pink frame). The three structural components indicated regions including STG, precuneus, prefrontal cortex, insula, and thalamus, Hence, GM concentrations were significantly reduced in the above regions in the SZ group, consistent with other findings (Ha et al.,
Positive symptoms refer to an excess or distortion of normal psychological functions, e.g., hallucinations and delusions. In Figure
The classification in Figure
In this paper we develop and evaluate a novel multivariate method that can explore cross-information in multiple (more than two) data types and applied it to compare SZ patients to controls using an fMRI-DTI-sMRI combination. This is a novel attempt to perform a fusion of three different imaging modalities. The method described here could be applied straightforwardly to study other brain diseases (or subsets of a particular illness, such as psychotic or non-psychotic bipolar disorder). In addition, the choice of which multimodal data type to utilize is flexible, i.e., EEG, MEG, or genetic data, different features like fractional ALFF (fALFF) from fMRI (Kalcher et al.,
A limitation to the current study is that the subject number is not very high. Several statistical tests did not survive from the multiple comparisons, which may be complemented in future studies by including more subject samples or by multi-site recruitment. Additionally, mCCA + jICA operates on extracted features, rather than the original imaging data (e.g., using FA values instead of raw DTI data). Although some of the information is lost using this method, a “feature” tends to be more tractable than working with the large-scale original data due to the reduced number of dimensions (Calhoun and Adali,
Another point worth noting is that we did not collect physiologic data during the rest fMRI session as studies of patients tend to make this more difficult to collect. However it would be worth evaluating this in future work. With the advent of more rapid scanning (e.g., multiband sequences) which can adequately sample the cardiac noise, it is becoming much more feasible to characterize physiologic noise in large patient studies. We did not collect information on nicotine use either in these subjects, which may have potential effects on the imaging results, and would better be taken into account in the future. For example, recent studies indicated evidences of smoking effect in resting-state networks (Janes et al.,
Multimodal fusion is an effective approach for analyzing biomedical imaging data that combines multiple data types in a joint analysis. It helps to identify the unique and shared variance associated with each imaging modality that underlies cognitive functioning in HCs and impairment in mental illness. In this real-world fusion application, we highlighted data from rest fMRI, WM tract, and GM concentration from SZ and healthy control subjects. We identified both modality-common and modality-unique group-discriminating aspects that verified the abnormalities in SZ, as well as replicated and extended previous findings. Such observations add to our understanding of the neural correlates of SZ. The proposed model promises a widespread utilization in the neuroimaging community and may be used to identify potential brain illness biomarkers.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
This work was supported by the National Institutes of Health grants R01EB 006841, R01EB 005846, and 5P20RR021938 (to Vince D. Calhoun), and R01MH43775, R01MH074797, and R01MH077945 (to Godfrey D. Pearlson).
1
2
3