Stereoscopic three-dimensional visualization applied to multimodal brain images: clinical applications and a functional connectivity atlas

Effective visualization is central to the exploration and comprehension of brain imaging data. While MRI data are acquired in three-dimensional space, the methods for visualizing such data have rarely taken advantage of three-dimensional stereoscopic technologies. We present here results of stereoscopic visualization of clinical data, as well as an atlas of whole-brain functional connectivity. In comparison with traditional 3D rendering techniques, we demonstrate the utility of stereoscopic visualizations to provide an intuitive description of the exact location and the relative sizes of various brain landmarks, structures and lesions. In the case of resting state fMRI, stereoscopic 3D visualization facilitated comprehension of the anatomical position of complex large-scale functional connectivity patterns. Overall, stereoscopic visualization improves the intuitive visual comprehension of image contents, and brings increased dimensionality to visualization of traditional MRI data, as well as patterns of functional connectivity.


INTRODUCTION
Visualization is critical to understanding and reporting neuroimaging data. The complexities of brain geometry and its variation from one individual to the next, combined with the increasing number of imaging modalities (MRI, PET, SPECT, fMRI, DTI) and representations (e.g., parcellations, voxels, vertices) required to fully characterize its structure and function, make visualization a formidable challenge. Arguably, the increased emphasis on patterns of brain connectivity has ushered in a new level of challenge due to the vast complexity of the connectome. Classical methods for viewing clinical and research neuroimaging data, such as print media or radiological film, reduce the three-dimensional (3D) structure of the brain to two-dimensional (2D) representations, resulting in significant loss of information. However, these 2D visualizations are not mandated by technical limitations or fundamental limits of human perception. By taking advantage of stereoscopy-an approach for the visualization of data in 3D that dates back Abbreviations: ICN, intrinsic connectivity networks; MRI, magnetic resonance imaging; MS, multiple sclerosis; PET, positron emission tomography; SPECT, single-photon emission computed tomography, fMRI, functional magnetic resonance imaging; DTI, diffusion tensor imaging; FLAIR, fluid attenuated inversion recovery.
nearly two centuries-the higher dimensional structure of neuroimaging data can be faithfully reproduced.
In 1838 the English physicist Sir Charles Wheatstone first defined the concept of stereopsis as the perception of depth that results from slight differences in the visual projection of the world onto the two retinas due to their distinct anatomical positions (Wheatstone, 1838). This phenomenon is commonly referred to as binocular disparity, retinal disparity, or retinal rivalry (Lipton, 1982). Later in the same year, Sir Charles Wheatstone created a device called a "stereoscope," which provides each eye with a distinct picture. Using two angled mirrors, he demonstrated the human brain's tendency to bring together the two images in a manner that yields perception of a 3D visualization (Wheatstone, 1842). Finally, Wilhelm Rollmann, a physicist, built upon this concept to devise the anaglyph, which employs red and green lenses to create a 3D visualization by showing different pictures filtered by red and green lenses to each eye (Rollmann, 1853).
For anaglyphs, the viewer wears eyeglasses with two different color lenses (usually chromatically opposite), such as red and cyan. The picture contains two differently colored images that have a slight offset, which is seen by each eye when filtered by the colored lenses. Such anaglyphs can be viewed in several types of media: computer screens (for example, Internet web pages), cinema, TV (videogames, DVD), digital projectors, or print. Different color combinations can be used for anaglyph eyeglasses. For example (generally in left-right eye order; Ribas et al., 2001;Hawkins, 2002): red-green (pure colors), red-blue (pure colors), or red-cyan (pure red, pure green+blue). In the first two cases the color rendering is monochrome, and in the third it is full color (good color perception of green and blue, poor perception of red). Over the years, various applications of red-cyan anaglyph 3D method have emerged for the purposes of anatomical visualization in medicine. For example, Hirsch and Kramer used cyan-red anaglyphs to create a paper and CD-ROM based human brain atlas, entitled "Neuroanatomy: 3D-Stereoscopic Atlas of the Human Brain" (Hirsch and Kramer, 1999). Providing 173 illustrations of interactive and rotatable 3D models, this work demonstrated the value of 3D stereoscopy for visualizing the human brain-especially when augmented with modern-day computing capabilities. Another example is provided by Guilherme Carvalhal Ribas (Ribas et al., 2001) who devised a 3D anaglyph-based approach for visualizing anatomical and surgical images obtained by cameras affixed to a surgical microscope; this resource was intended for both teaching and generating surgical reports.
Here we introduce the application of anaglyph-based 3D stereoscopy to visualization of data obtained from various MR-based imaging modalities. First, we illustrate the utility of anaglyphbased viewing of MRI data spanning commonly used imaging modalities (i.e., morphometry, diffusion imaging, and functional MRI), including clinical data (e.g., neoplasms). Then, we provide illustrations and descriptions of an interactive full brain functional connectivity atlas we developed using 3D stereoscopic visualization techniques. The atlas uses seed-based correlation analyses and contains illustrations for the voxel-wise connectivity maps associated with each of the 200 functional regions in the brain recently identified by Craddock et al. (2012) using datadriven cluster analysis approaches. While various parcellation schemes are available for the brain, this approach was selected due to its emphasis on the delineation of functional (rather than structural) regions as the basis for functional connectivity regions-of-interest (ROIs).

MORPHOMETRIC ILLUSTRATIONS FOR CLINICAL CASES
Morphometric studies for eight clinical cases are included in the present work for the purpose of demonstration, after receiving express written permission from the patients. Stereoscopic illustrations were generated for each of the cases using 3D Slicer version 3.6.3 anaglyph stereo option (Brigham & Women's Hospital, Boston, Massachusetts, USA, www.slicer.org; Gering et al., 1999;Pieper et al., 2004Pieper et al., , 2006. The imaging parameters used for the clinical cases are provided below.

Clinical case P6
For case P6, a 24 year-old female patient with a left frontal glioma, a standard fMRI block-design language generation task was used to activate Broca's area in the left posterior inferior frontal gyrus (Binder, 2006). Blood-oxygenation level-dependent (BOLD) signals were measured with a T2 * -weighted echo-planar imaging (EPI) sequence (TR = 2890 ms, TE = 50 ms, flip angle = 90 • , FOV = 192 × 192 mm, in-plane resolution = 64 × 64 pixels or 3 × 3 mm). Twenty interlaced axial slices, with 5.0 mm thickness (gap = 1.25 mm) were acquired. Each acquisition had 55 scans including 5 dummy scans that were discarded from analysis. A standard 3D structural sagittal T1-weighted MPRAGE image used the acquisition parameters described previously. See standard T2-Flair image in Figure 1F. The fMRI was processed using FSL 4.1.8 software package (Smith et al., 2004;Woolrich et al., 2009). A 3D sagittal Fluid Attenuated Inversion Recovery (FLAIR) image (TE = 354 ms, TR = 7000 ms, flip angle = 180 • , FOV = 294 × 212, in-plane resolution = 256 × 184 pixels or 1.15 × 1.15 mm) was used to segment the glioma using a simple region growing algorithm and then to create a 3D model of the tumor using the Model Maker module of 3D Slicer software package.

Clinical cases P7-P8
For two individuals with multiple sclerosis (P7: a 37 year-old female, P8: a 42 year-old male), the imaging protocols included 3D sagittal FLAIR images (same parameters as above) and a 3D structural sagittal T1-weighted MPRAGE image using the acquisition parameters described previously (Figures 1G,H). Multiple sclerosis (MS) lesion segmentation and lesion load was done using Toads-Cruise 2010 R2a Medical Image Processing, Analysis, and Visualization (MIPAV) software (http://mipav.cit.nih.gov) plug in (McAuliffe et al., 2001;Shiee et al., 2010). The Model Maker module of 3D Slicer software package was used to create the 3D model of the MS lesions.

Image preprocessing
Data processing was performed using Analysis of Functional NeuroImaging (AFNI; http://afni.nimh.nih.gov/afni; Cox, 1996) and fMRIB Software Library (FSL; www.fmrib.ox.ac.uk). Image preprocessing consisted of slice time correction for interleaved acquisitions; 3D motion correction with Fourier interpolation; despiking (detection and compression of extreme time series outliers); spatial smoothing using a 6 mm FWHM Gaussian kernel; mean-based intensity normalization of all volumes by the same factor; temporal bandpass filtering (0.009-0.1 Hz); and linear and quadratic detrending.
Linear registration of high-resolution structural images to the MNI152 template was carried out using the FSL tool FLIRT (Jenkinson and Smith, 2001;Jenkinson et al., 2002). This transformation was then refined using FNIRT non-linear registration (Andersson et al., 2007a,b). Linear registration of each participant's functional time series to the high-resolution structural image was performed using FLIRT. This functionalto-anatomical co-registration was improved by intermediate registration to a low-resolution image and b0 unwarping.
To control for the effects of physiological processes (such as fluctuations related to cardiac and respiratory cycles) and motion, we regressed each participant's 4-D preprocessed volume on nine predictors that modeled nuisance signals from white matter, CSF, the global signal, and six motion parameters. Each participant's resultant 4-D residuals volume was spatially normalized by applying the previously computed transformation to MNI152 standard space with 2 mm 3 resolution.
The image fusion and stereoscopic 3D images of each case were created using 3D Slicer 3.6.3 software (Gering et al., 1999;Pieper et al., 2004Pieper et al., , 2006. 3D Slicer implemented the red-cyan anaglyph method for producing stereoscopic 3D images.

Craddock functional connectivity atlas
A 3D stereoscopic full brain functional connectivity atlas was created using an atlas published by Craddock et al. (2012). Using 3D Slicer 3.6.3 and the 200 ROI version of the Craddock atlas, 200 grayscale surface models were created using a z-stat threshold > 2.3, and each surface model was processed with a surface decimation algorithm (Target reduction = 0.2), smoothed with the Taubin algorithm (5 iterations, Taubin passband = 0.1) and without surface normals. The surface model of the 200 ROIs of the Craddock parcellation was created using the Model Maker 3D Slicer module with 10 smoothing iterations, Sinc type smoothing, and 0.25 decimal percentage of target reduction in the number of polygons.   For improved visualization of the functional connectivity networks and their relative anatomical position, the surface model of five subcortical anatomical structures (corpus callosum, bilateral caudate, pallidum, putamen, thalamus, and hippocampus) were included in ICN visualizations. These surfaces were created with 3D Slicer using the segmentation computed with FreeSurfer v. 5.1.

Evaluation of visualization techniques
Twenty-one 3D images first and subsequently the same stereoscopic versions were shown to a group of 14 healthcare professionals (8 pediatric and adult neurologists, 1 neuroradiologist, 2 neurosurgeons, 1 radiologist, and 2 nurses; 30-52 years old). For each image, they were asked about the utility of each version of the image (3D, stereoscopic) in the assessment of a distinct brain feature. The questionnaire is included in Table 1, and each

CLINICAL ILLUSTRATIONS
To demonstrate the added value of stereoscopy in the visualization of structural and functional abnormalities in clinical patients we created anaglyph-based 3D images for a series of magnetic resonance studies. Below we discuss highlights from representative cases of commonly found abnormalities (common and anaglyphic 3D images are provided for each case to enable comparison. Anaglyph images must be viewed using red-cyan anaglyph glasses from computer screen or printed matter). See Table 2 for clinical descriptions.

Mass lesions
One of the most obvious potential applications of stereoscopy is the visualization of mass lesions, such as tumors and cysts. To demonstrate this point, we first generated common and anaglyphic 3D visualizations from the MPRAGE data for patient P3, a 36 year-old female with history of epilepsy secondary to right precentral tumor (see Figures 3, 4, and Supplementary Video 1). FreeSurfer segmentation identified the tumor as an "unknown" tissue, and the resultant mesh of the right hemisphere was notable for a hole in the pial surface (see Figures 4A,B). While both common and anaglyph 3D images show the tumor in detail, as well as its anatomical position, the anaglyph version is notably more realistic, as it more effectively conveys the depth of the tumor. Next, we generated common and anaglyphic 3D images from the MPRAGE data for patient P4, who was diagnosed with epilepsy secondary to a right temporal porencephalic cyst. As is notable in Figure 5, the cyst, appearing as a large hole in the right temporal lobe of the brain, is better conveyed with stereoscopic visualization. This advantage is due to the more effective representation of the depth of the hole produced by the cyst, as well  as the relative geometry of the adjacent gyrus and sulcus, which further enhances the realism of the image. Beyond visualization of MPRAGE-based morphometric findings, stereoscopic approaches can be extended to the visualization of diffusion imaging-based findings. In this regard, we provide common and anaglyphic versions of tractography visualizations of patient P5, a 63-year old female with a medulla oblongata cavernoma depicted in yellow in Figure 6. Figure 6A shows a standard presentation of tractography, while Figure 6B (and Supplementary Video 2) is a stereoscopic depiction. The approaches are notably different, with Figure 6A using variations in color related to FA (fractional anisotropy) to convey details; in contrast, Figure 6B depicts the tracts in gray to facilitate stereoscopic viewing (colors need to be removed to prevent any unintended impact on the anaglyph glasses). Despite the lack of color, the stereoscopic image shows the relative position of the tracts with respect to the cavernoma and the different sections of the tracts.

Atrophy
Another potential use of stereoscopy is to enhance the ability to visually identify atrophy in brain structures commonly affected by aging, or mesial temporal sclerosis, such as the hippocampus. 3D images appear in Figures 2A,C, and stereoscopic images of P1 appear in Figures 2B,D. P2 related images appear in Figures 2E,F in 3D and stereoscopic versions, respectively. The hippocampus is shown in yellow. The left hippocampus is slightly atrophied and the right shows moderate atrophy. The stereoscopic visualization allows one to perceive the difference in size between both hippocampi and their differences in relative position in a realistic way.

White matter lesions
Central to the diagnosis of myelin-related disorders such as multiple sclerosis is the visualization and identification of white matter lesions. To demonstrate the potential utility of stereoscopy, we provide common and anaglyphic 3D illustrations of MPRAGE and FLAIR images from two patients with multiple sclerosis (MS patients P7 and P8; see Figure 8 and Supplementary Video 3). Figures 8A,C,E depict the common "3D" images, and Figures 8B,D,F illustrate the stereoscopic forms. The stereoscopic images show the relative position of the MS lesions between patients, and between lesions and anatomical slices. Figure 8F shows the relative position of each MS lesion: the lesions in the left hemisphere appear closer to the observer, and the lesions in the right hemisphere appear farther away, with the corpus callosum located between them.

Altered functional activation
A final application to clinical examinations is provided by the illustration of altered functional activation in pathologic brain regions. Figure 7 shows the 3D (Figures 7A,C,E) and stereoscopic images (Figures 7B,D,F) of a left frontal glioma in a 24-year old patient (P6) with expressive language fMRI activation. The fMRI activation and glioma are shown in 3D models (pink and brown colors respectively). The models show that the expressive language activation (Broca's area) is in close contact with the frontal glioma.

Resting state fMRI based functional connectivity stereoscopic atlas
We selected the default network for our primary demonstration of the utility of stereoscopy in visualizing large-scale functional networks, as it is one of the most commonly examined and widely distributed networks in the brain. As demonstrated in Figure 9 (and Supplementary Video 4), using the anaglyph approach, we can perceive the relative location, morphology and size of key structures in the network (i.e., medial temporal lobe, medial prefrontal cortex, posterior cingulate cortex, precuneus, and parietal cortex; see Figures 9B,D,F, and  Supplementary Video 4).
The anaglyph-based approach can be applied to any functional network in the brain. Building upon this strength, we created a functional atlas of the brain (see Figure 10), including the networks for each of the 200 functional units identified in a recent functional-connectivity based parcellation of the brain (Craddock et al., 2012). The complete stereoscopic functional connectivity atlas (Stereoscopic Atlas of Intrinsic Brain Networks; SAIBN) is available at NITRC (www.nitrc.org/projects/saibn/). Of note, to better visualize the position of a functional network in the brain, a solution used in this paper was the addition of a surface model of subcortical structures (such as hippocampus, amygdala, corpus callosum, 3rd and 4th ventricles, lateral ventricles, putamen, pallidum, caudate, thalamus). The same approach could be used for our functional connectivity atlas to perceive the relative position of a functional connectivity network with respect to the hippocampus or the midline of the corpus callosum, for example.

Evaluation of visualization techniques
Answers to questions related to the 3D and stereoscopic images are summarized in Table 3. The mean score for the 19 questions, across the 14 healthcare professionals is 2.5 (range: 1.5-3.3) for 3D images and 4.3 (range: 3.8-4.8) for stereoscopic versions with a standard deviation of 2.6 and 0.9, respectively. Using a Student's t-Test (α = 0.05) the difference between 3D and stereoscopic visualization was statistically significant [t (18) = 9.88, p = 1.07E-08]. Stereoscopic visualizations were consistently ranked higher by raters as indicated by a Fleiss' Kappa = 0.255, p < 0.001 (Altman, 1991).
Importantly, stereoscopy may not be an ideal solution for all visualizations. Specifically, some individuals may have problems with 3D perception. Stereoblindness is the complete deficiency of stereopsis, and is often caused by strabismus (crossed-eye), amblyopia (lazy eye), or blindness in one eye (Lipton, 1982). Lipton (1982) specified that 2% of people are stereo-blind, and another 2% or 3% of healthy individuals suffer fatigue and discomfort when looking at stereo projections.
Additionally, Sherman (1953) found that approximately 2% of the population has hyperphoria, an upward deviation of the visual axis of one eye relative to the other, which prevents the visual axes of the eyes from crossing at any point in space and hence precludes stereoscopy. Total or partial color-blindness can also produce some limitations for viewing in 3D using anaglyphbased methods or other similar ones, and has a prevalence of 1.3% (Lipton, 1982;Pokorny and Smith, 1986). Beyond these various conditions, additional considerations are the potential discomfort of wearing 3D stereoscopic glasses for those that do not wear glasses regularly, and the need to overlay them over corrective lenses for those who do wear glasses.
Our evaluation shows that a diverse array of healthcare professionals that explored the 3D and stereoscopic images perceived the increased dimensionality of stereoscopy as beneficial. Depth of tumors, relative position of MS lesions, relative position, and size of both hippocampi, and relative position of different functional connectivity areas in resting state images were better perceived in stereoscopic images than in common 3D images (percentage difference greater than 30%). Higher percentage difference was obtained for the tumor related images (more that 98%) and lower percentage difference (lower than 20%) for images that include a functional area in close contact with a tumor (for example Figure 7).
In conclusion, the present work illustrates the potential utility of 3D stereoscopic visualization approaches in visualizing complex imaging findings. Using several imaging approaches (e.g., task activation fMRI, fMRI-based functional connectivity, morphometry, and tissue segmentation) and findings (e.g., cyst, tumor, multiple sclerosis), we demonstrated the ease with which stereoscopy can be applied to imaging data to provide more realistic and informative representations of brain structure and function. In particular, stereoscopy uniquely provided intuitive descriptions of the exact location and the relative size of various normal elements and lesions in a 3D representation, and facilitated comprehension of the anatomical position of complex large-scale functional connectivity patterns. Novel 3D visualization techniques can enhance the utility of imaging data for clinical applications and offer, through their intuitive nature, substantial educational value. Future work will benefit from further quantitative evaluation of the benefits of 3D stereoscopic visualization of neuroimages.