Editorial: Reliability and Reproducibility in Functional Connectomics

1 Key Laboratory of Brain and Education, Nanning Normal University, Nanning, China, Department of Psychology, University of Chinese Academy of Science, Beijing, China, CAS Key Laboratory of Behavioral Sciences, Institute of Psychology, Beijing, China, Magnetic Resonance Imaging Research Center, CAS Institute of Psychology, Beijing, China, 5 Research Center for Lifespan Development of Mind and Brain, CAS Institute of Psychology, Beijing, China, 6 Institute for Brain Research and Rehabilitation, South China Normal University, Guangzhou, China, Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ, United States, Department of Psychology, Stanford University, Stanford, CA,

Given the sensitivity of resting-state fMRI (rfMRI) connectivity measurements to physiological variables, the development of improved strategies for correction of physiological artifacts is imperative. Golestani et al. demonstrated significant improvements of reproducibility of common rfMRI metrics by the low-frequency physiological correction with end-tidal CO 2 . Related to human arousal, as demonstrated in Wang et al., test-retest reliability of human functional connectomics can be significantly improved by removing the impact of sleep using measures of heart rate variability derived from simultaneous electrocardiogram recording. These findings highlight the need of recordings of physiological variables for reproducible functional connectomics. In addition, the use of eyes-open versus eyes-closed resting is an important aspect of rfMRI experimental design and has been of great research interest due to its relationships with visual function (Yang et al., 2007) and arousal (Yan et al., 2009;Tagliazucchi and Laufs, 2014). The study by Yuan et al. provides a novel multivariate method to examine the amplitude differences of brain oscillations between eyes open and eyes close conditions during resting state as well as their scanner-related reliability. Head motion during scanning is another potential source of variability and has been relatively well investigated regarding its impacts on reliability of rfMRI derivatives by using various preprocessing strategies (Yan et al., 2013;Ciric et al., 2017;Parkes et al., 2018). Furthermore, how these variables are modeled and the order in the preprocessing pipelines they are modeled can have significant impacts on results (Chen et al., 2017;Lindquist et al., 2019). These advances have implications on the way of further optimizing the reliability observed (Golestani et al.; Wang et al.).
Many computational algorithms exist for characterizing features of the organization in the functional connectomes across different spatial and temporal scales (Zuo and Xing, 2014). Reliability can guide both methodological choices between these algorithms as well as the validation of new algorithms. Common algorithms have been recently given a state of art review in terms of their test-retest reliability (Zuo and Xing, 2014), indicating that network metrics derived from graph theory applied to rfMRI signal are less reliable (Zuo et al., 2012) than usually required while both local functional homogeneity measure (Zuo et al., 2013) and global network measure with dual regression of independent component analysis (drICA) (Zuo et al., 2010a) almost reach the clinical standard of reliability. This topic offers five studies to illustrate more sophisticated developments of reliability of these algorithms. This topic proposed a novel algorithm for network generation at individual level, using topological filtering based on orthogonal minimal spanning trees to show both functional and structural networks with highly reliable graph theoretical measures using magnetoencephalography (Dimitriadis et al.  (Salman et al., 2018). Additionally, Di and Biswal warned the field by demonstrating the poor reliability of using psychophysiological interaction analyses in the context of interindividual correlation or group comparisons.
As commented by Sato et al., open science with sharing of large datasets has paved the way for delineating the fingerprints of human brain function. This is reflected by the fact that most studies in the topic employed the data from Consortium for Reliability and Reproducibility , representing a means of accelerating science by facilitating collaboration, transparency, and reproducibility (Milham et al., 2018). To address the reproducibility issue in the field of human brain mapping, the Organization for Human Brain Mapping (OHBM) have created a Committee on Best Practices in Data Analysis and Sharing (COBIDAS) and published its report (Nichols et al., 2017). Beyond the advances, two studies also raised challenges of big-data applications to clinical population, particularly in understanding the high heterogeneity of spontaneous brain activity in ADHD and autism (Wang et al.; Syed et al.). As noted in Button et al. (2013), large samples may produce statistically significant results even for extremely small effects which have little add to diagnostic or clinical utility. These observable but small effects are likely caused by weighing the low measurement reliability with the true effect (Streiner et al., 2015), which could be moderate to large. It is thus very fundamental to estimate effect size in neuroimaging and its relationship with statistical power although most existing studies have not factored the reliability in doing so (Reddan et al., 2017;Geuter et al., 2018). This is particularly valuable for some widely used but less reliable measures (e.g., seed-based functional connectivity) (Shou et al., 2013;Zuo and Xing, 2014;Siegel et al., 2017) to be improved with acceptable reliability ahead of its clinical use (Fox, 2018). Meanwhile, data harmonization techniques such as ComBat (Yu et al., 2018) should be developed to reduce inter-scan or intersite differences in multi-center big-data studies. One possibility of filling these gaps between empirical computation and clinical application is theoretical development of brain dynamics (Woo et al., 2017). The work by Tomasi et al. demonstrated a power law of the brain network dynamics, which has been framed into a theory of neural oscillations (Buzsáki and Draguhn, 2004). Combination of theory and data via structure-function fusion (Zuo et al., 2010b;Jiang and Zuo, 2016) will remove the reliability barriers of developing clinically useful human brain mapping, which is the final call of the current research topic.

AUTHOR CONTRIBUTIONS
X-NZ drafted the editorial and worked on the revisions with BB and RP.

ACKNOWLEDGMENTS
We would like to thank Dr. Xiu-Xia Xing from School of Applied Sciences, Beijing University of Technology for her work on drafting the first version of this editorial as well as highly valuable comments on the importance of reliability to research and clinical implications.