Capturing Human Interaction in the Virtual Age: A Perspective on the Future of fNIRS Hyperscanning
- 1Department of Psychiatry and Behavioral Sciences, School of Medicine, Stanford University, Stanford, CA, United States
- 2Hasso Plattner Institute of Design, Stanford University, Stanford, CA, United States
- 3Department of Radiology, School of Medicine, Stanford University, Stanford, CA, United States
- 4Department of Pediatrics, School of Medicine, Stanford University, Stanford, CA, United States
Advances in video conferencing capabilities combined with dramatic socio-dynamic shifts brought about by COVID-19, have redefined the ways in which humans interact in modern society. From business meetings to medical exams, or from classroom instruction to yoga class, virtual interfacing has permeated nearly every aspect of our daily lives. A seemingly endless stream of technological advances combined with our newfound reliance on virtual interfacing makes it likely that humans will continue to use this modern form of social interaction into the future. However, emergent evidence suggests that virtual interfacing may not be equivalent to face-to-face interactions. Ultimately, too little is currently understood about the mechanisms that underlie human interactions over the virtual divide, including how these mechanisms differ from traditional face-to-face interaction. Here, we propose functional near-infrared spectroscopy (fNIRS) hyperscanning—simultaneous measurement of two or more brains—as an optimal approach to quantify potential neurocognitive differences between virtual and in-person interactions. We argue that increased focus on this understudied domain will help elucidate the reasons why virtual conferencing doesn't always stack up to in-person meetings and will also serve to spur new technologies designed to improve the virtual interaction experience. On the basis of existing fNIRS hyperscanning literature, we highlight the current gaps in research regarding virtual interactions. Furthermore, we provide insight into current hurdles regarding fNIRS hyperscanning hardware and methodology that should be addressed in order to shed light on this newly critical element of everyday life.
The COVID-19 pandemic has dramatically disrupted the daily lives of much, if not all, of the world's population. Overnight, in-person social interactions have been replaced by video conferencing. Today “Zoom meetings” are commonplace and have largely allowed us to continue engaging in our daily routines. Indeed, in the weeks after COVID-19 emerged across the globe downloads of videoconferencing apps increased by >90% of the 2019 average (AppAnnie.com 2020). Since then, videoconferencing has been a vital tool for business, medicine, education, and social interactions alike. Despite our ability to stay “connected,” there is both empirical and anecdotal evidence to suggest that these mediums are inadequate substitutes for traditional in-person social interactions. For example, virtual interactions have been shown to have adverse effects on emotional and mental health (Holmes et al., 2020; Pfefferbaum and North, 2020), education outcome (Ahmed et al., 2020; Schwartz et al., 2020), and medical care service (Hollander and Carr, 2020; Pappot et al., 2020). Moreover, a glut of popular press articles lamenting the negative effects of “Zoom fatigue” in its many forms (BBC April 22, 2020; National Geographic, April 24, 2020; New York Times, May 4, 2020; the Wall Street Journal, June 5, 2020) are testimony to the negative impact that this new form of communication may have on human-to-human interaction.
These reports are concerning given that video conferencing is likely to play a significant role in human's lives for the foreseeable future (Van Bavel et al., 2020). Critically, too little is currently understood about the underlying neurocognitive mechanisms that result in the adverse effects reported above (e.g., increase in social isolation, decrease in learning outcome, increase in fatigue, etc.). In fact, to our knowledge, there is currently no study that directly compares the differences in neural signatures of social interactions between virtual and in-person interactions. We argue that it is critically important to understand these differences in neural mechanisms that underlie digital human-to-human interaction, and specifically how these neural mechanisms may differ from traditional in-person interactions. We propose functional near-infrared spectroscopy (fNIRS) hyperscanning (i.e., measuring two or more brains simultaneously as they interact socially) as a tool to quantify and understand the potential differences between virtual and in-person interactions. As we argue below, fNIRS hyperscanning may provide an ideal approach to elucidate the neurocognitive differences between virtual and in-person interactions that may result from changes in social behavior (e.g., eye-to-eye contact), from differences in environmental information (e.g., disparate background/foreground lighting), and/or from technological parameters (e.g., unequal frame rates). A clear understanding of the underlying neural mechanisms could inform the development of behavioral interventions and/or the design and engineering of technology that help to mitigate adverse effects. For example, imagine brief yet highly-effective pro-social behavioral exercises that combat social isolation or software that simply synchronizes frame rates to decrease fatigue during virtual teaching/learning activities.
There is conceptual and empirical evidence that social cognition is fundamentally different when we are in interaction with others rather than merely observing them (Schilbach et al., 2013). Hyperscanning technology has allowed us to shed light onto the neural processes underpinning social cognition (Babiloni and Astolfi, 2014; Wang et al., 2018). Over the past decade the field of hyperscanning with functional near-infrared spectroscopy has increased dramatically and has provided unique insight into signatures of brain-to-brain connectivity that are invisible to the naked eye (Dumas et al., 2011; Babiloni and Astolfi, 2014; Redcay and Schilbach, 2019). Specifically, fNIRS hyperscanning has highlighted inter-brain coherence (i.e., correlation of cortical activity between brains) that occurs during social interactions, such as cooperation (Cui et al., 2012; Yang et al., 2020), and is often associated with enhanced behavioral metrics of interaction (Baker et al., 2016). Importantly, given fNIRS' relatively robust tolerance to movement and methodological flexibility, hyperscanning in this modality allows researchers to observe the neural correlates of shared human neural activity in naturalistic environments that are often not feasible in other modalities, such as fMRI or EEG (Scholkmann et al., 2013; Baker et al., 2017; Quaresima and Ferrari, 2019; Gvirts and Perlmutter, 2020). The dramatic increase in fNIRS hyperscanning research has spurred the publication of several systematic reviews, to which we refer the interested reader (Babiloni and Astolfi, 2014; Wang et al., 2018; Czeszumski et al., 2020). In this paper, we focus on providing a review of methodology used in fNIRS hyperscanning research and provide a novel framework to help guide the development of future studies for advancing the field toward capturing human interaction in the virtual age.
2. Deriving an fNIRS Hyperscanning Framework
We executed a keyword search via Google Scholar and PubMed up to May 15, 2020 that included the following keywords: “fNIRS hyperscanning” and “NIRS hyperscanning.” For each search engine, we inspected the first 250 entries for each keyword category and checked the reference lists of the included articles for any additional relevant articles. We included journal and conference articles in the English language only, resulting in a total of 69 fNIRS hyperscanning studies. For the scope of this paper, we focused only on those studies that investigated interaction between adults. As such, we excluded nine infant-parent fNIRS hyperscanning studies (Leong et al., 2017; Reindl et al., 2018; Azhari et al., 2019, 2020; Miller et al., 2019; Quiñones-Camacho et al., 2019; Behrendt et al., 2020; Nguyen et al., 2020; Piazza et al., 2020). Furthermore, we excluded two papers that included comparisons of temporally non-congruent fNIRS scans (Liu Y et al., 2017; Hou et al., 2020), resulting in a total of 58 fNIRS hyperscanning papers (see Table 1 for an overview). From each of the resulting 58 fNIRS hyperscanning papers, we extracted all experimental conditions (i.e., “hyperscan” conditions) that were utilized and from which data were analyzed.
In order to find a consistent methodological structure across the resulting 151 hyperscans, two researchers (SB and JMB) executed a thematic analysis. Two naturally occurring dimensions (i.e., Transfer of Information and Type of Communication) emerged from each scan. First, Transfer of Information (ToI) refers to the interface through which human-to-human interaction was conveyed. We clustered TOI into three levels: (1) hyperscans that comprised human-to-human interaction in a face-to-face setting (i.e., Analog), where no digital medium was present; (2) hyperscans that comprised a combination of analog and digital transfer methods (i.e., Mixed ToI), such as sitting side-by-side while problem solving on a computer screen; and (3) hyperscans in which all interactions were made via technology (i.e., Digital). Next, Type of Communication (ToC) refers to the objective of the interaction and varied between Joint goal-driven, Joint open-ended, and Mixed ToC interactions. For this classification, any hyperscanning task that had an explicit and clearly defined objective, goal, or competitive outcome (e.g., zero-sum game), or one that compared task accuracy or response time was classified as goal-driven. Conversely, any task that required no explicit objective (e.g., chitchat between therapist and client before the therapy session) was classified as open-ended. Scans that contained elements of both (e.g., two participants cooperate while a 3rd watches) was defined as mixed ToC.
As shown in Figure 1, the distribution of hyperscan conditions dedicated to each of the nine categories defined by our framework is highly unequal. Over half of all reported hyperscan conditions (57.8%, N = 87) were conducted when the interacting dyad were in the same room without any means of digital interaction (i.e., Analog ToI) (Funane et al., 2011; Holper et al., 2012, 2013; Jiang et al., 2012, 2015; Osaka et al., 2014, 2015; Duan et al., 2015; Liu N et al., 2016; Nozawa et al., 2016, 2019; Hirsch et al., 2017, 2018; Ikeda et al., 2017; Zhang et al., 2017a,b; Zhao et al., 2017; Dai et al., 2018a,b; Fishburn et al., 2018; Pan et al., 2018, 2020a; Xue et al., 2018; Zhang Y et al., 2018; Lu et al., 2019, 2020; Mayseless et al., 2019; Niu et al., 2019; Vanzella et al., 2019; Noah et al., 2020), while (42.4%, N = 64) included some element of technology (e.g., playing a computer game) while participants were in the same room (Cui et al., 2012; Dommer et al., 2012; Duan et al., 2013; Cheng et al., 2015, 2019; Liu T et al., 2015, 2016, 2017; Baker et al., 2016; Tang et al., 2016; Balconi and Vanutelli, 2017a,b; Hu et al., 2017; Pan et al., 2017, 2020b; Piva et al., 2017; Takeuchi et al., 2017; Fishburn et al., 2018; Zhang M et al., 2018; Zheng et al., 2018, 2020; Balconi et al., 2019; Liu et al., 2019; Nozawa et al., 2019; Sarinasadat et al., 2019a,b; Balconi and Fronda, 2020; Feng et al., 2020; Noah et al., 2020; Sun et al., 2020; Yang et al., 2020). Notably, no study employed a method in which both interacting members of a dyad were entirely physically separated, such that all interactions were conducted via technology (e.g., virtual meeting). A similar lack of empirical focus currently exists for open-ended ToC. Only 3.3% of all scans (N = 5) focused on inter-brain synchrony during open-ended communication (Liu N et al., 2016; Ikeda et al., 2017; Fishburn et al., 2018; Zhang Y et al., 2018; Yang et al., 2020), whereas 77.5% (N = 117) focused on Joint goal-driven interactions. The remaining 19.2% (N = 29) tasks contained elements of both communication types (Cui et al., 2012; Jiang et al., 2012; Holper et al., 2013; Osaka et al., 2014, 2015; Cheng et al., 2015; Liu T et al., 2015, 2016, 2017; Fishburn et al., 2018; Hirsch et al., 2018; Niu et al., 2019; Vanzella et al., 2019).
Figure 1. The figure above visualizes the cross-sectional distribution of all 151 conducted hyperscan conditions across the Transfer of Information (ToI) and Type of Communication (ToC) axes. The color of each circle provides the number of scans that belong to each cross-sectional condition. Light colors indicate fewer scans and darker colors indicate more scans. The lines indicate the cross-condition comparisons that were reported. The width of each line provides an illustration of the number of scans conducted within each cross-condition comparison.
The thickness of the lines in Figure 1 represent the frequency of cross-condition comparisons reported. A cross-condition comparison occurred when the ToI or ToC during a hyperscan differed between experimental tasks. A total of 19 (33.3%) papers included in our analysis included one or more cross-condition comparisons. However, the classification of the comparisons reported were limited to Analog vs. Mixed and Inter-ToI comparisons. The studies comprised comparisons between Mixed ToI/Joint goal-driven ToC and Mixed ToI/Mixed ToC (N = 5) (Cui et al., 2012; Cheng et al., 2015; Liu T et al., 2015, 2016, 2017), Analog ToI/Mixed ToC and Analog ToI/Joint goal-driven ToC (N = 7) (Jiang et al., 2012; Holper et al., 2013; Osaka et al., 2014, 2015; Hirsch et al., 2018; Niu et al., 2019; Vanzella et al., 2019), Analog ToI/Joint open-ended ToC and Analog ToI/Joint goal-driven ToC (N = 3) (Liu N et al., 2016; Ikeda et al., 2017; Zhang Y et al., 2018), and Analog ToI/Mixed ToC and Mixed ToI/Joint goal-driven ToC (N = 2) (Nozawa et al., 2019; Noah et al., 2020). Two studies included three cross-condition comparisons, including comparisons between Analog ToI/Mixed ToC, Analog ToI/Joint goal-driven ToC, and Mixed ToI/Joint open-ended ToC (N = 1) (Fishburn et al., 2018), as well as between Analog ToI/Joint goal-driven ToC, Analog ToI/Joint open-ended ToC, and Mixed ToI/Joint goal-driven ToC (N = 1) (Yang et al., 2020).
3. Existing fNIRS Hyperscanning Hurdles
Taken together, our analysis highlights the areas of study that have received little to no attention. Specifically, no fNIRS hyperscanning study has, to date, focused on understanding pure Digital ToI (i.e., virtual meeting) nor has any study focused on comparing Digital ToI with Analog ToI (i.e., in-person meeting). Similarly, Joint open-ended ToC (e.g., chit chat with a friend via zoom) has received very little empirical attention.
The lack of focus on Digital ToI has likely been due, in part, to technological or methodological shortcomings that constrain this line of research. For instance, many fNIRS devices do not easily accommodate a digital hyperscanning interface, which would ostensibly take place in separate rooms so that no in-person communication may occur. While it may be feasible, for example, to build a structure that splits optodes of one device allowing to scan two distant participants, this may be unrealistic for researchers in many instances. Thus, when faced with this challenge, even interested researchers may find such methodology prohibitively difficult. One alternative may be the use of two individual fNIRS devices, each positioned in their own room. However, aside from cost-related drawbacks, in this instance researchers must be able to accurately sync the time series' recorded from both devices in order to facilitate downstream processing and analysis of their data. This may require the development of sophisticated software to sync and timestamp event markers wirelessly across both devices. Notably, while promising examples for such analytical tools do exist (e.g., Labstreaminglayer), there is currently no readily available tool designed specifically for fNIRS hyperscanning. We argue that more effort is needed to develop and disseminate such analytical tools via peer-reviewed publication and open-source file sharing. Alternatively, researchers may video record both members of a separated dyad to capture events, then code the event timestamps post-hoc. This procedure is useful but requires a considerable amount of time and manual effort. Moreover, such procedures should be performed in tandem, so that inter-rater reliability may be established. It is our hope that advances within the community will help overcome this hardware hurdle in order to facilitate the study of the digital ToI domain.
The lack of data within the open-ended ToC domain may be less due to technological drawbacks, and more due to a lack of established analytical approaches to tasks that are not trial based. To quantify and analyze brain-to-brain coupling, researchers have applied more traditional statistical approaches, such as block-averaging (e.g., Holper et al., 2013); analysis of co-variance (e.g., Funane et al., 2011); and correlation analysis (e.g., Duan et al., 2013). Cui et al. (2012) introduced a novel analytical approach for fNIRS hyperscanning (i.e., Wavelet Transform Analysis or “WTC”), wherein the coherence and phase lag in two time series is assessed across both time and frequency. By contrasting the average task-related coherence during the task (i.e., cooperation paradigm) and rest, the authors demonstrated an increase in coherence during cooperation that dissipated during rest. Wavelet coherence analysis has been widely adopted within the fNIRS hyperscanning research (as shown in Table 1, roughly 70% of all studies included WTC analysis), and there are efforts to further improve WTC's efficacy (Zhang et al., 2020). However, while the method was originally developed for block-design studies in which a task frequency band and condition markers may be identified, it currently lacks the ability to derive instant and fluctuating components of social interactions. Recent approaches (e.g., Mayseless et al., 2019) have therefore attempted to develop novel analytical methods that do not rely on task blocks, and which may be applicable to open-ended task designs. Finally, Granger Causality, a method that allows for the derivation of directionality of synchrony between two time series, has also been shown to be a useful analytical approach to investigate the fluctuations of interactive dynamics between individuals (Holper et al., 2012). Similar to WTC, further advances in Granger Causality analysis might allow for investigations of fluctuating social dynamics during joint open-ended interactions. It will be important for future research to build upon these approaches, and to develop algorithms and techniques to better facilitate analysis of hyperscanning data.
4. A Perspective of the Future Potential of fNIRS Hyperscanning
The structure presented in Figure 1 is reminiscent of a similar framework that was introduced earlier in this journal (Liu and Pelowski, 2014). Specifically, Liu and Pelowski (2014) proposed a framework that distinguished between task structure (interdependent vs. independent), interaction structure (concurrent vs. turn-based), and goal structure (cooperative vs. competitive) as variables that hyperscanning studies should consider during task design. As the field of fNIRS hyperscanning progresses toward Real-life Neuroscience (Shamay-Tsoory and Mendelsohn, 2019; Holleman et al., 2020), the need for an updated framework that includes virtual social interactions (i.e., Digital ToI) as well as open-ended interactions (i.e., Joint open-ended ToC) is warranted. We propose that our updated framework, as depicted in Figure 2A, can help guide hyperscanning researchers toward a future where all forms of human-to-human social interactions are fairly represented. In order to achieve equal distributions, the community has to overcome the current hurdles as described above. These hurdles include, but are not limited to, developing methodological designs that address each condition in Figure 2A, hardware that is amenable to hyperscanning when participants are separated physically, and software that is capable of managing back-end data streams of such tasks. It is our hope that both hardware and software will be flexible enough to approach more and more realistic scenarios in which complex and sudden social interactions can be captured (see Figure 2B).
Figure 2. (A) This matrix provides a schematic of all nine possible intersections of ToI and ToC within our framework. The schematic shows three hypothetical tasks being conducted across each intersection. First, data analysis (denoted by the bar chart) provides an example of a Joint goal-directed ToC. Next, the instance of one person performing (denoted by the star) while one or more people watch passively (denoted by the eye), provides an example of a Mixed ToC task. Finally, friendly chit-chat (denoted by the chat bubbles) provides an example of a Joint open-ended ToC. Importantly, each of these activities may be conducted under Analog, Mixed, or Digital ToI. (B) This schematic demonstrates a hypothetical 3-person hyperscan that fluctuates continuously across time through multiple domains outlined in our framework. First, a pair of participants situated in the same room engage in open-ended conversation for a period of time (1). Next, a third participant joins the pair via a live video feed, which introduces a mixed digital interface between the three participants (2). Following a period of chit-chat, the triad begins work on a goal-driven task together (3). Next, one of the two participants situated together exits, leaving an interacting pair separated by a digital divide that work together on a goal-driven task (4). These participants continue to work on the goal-driven task until completion (5).
Efforts from the broader fNIRS community will be required to make fNIRS truly ready for realistic scenarios. With respect to hardware, this includes increased device portability and robustness (e.g., with respect to movement and environmental light), increased optode number to cover more cortical areas, and short-channels to account for extra-cerebellar blood flow that may contaminate fNIRS signals (Brigadoi and Cooper, 2015; Baker et al., 2017; Herold et al., 2017). Furthermore, efforts should be made with respect to standardizing fNIRS procedures, such as optode placement, data processing, choice of activation proxy (i.e., oxy- vs. de-oxygenated hemoglobin) (Brigadoi et al., 2014; Tachtsidis and Scholkmann, 2016; Herold et al., 2017; Di Lorenzo et al., 2019), and adoption of standardized open-source fNIRS-specific data analysis packages (e.g., HOMER2, NIRS SPM, nirsLAB, open-potato, etc.).
While adherence to our framework will help to more completely elucidate the neurobiological signatures of human-to-human interactions across all platforms, future research in this field will not be without limitations. Primarily, this includes the cortical depth at which fNIRS may sample while maintaining acceptable signal quality. While efforts have been made to infer deep-brain activity using fNIRS (Liu N et al., 2015), the relatively low sampling depth of ~3cm (Brigadoi and Cooper, 2015) limits the neurocognitive functions that may be directly measured by fNIRS. As shown in Table 1, the existing fNIRS hyperscanning research has focused on studying cognitive functions within cortical regions underlying attention, executive function, language, social cognition, visuospatial processing, and motor activity. Methodological approaches to the existing fNIRS hyperscanning studies have been diverse and focused on social interactions during simple motor-synching (e.g., Holper et al., 2012), cooperative and competitive gameplay (e.g., Cui et al., 2012), unstructured and structured conversation including singing (e.g., Osaka et al., 2014), teaching activities (e.g., Nozawa et al., 2019), and creative problem solving (e.g., Lu et al., 2019). Studies also tested for effects of moderators, such as sex (Cheng et al., 2015), level of acquaintance (Pan et al., 2017), eye-to-eye contact (e.g., Hirsch et al., 2017), and pro-social priming effects (e.g., Balconi et al., 2019) on inter-brain cognitive functioning and task outcome. In fact, the methodological flexibility afforded by fNIRS is so great that researchers may run the risk of creating methods that are so creative as to be difficult to interpret, replicate or compare. Therefore, we encourage researchers in the immediate future to parsimoniously advance into the understudied areas of our framework (i.e., Digital ToI and JoinT open-ended ToC). For instance, it may be useful to commence the study of differences between virtual and in-person interactions with established hyperscanning tasks, such as simple computer-based cooperation tasks (Cui et al., 2012). In this manner, researchers may directly investigate the effect of ToC on inter-brain coherence and are further able to compare new data with existing outcomes (i.e., confirmatory science). Another interesting inroad could be to extend the study of differences in social cognition between “observing others” and “actually interacting with them” (Schilbach et al., 2013) to video/virtual interactions. In that case, prior fNIRS studies assessing the temporally non-congruent inter-brain coherence of video-recorded individuals and spectators (who watch the videos at a later stage) could serve as entry points (Liu Y et al., 2017; Hou et al., 2020).
Ultimately, multi-dimensional data approaches will allow us to determine which parameters (i.e., behavioral, environmental, and/or technological) are most explanatory with respect to potential differences in neurocognitive signatures between virtual and in-person interactions. For example, using congruent fNIRS-EEG systems will improve temporal resolution. Physiological metrics (e.g., heart rate, heart rate variability, galvanic skin response, pupil dilation, etc.) along with behavioral measures (e.g., eye-gaze-tracking, body-motion tracking, analysis of voice, emotional face tracking, etc.) will provide vital information to better understand the humans' psychophysiological response during social interactions. Lastly, the monitoring of environmental information (e.g., ambient noise, reflecting light on reading glasses, etc.) and technological parameters (e.g., computer frame-rate, computer audio, internet speed, computer screen activity, etc.) will be essential to control and account for potential external biases.
The future of fNIRS hyperscanning is limitless and very well may be a key component of our understanding of the neurobiological underpinnings of social behavior. From tele-health to tele-education, and from internet dating to online gaming, technology driven activities will likely play a ubiquitous role in our social interactions moving forward. The framework presented here is meant to advance discussion among researchers in their study of all aspects of human interaction, including those that technology has yet to make possible.
Data Availability Statement
All datasets generated for this study are included in the article/supplementary material.
SB: conceptualization, literature review, methodology, and writing. JMB: conceptualization, methodology, and writing. GH: conceptualization. ALR: conceptualization, methodology, supervision, and writing. All authors contributed to the article and approved the submitted version.
This work was supported by a Hasso Plattner Design Thinking Research Program grant. This research was also supported in part by National Institutes of Health grants HD092883 (JMB, Principal Investigator), MH099630, HD049653, and T32MH019908 (ALR, Principal Investigator), a grant from the Stanford Maternal and Child Health Research Institute (JMB, Principal Investigator), and gifts from the Albert Yu and Mary Bechmann Foundation, the Feldman Foundation, and Peter Georgescu.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Azhari, A., Gabrieli, G., Bizzego, A., Bornstein, M. H., and Esposito, G. (2020). Maternal anxious attachment style is associated with reduced mother-child brain-to-brain synchrony. bioRxiv. doi: 10.1101/2020.01.23.917641
Azhari, A., Leck, W., Gabrieli, G., Bizzego, A., Rigo, P., Setoh, P., et al. (2019). Parenting stress undermines mother-child brain-to-brain synchrony: a hyperscanning study. Sci. Rep. 9:11407. doi: 10.1038/s41598-019-47810-4
Baker, J. M., Liu, N., Cui, X., Vrticka, P., Saggar, M., Hosseini, S. H., et al. (2016). Sex differences in neural and behavioral signatures of cooperation revealed by fnirs hyperscanning. Sci. Rep. 6:26492. doi: 10.1038/srep26492
Baker, J. M., Rojas-Valverde, D., Gutiérrez, R., Winkler, M., Fuhrimann, S., Eskenazi, B., et al. (2017). Portable functional neuroimaging as an environmental epidemiology tool: a how-to guide for the use of fnirs in field studies. Environ. Health Perspect. 125:094502. doi: 10.1289/EHP2049
Balconi, M., and Fronda, G. (2020). The “gift effect” on functional brain connectivity. Inter-brain synchronization when prosocial behavior is in action. Sci. Rep. 10:5394. doi: 10.1038/s41598-020-62421-0
Balconi, M., and Vanutelli, M. E. (2017a). Brains in competition: improved cognitive performance and inter-brain coupling by hyperscanning paradigm with functional near-infrared spectroscopy. Front. Behav. Neurosci. 11:163. doi: 10.3389/fnbeh.2017.00163
Behrendt, H. F., Konrad, K., Perdue, K. L., and Firk, C. (2020). Infant brain responses to live face-to-face interaction with their mothers: combining functional near-infrared spectroscopy (fNIRS) with a modified still-face paradigm. Infant Behav. Dev. 58:101410. doi: 10.1016/j.infbeh.2019.101410
Brigadoi, S., Ceccherini, L., Cutini, S., Scarpa, F., Scatturin, P., Selb, J., et al. (2014). Motion artifacts in functional near-infrared spectroscopy: a comparison of motion correction techniques applied to real cognitive data. Neuroimage 85, 181–191. doi: 10.1016/j.neuroimage.2013.04.082
Brigadoi, S., and Cooper, R. J. (2015). How short is short? Optimum source-detector distance for short-separation channels in functional near-infrared spectroscopy. Neurophotonics 2:025005. doi: 10.1117/1.NPh.2.2.025005
Cheng, X., Li, X., and Hu, Y. (2015). Synchronous brain activity during cooperative exchange depends on gender of partner: a fNIRS-based hyperscanning study. Hum. Brain Mapp. 36, 2039–2048. doi: 10.1002/hbm.22754
Cui, X., Bryant, D. M., and Reiss, A. L. (2012). Nirs-based hyperscanning reveals increased interpersonal coherence in superior frontal cortex during cooperation. Neuroimage 59, 2430–2437. doi: 10.1016/j.neuroimage.2011.09.003
Czeszumski, A., Eustergerling, S., Lang, A., Menrath, D., Gerstenberger, M., Schuberth, S., et al. (2020). Hyperscanning: a valid method to study neural inter-brain underpinnings of social interaction. Front. Hum. Neurosci. 14:39. doi: 10.3389/fnhum.2020.00039
Dai, B., Chen, C., Long, Y., Zheng, L., Zhao, H., Bai, X., et al. (2018a). Neural mechanisms for selectively tuning in to the target speaker in a naturalistic noisy situation. Nat. Commun. 9:2405. doi: 10.1038/s41467-018-04819-z
Dai, R., Liu, R., Liu, T., Zhang, Z., Xiao, X., Sun, P., et al. (2018b). Holistic cognitive and neural processes: a fNIRS-hyperscanning study on interpersonal sensorimotor synchronization. Soc. Cogn. Affect. Neurosci. 13, 1141–1154. doi: 10.1093/scan/nsy090
Di Lorenzo, R., Pirazzoli, L., Blasi, A., Bulgarelli, C., Hakuno, Y., Minagawa, Y., et al. (2019). Recommendations for motion correction of infant fNIRS data applicable to data sets acquired with a variety of experimental designs and acquisition systems. Neuroimage 200, 511–527. doi: 10.1016/j.neuroimage.2019.06.056
Dommer, L., Jäger, N., Scholkmann, F., Wolf, M., and Holper, L. (2012). Between-brain coherence during joint N-back task performance: a two-person functional near-infrared spectroscopy study. Behav. Brain Res. 234, 212–222. doi: 10.1016/j.bbr.2012.06.024
Duan, L., Dai, R., Xiao, X., Sun, P., Li, Z., and Zhu, C. (2015). Cluster imaging of multi-brain networks (CIMBN): a general framework for hyperscanning and modeling a group of interacting brains. Front. Neurosci. 9:267. doi: 10.3389/fnins.2015.00267
Duan, L., Liu, W.-J., Dai, R.-N., Li, R., Lu, C.-M., Huang, Y.-X., et al. (2013). Cross-brain neurofeedback: scientific concept and experimental platform. PLoS ONE 8:e64590. doi: 10.1371/journal.pone.0064590
Dumas, G., Lachat, F., Martinerie, J., Nadel, J., and George, N. (2011). From social behaviour to brain synchronization: review and perspectives in hyperscanning. IRBM 32, 48–53. doi: 10.1016/j.irbm.2011.01.002
Feng, X., Sun, B., Chen, C., Li, W., Wang, Y., Zhang, W., et al. (2020). Self-other overlap and interpersonal neural synchronization serially mediate the effect of behavioral synchronization on prosociality. Soc. Cogn. Affect. Neurosci. 15, 203–214. doi: 10.1093/scan/nsaa017
Fishburn, F. A., Murty, V. P., Hlutkowsky, C. O., MacGillivray, C. E., Bemis, L. M., Murphy, M. E., et al. (2018). Putting our heads together: interpersonal neural synchronization as a biological mechanism for shared intentionality. Soc. Cogn. Affect. Neurosci. 13, 841–849. doi: 10.1093/scan/nsy060
Funane, T., Kiguchi, M., Atsumori, H., Sato, H., Kubota, K., and Koizumi, H. (2011). Synchronous activity of two people's prefrontal cortices during a cooperative task measured by simultaneous near-infrared spectroscopy. J. Biomed. Opt. 16:077011. doi: 10.1117/1.3602853
Gvirts, H. Z., and Perlmutter, R. (2020). What guides us to neurally and behaviorally align with anyone specific? A neurobiological model based on fNIRS hyperscanning studies. Neuroscientist 26, 108–116. doi: 10.1177/1073858419861912
Herold, F., Wiegel, P., Scholkmann, F., Thiers, A., Hamacher, D., and Schega, L. (2017). Functional near-infrared spectroscopy in movement science: a systematic review on cortical activity in postural and walking tasks. Neurophotonics 4:041403. doi: 10.1117/1.NPh.4.4.041403
Hirsch, J., Adam Noah, J., Zhang, X., Dravida, S., and Ono, Y. (2018). A cross-brain neural mechanism for human-to-human verbal communication. Soc. Cogn. Affect. Neurosci. 13, 907–920. doi: 10.1093/scan/nsy070
Hirsch, J., Zhang, X., Noah, J. A., and Ono, Y. (2017). Frontal temporal and parietal systems synchronize within and across brains during live eye-to-eye contact. Neuroimage 157, 314–330. doi: 10.1016/j.neuroimage.2017.06.018
Holleman, G. A., Hooge, I. T., Kemner, C., and Hessels, R. S. (2020). The reality of “real-life” neuroscience: a commentary on Shamay-Tsoory and Mendelsohn (2019). Perspect. Psychol. Sci. doi: 10.1177/1745691620917354. [Epub ahead of print].
Holmes, E. A., O'Connor, R. C., Perry, V. H., Tracey, I., Wessely, S., Arseneault, L., et al. (2020). Multidisciplinary research priorities for the COVID-19 pandemic: a call for action for mental health science. Lancet Psychiatry 7, 547–560. doi: 10.1016/S2215-0366(20)30168-1
Holper, L., Goldin, A. P., Shalóm, D. E., Battro, A. M., Wolf, M., and Sigman, M. (2013). The teaching and the learning brain: a cortical hemodynamic marker of teacher-student interactions in the socratic dialog. Int. J. Educ. Res. 59, 1–10. doi: 10.1016/j.ijer.2013.02.002
Hou, Y., Song, B., Hu, Y., Pan, Y., and Hu, Y. (2020). The averaged inter-brain coherence between the audience and a violinist predicts the popularity of violin performance. Neuroimage 211:116655. doi: 10.1016/j.neuroimage.2020.116655
Hu, Y., Hu, Y., Li, X., Pan, Y., and Cheng, X. (2017). Brain-to-brain synchronization across two persons predicts mutual prosociality. Soc. Cogn. Affect. Neurosci. 12, 1835–1844. doi: 10.1093/scan/nsx118
Ikeda, S., Nozawa, T., Yokoyama, R., Miyazaki, A., Sasaki, Y., Sakaki, K., et al. (2017). Steady beat sound facilitates both coordinated group walking and inter-subject neural synchrony. Front. Hum. Neurosci. 11:147. doi: 10.3389/fnhum.2017.00147
Jiang, J., Chen, C., Dai, B., Shi, G., Ding, G., Liu, L., et al. (2015). Leader emergence through interpersonal neural synchronization. Proc. Natl. Acad. Sci. U.S.A. 112, 4274–4279. doi: 10.1073/pnas.1422930112
Leong, V., Byrne, E., Clackson, K., Georgieva, S., Lam, S., and Wass, S. (2017). Speaker gaze increases information coupling between infant and adult brains. Proc. Natl. Acad. Sci. U.S.A. 114, 13290–13295. doi: 10.1073/pnas.1702493114
Liu, J., Zhang, R., Geng, B., Zhang, T., Yuan, D., Otani, S., et al. (2019). Interplay between prior knowledge and communication mode on teaching effectiveness: interpersonal neural synchronization as a neural marker. Neuroimage 193, 93–102. doi: 10.1016/j.neuroimage.2019.03.004
Liu, N., Cui, X., Bryant, D. M., Glover, G. H., and Reiss, A. L. (2015). Inferring deep-brain activity from cortical activity using functional near-infrared spectroscopy. Biomed. Opt. Express 6, 1074–1089. doi: 10.1364/BOE.6.001074
Liu, N., Mok, C., Witt, E. E., Pradhan, A. H., Chen, J. E., and Reiss, A. L. (2016). Nirs-based hyperscanning reveals inter-brain neural synchronization during cooperative jenga game with face-to-face communication. Front. Hum. Neurosci. 10:82. doi: 10.3389/fnhum.2016.00082
Liu, T., Saito, G., Lin, C., and Saito, H. (2017). Inter-brain network underlying turn-based cooperation and competition: a hyperscanning study using near-infrared spectroscopy. Sci. Rep. 7:8684. doi: 10.1038/s41598-017-09226-w
Liu, T., Saito, H., and Oi, M. (2015). Role of the right inferior frontal gyrus in turn-based cooperation and competition: a near-infrared spectroscopy study. Brain Cogn. 99, 17–23. doi: 10.1016/j.bandc.2015.07.001
Liu, Y., Piazza, E. A., Simony, E., Shewokis, P. A., Onaral, B., Hasson, U., et al. (2017). Measuring speaker-listener neural coupling with functional near infrared spectroscopy. Sci. Rep. 7:43293. doi: 10.1038/srep43293
Miller, J. G., Vrtička, P., Cui, X., Shrestha, S., Hosseini, S. H., Baker, J. M., et al. (2019). Inter-brain synchrony in mother-child dyads during cooperation: an fNIRS hyperscanning study. Neuropsychologia 124, 117–124. doi: 10.1016/j.neuropsychologia.2018.12.021
Nguyen, T., Schleihauf, H., Kayhan, E., Mathes, D., Vrticka, P., and Hoehl, S. (2020). Neural synchrony in mother-child conversation: exploring the role of communicative features. Soc. Cogn. Affect. Neurosci. doi: 10.1093/scan/nsaa079. [Epub ahead of print].
Niu, R., Yu, Y., Li, Y., and Liu, Y. (2019). Use of fnirs to characterize the neural mechanism of inter-individual rhythmic movement coordination. Front. Physiol. 10:781. doi: 10.3389/fphys.2019.00781
Noah, J. A., Zhang, X., Dravida, S., Ono, Y., Naples, A., McPartland, J. C., et al. (2020). Real-time eye-to-eye contact is associated with cross-brain neural coupling in angular gyrus. Front. Hum. Neurosci. 14:19. doi: 10.3389/fnhum.2020.00019
Nozawa, T., Sakaki, K., Ikeda, S., Jeong, H., Yamazaki, S., dos Santos Kawata, K. H., et al. (2019). Prior physical synchrony enhances rapport and inter-brain synchronization during subsequent educational communication. Sci. Rep. 9:12747. doi: 10.1038/s41598-019-49257-z
Nozawa, T., Sasaki, Y., Sakaki, K., Yokoyama, R., and Kawashima, R. (2016). Interpersonal frontopolar neural synchronization in group communication: an exploration toward fNIRS hyperscanning of natural interactions. Neuroimage 133, 484–497. doi: 10.1016/j.neuroimage.2016.03.059
Osaka, N., Minamoto, T., Yaoi, K., Azuma, M., and Osaka, M. (2014). Neural synchronization during cooperated humming: a hyperscanning study using fNIRS. Proc. Soc. Behav. Sci. 126, 241–243. doi: 10.1016/j.sbspro.2014.02.395
Osaka, N., Minamoto, T., Yaoi, K., Azuma, M., Shimada, Y. M., and Osaka, M. (2015). How two brains make one synchronized mind in the inferior frontal cortex: fNIRS-based hyperscanning during cooperative singing. Front. Psychol. 6:1811. doi: 10.3389/fpsyg.2015.01811
Pan, Y., Dikker, S., Goldstein, P., Zhu, Y., Yang, C., and Hu, Y. (2020a). Instructor-learner brain coupling discriminates between instructional approaches and predicts learning. Neuroimage 211:116657. doi: 10.1016/j.neuroimage.2020.116657
Pan, Y., Guyon, C., Borragan, G., Hu, Y., and Peigneux, P. (2020b). Interpersonal brain synchronization with instructor compensates for learner's sleep deprivation in interactive learning. bioRxiv. doi: 10.1101/2020.04.03.022954
Pan, Y., Novembre, G., Song, B., Li, X., and Hu, Y. (2018). Interpersonal synchronization of inferior frontal cortices tracks social interactive learning of a song. Neuroimage 183, 280–290. doi: 10.1016/j.neuroimage.2018.08.005
Piazza, E. A., Hasenfratz, L., Hasson, U., and Lew-Williams, C. (2020). Infant and adult brains are coupled to the dynamics of natural communication. Psychol. Sci. 31, 6–17. doi: 10.1177/0956797619878698
Piva, M., Zhang, X., Noah, J. A., Chang, S. W., and Hirsch, J. (2017). Distributed neural activity patterns during human-to-human competition. Front. Hum. Neurosci. 11:571. doi: 10.3389/fnhum.2017.00571
Quaresima, V., and Ferrari, M. (2019). Functional near-infrared spectroscopy (fNIRS) for assessing cerebral cortex function during human behavior in natural/social situations: a concise review. Organ. Res. Methods 22, 46–68. doi: 10.1177/1094428116658959
Quiñones-Camacho, L. E., Fishburn, F. A., Camacho, M. C., Hlutkowsky, C. O., Huppert, T. J., Wakschlag, L. S., et al. (2019). Parent-child neural synchrony: a novel approach to elucidating dyadic correlates of preschool irritability. J. Child Psychol. Psychiatry. 61, 1213–1223. doi: 10.1111/jcpp.13165
Reindl, V., Gerloff, C., Scharke, W., and Konrad, K. (2018). Brain-to-brain synchrony in parent-child dyads and the relationship with emotion regulation revealed by fNIRS-based hyperscanning. Neuroimage 178, 493–502. doi: 10.1016/j.neuroimage.2018.05.060
Sarinasadat, H., Hattori, Y., Miyake, Y., and Nozawa, T. (2019a). “Music valence and genre influence group creativity,” in International Conference on Human-Computer Interaction (Orlando, FL: Springer), 410–422. doi: 10.1007/978-3-030-22507-0_32
Sarinasadat, H., Miyake, Y., and Nozawa, T. (2019b). “Combined effects of background music and nonverbal synchrony measures on group creativity—a multiple regression approach,” in 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC) (Bari: IEEE), 3752–3757.
Scholkmann, F., Holper, L., Wolf, U., and Wolf, M. (2013). A new methodical approach in neuroscience: assessing inter-personal brain coupling using functional near-infrared imaging (fNIRI) hyperscanning. Front. Hum. Neurosci. 7:813. doi: 10.3389/fnhum.2013.00813
Schwartz, A. M., Wilson, J. M., Boden, S. D., Moore, T. J. Jr, Bradbury, T. L. Jr, and Fletcher, N. D. (2020). Managing resident workforce and education during the COVID-19 pandemic: evolving strategies and lessons learned. JBJS Open Access 5:e0045. doi: 10.2106/JBJS.OA.20.00045
Sun, B., Xiao, W., Feng, X., Shao, Y., Zhang, W., and Li, W. (2020). Behavioral and brain synchronization differences between expert and novice teachers when collaborating with students. Brain Cogn. 139:105513. doi: 10.1016/j.bandc.2019.105513
Tachtsidis, I., and Scholkmann, F. (2016). False positives and false negatives in functional near-infrared spectroscopy: issues, challenges, and the way forward. Neurophotonics 3:031405. doi: 10.1117/1.NPh.3.3.031405
Takeuchi, N., Mori, T., Suzukamo, Y., and Izumi, S.-I. (2017). Integration of teaching processes and learning assessment in the prefrontal cortex during a video game teaching-learning task. Front. Psychol. 7:2052. doi: 10.3389/fpsyg.2016.02052
Tang, H., Mai, X., Wang, S., Zhu, C., Krueger, F., and Liu, C. (2016). Interpersonal brain synchronization in the right temporo-parietal junction during face-to-face economic exchange. Soc. Cogn. Affect. Neurosci. 11, 23–32. doi: 10.1093/scan/nsv092
Van Bavel, J. J., Baicker, K., Boggio, P. S., Capraro, V., Cichocka, A., Cikara, M., et al. (2020). Using social and behavioural science to support covid-19 pandemic response. Nat. Hum. Behav. 4, 460–471. doi: 10.1038/s41562-020-0884-z
Vanzella, P., Balardin, J. B., Furucho, R. A., Zimeo Morais, G. A., Braun Janzen, T., Sammler, D., et al. (2019). fNIRS responses in professional violinists while playing duets: evidence for distinct leader and follower roles at the brain level. Front. Psychol. 10:164. doi: 10.3389/fpsyg.2019.00164
Wang, M.-Y., Luan, P., Zhang, J., Xiang, Y.-T., Niu, H., and Yuan, Z. (2018). Concurrent mapping of brain activation from multiple subjects during social interaction by hyperscanning: a mini-review. Quant. Imaging Med. Surg. 8:819. doi: 10.21037/qims.2018.09.07
Yang, J., Zhang, H., Ni, J., De Dreu, C. K., and Ma, Y. (2020). Within-group synchronization in the prefrontal cortex associates with intergroup conflict. Nat. Neurosci. 23, 754–760. doi: 10.1038/s41593-020-0630-x
Zhang, M., Ding, K., Jia, H., and Yu, D. (2018). “Brain-to-brain synchronization of the expectation of cooperation behavior: a fNIRS hyperscanning study,” in 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (Honolulu, HI: IEEE), 546–549. doi: 10.1109/EMBC.2018.8512315
Zhang, M., Liu, T., Pelowski, M., Jia, H., and Yu, D. (2017a). Social risky decision-making reveals gender differences in the TPJ: a hyperscanning study using functional near-infrared spectroscopy. Brain Cogn. 119, 54–63. doi: 10.1016/j.bandc.2017.08.008
Zhang, M., Liu, T., Pelowski, M., and Yu, D. (2017b). Gender difference in spontaneous deception: a hyperscanning study using functional near-infrared spectroscopy. Sci. Rep. 7:7508. doi: 10.1038/s41598-017-06764-1
Zhang, X., Noah, J. A., Dravida, S., and Hirsch, J. (2020). Optimization of wavelet coherence analysis as a measure of neural synchrony during hyperscanning using functional near-infrared spectroscopy. Neurophotonics 7:015010. doi: 10.1117/1.NPh.7.1.015010
Zhang, Y., Meng, T., Hou, Y., Pan, Y., and Hu, Y. (2018). Interpersonal brain synchronization associated with working alliance during psychological counseling. Psychiatry Res. Neuroimaging 282, 103–109. doi: 10.1016/j.pscychresns.2018.09.007
Zhao, Y., Dai, R.-N., Xiao, X., Zhang, Z., Duan, L., Li, Z., et al. (2017). Independent component analysis-based source-level hyperlink analysis for two-person neuroscience studies. J. Biomed. Opt. 22:027004. doi: 10.1117/1.JBO.22.2.027004
Zheng, L., Chen, C., Liu, W., Long, Y., Zhao, H., Bai, X., et al. (2018). Enhancement of teaching outcome through neural prediction of the students' knowledge state. Hum. Brain Mapp. 39, 3046–3057. doi: 10.1002/hbm.24059
Zheng, L., Liu, W., Long, Y., Zhai, Y., Zhao, H., Bai, X., et al. (2020). Affiliative bonding between teachers and students through interpersonal synchronisation in brain activity. Soc. Cogn. Affect. Neurosci. 15, 97–109. doi: 10.1093/scan/nsaa016
Keywords: hyperscanning, functional near-infrared spectroscopy, fNIRS, social interaction, virtual interaction
Citation: Balters S, Baker JM, Hawthorne G and Reiss AL (2020) Capturing Human Interaction in the Virtual Age: A Perspective on the Future of fNIRS Hyperscanning. Front. Hum. Neurosci. 14:588494. doi: 10.3389/fnhum.2020.588494
Received: 29 July 2020; Accepted: 02 October 2020;
Published: 03 November 2020.
Edited by:Ilias Tachtsidis, University College London, United Kingdom
Reviewed by:Felix Scholkmann, University Hospital Zürich, Switzerland
Artur Czeszumski, University of Osnabrück, Germany
Joy Hirsch, Yale University, United States
Hasan Ayaz, Drexel University, United States
Copyright © 2020 Balters, Baker, Hawthorne and Reiss. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Stephanie Balters, email@example.com