Abstract
Hyperscanning, which enables the recording of brain activity from multiple individuals simultaneously, has been increasingly used to investigate the neuropsychological processes underpinning social interaction. Previous hyperscanning research has primarily focused on interbrain synchrony, demonstrating an enhanced alignment of brain waves across individuals during social interaction. However, using EEG hyperscanning simulations, we here show that interbrain synchrony has low sensitivity to information alignment across people. Surprisingly, interbrain synchrony remains largely unchanged despite manipulating whether two individuals are seeing same or different things at the same time. Furthermore, we show that hyperscanning recordings do contain indices of interpersonal information alignment and that they can be captured using representational analyses. These findings highlight major limitations of current hyperscanning research and offer a promising alternative for investigating interactive minds.
1 Introduction
Understanding how social interactions dynamically shape human mind and behavior and vice versa is a key question in cognitive neuroscience and psychology (Sebanz et al., 2006; Hari et al., 2015). Hyperscanning, which enables the recording of the brain activity of multiple individuals at the same time, has been increasingly used to address this question. Hyperscanning research revealed enhanced interbrain synchrony between individuals interacting socially, which has been argued to reflect resonant minds facilitating cooperation and communication (Dumas et al., 2010; Dikker et al., 2017; Pérez et al., 2017; Goldstein et al., 2018; Reindl et al., 2018). However, here we highlight an important limitation of interbrain synchrony measures by showing that these measures indexing the alignment of neural activity patterns across individuals have low sensitivity to their Information content. We show that representational analyses, the framework of Representational Similarity Analysis (RSA) in particular, is a promising alternative to address this issue and better index information alignment across individuals in hyperscanning research.
2 Interbrain synchrony does not measure information alignment
Interbrain synchrony has been investigated in hyperscanning research using a range of neuroimaging techniques including Electroencephalography (EEG), Magnetoencephalography (MEG), functional Magnetic Resonance Imaging (fMRI), and functional Near-InfraRed Spectroscopy (fNIRS). Numerous measures have been used to index interbrain synchrony in hyperscanning studies generally based on amplitude correlation and phase locking (Czeszumski et al., 2020). For EEG and MEG, enabling access to neural processing at faster time scales, these analyses have also focused on the amplitude and phase of neural oscillations, with larger emphases on activity in the alpha (8–12 Hz) and beta (13–30 Hz) bands (Dumas et al., 2010; Goldstein et al., 2018). See Czeszumski et al. (2020) for a comprehensive review of hyperscanning and interbrain synchrony measures.
While all these methods are relevant to measure synchrony in general, the processes captured by interbrain synchrony in the context of hyperscanning research remain unclear. The causal nature of this phenomenon has been the main source of concerns so far, whether interbrain synchrony directly drives synchronised mind and behavior or interbrain synchrony is simply caused by synchronised mind and behavior (Burgess, 2013; Hari et al., 2013; Hamilton, 2021; Novembre and Iannetti, 2021; Holroyd, 2022). If synchronised mind and behavior causes interbrain synchrony, then this suggests that there is no direct brain-to-brain coupling between individuals, and interbrain synchrony is the result of similar sensory and cognitive processes driven by shared environment and task. As pointed out by Hamilton (2021), a lack of direct brain-to-brain coupling may not be a critical issue because interbrain synchrony measures are still of interest for hyperscanning research if they provide insight into the alignment of sensory information and cognitive processes across individuals while socially interacting.
However, here we present results showing that interbrain synchrony measures do not effectively index information alignment between individuals because these measures are largely content agnostic. We leveraged a publicly available EEG dataset from the visual object recognition literature with 48 participants who were presented with 400 images from 40 categories (Shatek et al., 2022). All recordings were made individually with a 128-channel BrainVision actichamp EEG system. We used this dataset to run hyperscanning simulations to assess the sensitivity of interbrain synchrony to information alignment by comparing interbrain synchrony when two individuals see the same things (Same objects) vs. different things (Different objects) at the same time (Figures 1, 2).
Figure 1
Figure 2
We ran 10,000 hyperscanning simulations with 24 pseudo pairs randomly drawn from the 48 subjects available (Figure 2). Interbrain synchrony was computed for a wide range of frequencies as amplitude correlation and phase locking across subjects for all possible combinations of channels (Czeszumski et al., 2020). Results were averaged across all possible combinations of channels to capture any potential changes in interbrain synchrony. With optimal control of the timing and content of the presented stimuli, the results from the simulations revealed that interbrain synchrony has low sensitivity to information alignment across individuals. Amplitude correlation and phase locking values remained largely unchanged whether people were seeing the same object or not (Figure 2D). t-values after stimulus presentation across all frequencies obtained from the 10,000 simulations show that difference between Same objects and Different objects would in most cases be statistically undetectable despite a sample size at least similar (i.e., 24 pairs) to those generally used in the literature (Figure 2F). Figure 2F depicting t-values on averaged amplitude and phase values across frequencies within 180 and 220 ms after image presentation – time window yielding highest percentages of significant t-values and peak decoding accuracy in object literature in general (Grootswagers et al., 2019; Shatek et al., 2022) – shows that barely 50% of the simulations reached statistical significance for interbrain synchrony (IBS). See legend of Figure 2 and publicly available code at https://osf.io/etx64/ for further methodological details.
3 Interbrain representational similarity analysis to effectively measure information alignment
The Representational Similarity Analysis (RSA) framework has received significant interest in the field of cognitive neuroscience (Kriegeskorte et al., 2008; Haxby et al., 2014), including more recently in social neuroscience and the intersubject correlation community (Nastase et al., 2019; Popal et al., 2019), but has not been used yet in hyperscanning research. This analysis is based on the computation of Representational (Dis)similarity Matrices (RDMs) to abstract from the patterns of neural activity themselves and characterise their informational content, allowing testing how two different systems quantitatively relate to each other by comparing their RDMs. RSA makes it possible to compare the responses from different systems, including responses recorded with different neuroimaging systems, neuroimaging and behavioral responses, as well as responses from different individuals, as shown here.
We computed Interbrain RSA (IRSA) using the same amplitude and phase data as IBS (Figure 2E). RDMs that encode information content and abstract from activity patterns were first computed separately for each subject and channel by calculating the amplitude correlation and phase locking for all pairs of stimuli. This approach based on (dis)similar temporal patterns in time-frequency data separately for each channel differs from other approaches often used with RSA (Kriegeskorte et al., 2008; e.g., Grootswagers et al., 2017; Shatek et al., 2022), but allows here to directly compare between IRSA and IBS. The results of the simulations show that information alignment can be captured in EEG hyperscanning with better sensitivity using IRSA. IRSA values decrease dramatically when shuffling the stimuli (Figure 2E), as reflected in differences between Same object vs. Different objects being statistically detected more consistently than IBS with statistical significance being reached in most simulations (Figure 2F). See legend of Figure 2 and publicly available code at https://osf.io/etx64/ for further methodological details.
4 IBS vs. IRSA: statistical comparison
Statistical analyses on the 10,000 simulations indicated that t-values testing differences between Same vs. Different objects in the 180–220 ms selected time window, as depicted in Figure 2F, were significantly higher for IRSA than IBS for both amplitude, t(19998) = 117.22, p < 0.0001, d = 1.66, BF10 > 1,000, and phase, t(19998) = 109.31, p < 0.0001, d = 1.55, BF10 > 1,000, showing that IRSA has higher sensitivity to content shuffling than IBS. These effects held beyond the selected time window as indicated by significantly higher t-values for IRSA than IBS for both amplitude, t(19998) = 139.21, p < 0.0001, d = 1.97, BF10 > 1,000, and phase, t(19998) = 95.87, p < 0.0001, d = 1.36, BF10 > 1,000, when conducting these analyses on a larger time interval from 0 to 500 ms after stimulus presentation. Large effect sizes observed here demonstrate a critical advantage of IRSA over IBS to index information alignment from amplitude and phase data in hyperscanning research.
5 Perspectives and challenges
Using hyperscanning simulations with well-controlled visual stimuli, our work shows that content-related information in hyperscanning research is not effectively captured by interbrain synchrony measures. This contrasts with previous research that found reliable decrease in interbrain synchrony when shuffling data across trials and/or pairs (Zamm et al., 2021, 2024; Reindl et al., 2022), which our results suggest might most likely be due to breaking the alignment of the timing of sensory and cognitive processes occurring in shared tasks and environments rather than the alignment of their information content. Shuffling information content while keeping timing constant in our simulations only marginally decreased interbrain synchrony whereas changes in timing resulting in large variations in neural signals would have a strong influence (Burgess, 2013; Holroyd, 2022). Examining and controlling for timing (dis)alignment of sensory, cognitive, and motor responses when shuffling across pairs and/or trials will be essential in future research to better understand the processes reflected by interbrain synchrony.
More generally, these results support the hypothesis that interbrain synchrony is caused by synchronised mind and behavior rather than the opposite, interbrain synchrony causing synchronised mind and behavior. Showing that interbrain synchrony does not uniquely capture sensory and cognitive processes supporting social interactions, our results suggest that interbrain synchrony cannot be a causal mechanism (underpinned by direct brain-to-brain coupling), and importantly, might be at best a poor proxy of synchronised sensory and cognitive processes supporting social interactions (Hamilton, 2021; Novembre and Iannetti, 2021). To be an effective top-down mechanism enabling the myriads of social behaviors observed every day, interbrain synchrony would not only require reflecting reliable timing information but also content information, which is not supported by our hyperscanning simulations.
The lack of unique and direct relationship between interbrain synchrony and synchronised sensory and cognitive processes during social interactions might explain part of the discrepancies in hyperscanning literature, including interbrain synchrony not being observed in some studies despite participants interacting and synchronising with each other (Liu et al., 2018; Czeszumski et al., 2020; Newman et al., 2024). While further research with a wider range of sensory and cognitive processes and neuroimaging techniques will be needed to expand our work beyond EEG visual evoked responses, the limitations of interbrain synchrony revealed here add to the growing concerns having been expressed in the field (Burgess, 2013; Hari et al., 2013; Hamilton, 2021; Novembre and Iannetti, 2021; Holroyd, 2022).
Our results highlight representational analyses as a powerful alternative to synchrony measures to better index information alignment between individuals. Enabling information content to be compared across individuals, even while obtained from different (neuroimaging) systems, these analyses will have critical advantages for future hyperscanning research. While implementing representational analyses is relatively straightforward when having time-locked trials with many different stimuli, moving into the representational space in more naturistic tasks as often used in hyperscanning research will be more challenging. Advancing representational analyses methods for non-time locked data will be needed to reach full capacity of hyperscanning and enable a step change in understanding the sensory and cognitive processes supporting real-time social interactions.
Statements
Data availability statement
The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://osf.io/etx64/.
Ethics statement
The studies involving humans were approved by University of Sydney Human Research Ethics Committee. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
Author contributions
MV: Conceptualization, Methodology, Writing – original draft. TG: Conceptualization, Methodology, Writing – review & editing.
Funding
The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This research was supported by the Australian Research Council (DP220103047, DE230100380).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1
BurgessA. P. (2013). On the interpretation of synchronization in EEG hyperscanning studies: a cautionary note. Front. Hum. Neurosci.7:881. doi: 10.3389/fnhum.2013.00881
2
CzeszumskiA.EustergerlingS.LangA.MenrathD.GerstenbergerM.SchuberthS.et al. (2020). Hyperscanning: a valid method to study neural inter-brain underpinnings of social interaction. Front. Hum. Neurosci.14:39. doi: 10.3389/fnhum.2020.00039
3
DikkerS.WanL.DavidescoI.KaggenL.OostrikM.McClintockJ.et al. (2017). Brain-to-brain synchrony tracks real-world dynamic group interactions in the classroom. Curr. Biol.27, 1375–1380. doi: 10.1016/j.cub.2017.04.002
4
DumasG.NadelJ.SoussignanR.MartinerieJ.GarneroL. (2010). Inter-brain synchronization during social interaction. PLoS One5:e12166. doi: 10.1371/journal.pone.0012166
5
GoldsteinP.Weissman-FogelI.DumasG.Shamay-TsooryS. G. (2018). Brain-to-brain coupling during handholding is associated with pain reduction. Proc. Natl. Acad. Sci.115, E2528–E2537. doi: 10.1073/pnas.1703643115
6
GrootswagersT.RobinsonA. K.CarlsonT. A. (2019). The representational dynamics of visual objects in rapid serial visual processing streams. NeuroImage188, 668–679. doi: 10.1016/j.neuroimage.2018.12.046
7
GrootswagersT.WardleS. G.CarlsonT. A. (2017). Decoding dynamic brain patterns from evoked responses: a tutorial on multivariate pattern analysis applied to time series neuroimaging data. J. Cogn. Neurosci.29, 677–697. doi: 10.1162/jocn_a_01068
8
HamiltonA. F. C. (2021). Hyperscanning: beyond the hype. Neuron109, 404–407. doi: 10.1016/j.neuron.2020.11.008
9
HariR.HenrikssonL.MalinenS.ParkkonenL. (2015). Centrality of social interaction in human brain function. Neuron88, 181–193. doi: 10.1016/j.neuron.2015.09.022
10
HariR.HimbergT.NummenmaaL.HämäläinenM.ParkkonenL. (2013). Synchrony of brains and bodies during implicit interpersonal interaction. Trends Cogn. Sci.17, 105–106. doi: 10.1016/j.tics.2013.01.003
11
HaxbyJ. V.ConnollyA. C.GuntupalliJ. S. (2014). Decoding neural representational spaces using multivariate pattern analysis. Annu. Rev. Neurosci.37, 435–456. doi: 10.1146/annurev-neuro-062012-170325
12
HolroydC. B. (2022). Interbrain synchrony: on wavy ground. Trends Neurosci.45, 346–357. doi: 10.1016/j.tins.2022.02.002
13
KriegeskorteN.MurM.BandettiniP. A. (2008). Representational similarity analysis-connecting the branches of systems neuroscience. Front. Syst. Neurosci.2:4. doi: 10.3389/neuro.06.004.2008
14
LiuD.LiuS.LiuX.ZhangC.LiA.JinC.et al. (2018). Interactive brain activity: review and progress on EEG-based hyperscanning in social interactions. Front. Psychol.9:1862. doi: 10.3389/fpsyg.2018.01862
15
NastaseS. A.GazzolaV.HassonU.KeysersC. (2019). Measuring shared responses across subjects using intersubject correlation. Soc. Cogn. Affect. Neurosci.14, 667–685. doi: 10.1093/scan/nsz037
16
NewmanL. A.CaoM.TäuberS.van VugtM. (2024). Working memory load impairs tacit coordination but not inter-brain EEG synchronization. Soc. Cogn. Affect. Neurosci.19:nsae017. doi: 10.1093/scan/nsae017
17
NovembreG.IannettiG. D. (2021). Hyperscanning alone cannot prove causality. Multibrain stimulation can. Trends Cogn. Sci.25, 96–99. doi: 10.1016/j.tics.2020.11.003
18
OostenveldR.FriesP.MarisE.SchoffelenJ.-M. (2011). FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput. Intell. Neurosci.2011, 1–9. doi: 10.1155/2011/156869
19
PérezA.CarreirasM.DuñabeitiaJ. A. (2017). Brain-to-brain entrainment: EEG interbrain synchronization while speaking and listening. Sci. Rep.7, 1–12. doi: 10.1038/s41598-017-04464-4
20
PopalH.WangY.OlsonI. R. (2019). A guide to representational similarity analysis for social neuroscience. Soc. Cogn. Affect. Neurosci.14, 1243–1253. doi: 10.1093/scan/nsz099
21
ReindlV.GerloffC.ScharkeW.KonradK. (2018). Brain-to-brain synchrony in parent-child dyads and the relationship with emotion regulation revealed by fNIRS-based hyperscanning. NeuroImage178, 493–502. doi: 10.1016/j.neuroimage.2018.05.060
22
ReindlV.WassS.LeongV.ScharkeW.WistubaS.WirthC. L.et al. (2022). Multimodal hyperscanning reveals that synchrony of body and mind are distinct in mother-child dyads. NeuroImage251:118982. doi: 10.1016/j.neuroimage.2022.118982
23
SebanzN.BekkeringH.KnoblichG. (2006). Joint action: bodies and minds moving together. Trends Cogn. Sci.10, 70–76. doi: 10.1016/j.tics.2005.12.009
24
ShatekS. M.RobinsonA. K.GrootswagersT.CarlsonT. A. (2022). Capacity for movement is an organisational principle in object representations. NeuroImage261:119517. doi: 10.1016/j.neuroimage.2022.119517
25
ZammA.LoehrJ. D.VesperC.KonvalinkaI.KappelS. L.HeggliO. A.et al. (2024). A practical guide to EEG hyperscanning in joint action research: from motivation to implementation. Soc. Cogn. Affect. Neurosci.19:nsae026. doi: 10.1093/scan/nsae026
26
ZammA.PalmerC.BauerA.-K. R.BleichnerM. G.DemosA. P.DebenerS. (2021). Behavioral and neural dynamics of interpersonal synchrony between performing musicians: a wireless EEG hyperscanning study. Front. Hum. Neurosci.15:717810. doi: 10.3389/fnhum.2021.717810
Summary
Keywords
hyperscanning, EEG, interbrain synchrony, social interaction, representational alignment, information content
Citation
Varlet M and Grootswagers T (2024) Measuring information alignment in hyperscanning research with representational analyses: moving beyond interbrain synchrony. Front. Hum. Neurosci. 18:1385624. doi: 10.3389/fnhum.2024.1385624
Received
13 February 2024
Accepted
16 July 2024
Published
25 July 2024
Volume
18 - 2024
Edited by
Markus Franziskus Mueller, Autonomous University of the State of Morelos, Mexico
Reviewed by
Jorge Hermosillo Valadez, Autonomous University of the State of Morelos, Mexico
Updates
Copyright
© 2024 Varlet and Grootswagers.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Manuel Varlet, m.varlet@westernsydney.edu.au
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.