Abstract
Recent years have seen a dramatic increase in studies measuring brain activity, physiological responses, and/or movement data from multiple individuals during social interaction. For example, so-called “hyperscanning” research has demonstrated that brain activity may become synchronized across people as a function of a range of factors. Such findings not only underscore the potential of hyperscanning techniques to capture meaningful aspects of naturalistic interactions, but also raise the possibility that hyperscanning can be leveraged as a tool to help improve such naturalistic interactions. Building on our previous work showing that exposing dyads to real-time inter-brain synchrony neurofeedback may help boost their interpersonal connectedness, we describe the biofeedback application Hybrid Harmony, a Brain-Computer Interface (BCI) that supports the simultaneous recording of multiple neurophysiological datastreams and the real-time visualization and sonification of inter-subject synchrony. We report results from 236 dyads experiencing synchrony neurofeedback during naturalistic face-to-face interactions, and show that pairs' social closeness and affective personality traits can be reliably captured with the inter-brain synchrony neurofeedback protocol, which incorporates several different online inter-subject connectivity analyses that can be applied interchangeably. Hybrid Harmony can be used by researchers who wish to study the effects of synchrony biofeedback, and by biofeedback artists and serious game developers who wish to incorporate multiplayer situations into their practice.
Introduction
What does it mean to lose yourself in someone else? How is it possible that the mere physical presence of another human can make us believe we can conquer the world, or conversely, make us feel lonely and incapable? We know, both scientifically and intuitively, that relationships are crucial for our physical and mental well-being (Pietromonaco and Collins, 2017). But they are also sources of frustration in their fluid, messy mix of internal inconsistencies: love and hate, inclusion and exclusion, fascination and comfort, challenge and familiarity. Can we capture this seemingly subjective, fleeting, and elusive notion of “being on the same wavelength” with another person, with objective measurement tools? And if so, can we leverage this information to guide people in their interaction with others?
Successful social interactions require tight spatiotemporal coordination between participants at motor, perceptual, and cognitive levels. Around a decade ago, several labs began to use a variety of methods to record (neuro)physiological data from multiple people simultaneously, a technique known as “hyperscanning.” This has allowed researchers to study dynamic coordination in a range of social situations such as ensembles performing music, multiple people performing actions together, or carrying on a conversation (Babiloni et al., 2006; Tognoli et al., 2007; Dumas et al., 2010; Yun, 2013; Zamm et al., 2018b; see e.g., Hari et al., 2013; Babiloni and Astolfi, 2014; Czeszumski et al., 2020 for reviews). There now exists a growing body of work where pairs or groups of participants engage in social interactions while their brain activity, physiological responses, and (eye) movements are monitored.
While hyperscanning research is also conducted using hemodynamic neuroimaging tools, including functional Magnetic Resonance Imagining (fMRI; Koike et al., 2016, 2019; Abe et al., 2019) but especially functional near-infrared spectroscopy (fNIRS; Scholkmann et al., 2013; Nozawa et al., 2016; Reindl et al., 2018), we here focus on electroencephalography (EEG) hyperscanning. The extent to which EEG activity becomes synchronized between people is correlated with a range of factors. For example, it has been widely demonstrated, in both single-brain laboratory research and hyperscanning studies, that shared attention to the same stimuli leads to similar brain responses across individuals, and consequently, higher inter-brain synchrony (Figure 1A; Hasson, 2004; Dikker et al., 2017; Czeszumski et al., 2020). Importantly, social behavior has also been shown to serve as an (exogenous) source of interpersonal synchrony (Figures 1B,E): Behaviors such as joint action, language, eye contact, touch, and cooperation drive synchrony in various social contexts (Dumas et al., 2010; Dikker et al., 2014, 2021; Kinreich et al., 2017; Goldstein et al., 2018; Pérez et al., 2019; Reinero et al., 2021). Furthermore, both individuals' social closeness and personality traits (e.g., empathy) affect people's social engagement during an interaction, and thus the extent to which their brain responses become synchronized (Dikker et al., 2017, 2021; Kinreich et al., 2017; Goldstein et al., 2018; Bevilacqua et al., 2019). Participants' mental states (e.g., focus) similarly influence participants' engagement with each other, endogenously motivating them to make an effort to connect to each other (Figure 1C; Dikker et al., 2017, 2021).
Figure 1
Importantly, people can also be extrinsically motivated to socially engage with each other (Figure 1D). Specifically, our group recently reported that exposing people to a hyperscanning neurofeedback environment can motivate social engagement (Dikker et al., 2021). Using data from the interactive social neurofeedback installation The Mutual Wave Machine (wp.nyu.edu/mutualwavemachine), we show that dyads who were explicitly made aware of the social relevance of the neurofeedback environment, exhibited an increase in inter-brain coupling over time. This suggests that external factors may help boost interpersonal engagement, which raises the possibility that interpersonal synchrony biofeedback may be one fruitful avenue to pursue in such efforts.
However, while neurofeedback applications using data from individual brains are fairly widely used across scientific, clinical, educational, and artistic contexts (see e.g., van Hoogdalem et al., 2020 for a review), to our knowledge multi-person neurofeedback using hyperscanning EEG has been implemented primarily in game and art environments (see contributions in Kovacevic et al., 2015; Dikker et al., 2019, 2021; Nijholt, 2019; see Duan et al., 2013; Salminen et al., 2019 for examples of scientifically oriented dual-brain neurofeedback experiments). As a result, little is known about the possible effectiveness of hyperscanning neurofeedback in improving social communication.
This is further complicated by the fact that consensus is lacking with regard to how synchrony should be computed (Ayrolles et al., 2021). While some metrics have been shown to be “better” than others from a purely statistical perspective (Burgess, 2013), only very few scholars have attempted to map computational choices with regard to interpersonal neural connectivity to psychological processes or constructs (Dumas and Fairhurst, 2021; Hoehl et al., 2021). This distinguishes synchrony neurofeedback from other BCI applications, such as so-called “P3 spellers” (Fazel-Rezai et al., 2012), which are based on well-established neural signatures.
Because of the lack of consensus with regard to optimal synchrony metrics, we argue that it is desirable that multi-brain neurofeedback applications allow users to select from various synchrony metrics that can be used independently, and thus explore the utility of different metrics in different contexts.
To this end, we have developed Hybrid Harmony, a Brain-Computer Interface (BCI) that uses a hyperscanning approach to allow the collection of EEG data from two or more people simultaneously and enables users to visualize/sonify the extent to which participants' biometrics are coupled, choosing between different synchrony metrics. These metrics, described in section Connectivity Analysis, are developed in parallel with HyPyP, an open-source Python-based pipeline that allows researchers to compute and compare different inter-brain connectivity metrics on the same dataset (Ayrolles et al., 2021).
Software Description
Overview
Hybrid Harmony is an open-source software package written in Python (https://github.com/RhythmsOfRelating/HybridHarmony), accompanied by a visualization module and a sonification module (Figure 2). The software consists of a backend that handles data acquisition and performs analyses, and a Graphical User Interface (GUI) made with PyQt5 (https://www.riverbankcomputing.com/software/pyqt/), where users can control parameters for the analyses (Figure 2). We introduce the software and discuss compatible hardware (EEG systems) in section Hardware, the processing pipeline including preprocessing and connectivity analysis in section Data Preprocessing, Connectivity Analysis, Normalization, and the visualization and sonification modules in section Visualization and Sonification. Data transfer protocols, i.e., LabStreamingLayer (LSL, https://github.com/sccn/labstreaminglayer) and Open Sound Control (OSC, Wright and Momeni, 2011), are described in Supplementary Material, Data Transfer Protocol. Detailed instructions can be found on the GitHub page.
Figure 2
Hardware
The tool is compatible with any EEG device that interfaces with LabStreamingLayer, and has been tested with MUSE (https://choosemuse.com/; Bhayee et al., 2016), emotiv (EPOC and EPOC+; https://www.emotiv.com/; Williams et al., 2020), the SMARTING system from mBrainTrain (https://mbraintrain.com/; Grennan et al., 2021), and Brain Vision LiveAmp systems (https://www.brainproducts.com/; Fang et al., 2019), and can be expanded to accommodate other systems that export data to LabStreamingLayer.
Data Preprocessing
The first stage of data preprocessing is a buffering procedure that holds and segments incoming data time-series. Incoming streams from LSL are stored in a 30 s buffer updating at the EEG data's sampling rate (e.g., 250 Hz for Brain Vision LiveAmp system). Hybrid Harmony then selects the most recent time window to perform the signal processing procedure. The time window is determined by “window size” and is 3 s by default. The rate of the analysis depends on the computation bandwidth of the system Hybrid Harmony is running on. For example, on many systems we tested data are analyzed roughly 3.5 times per second.
The time window is filtered with the infinite impulse response (IIR) filter (Oppenheim, 1999) into frequency bands of interest (e.g., 8–12 Hz for the alpha frequency band), and then Hilbert transformed to generate the instantaneous analytic signal (Oppenheim, 1999). Users can choose to output spectral power concurrently by selecting the “sending power values” checkbox: power spectral density will be computed and sent to LSL along with connectivity values.
The GUI allows users to change processing parameters via “Frequency bands for analysis,” “Input data streams,” and “window size” (Figure 3). “Frequency bands for analysis” is an editable table specifying the frequency bands of interest, where “Freq. Band” denotes the frequency band name, “Min. Freq” and “Max. Freq” denote the lower and upper bounds of the frequency band, and “weight” determines how the different bands are weighted relative to one another. Figures 3, 4 show the default setup for this table. “Input data streams” displays the name, channel count, and sampling rate of the incoming streams, and its editable cells (e.g., “theta channels,” “alpha channels,” etc.) determine the specific channels to use for each frequency band. Lastly, the “window size” text field determines the length of the data segment for the analysis.
Figure 3
Figure 4
Connectivity Analysis
Connectivity Analysis takes the analytic signal from Data Preprocessing as input, and computes one connectivity value for each electrode pair per frequency band in every participant pair. Then, connectivity values are averaged across electrode pairs, so the output is one connectivity value per frequency band for every participant pair. The exact computations are adapted from the python-based inter-brain analysis pipeline HyPyP (Ayrolles et al., 2021), and listed in Table 1. The user may choose from a list of connectivity measures by changing the “Connectivity metric” parameter (Figure 5). Currently implemented metrics include coherence, imaginary coherence, envelope correlation, power correlation, phase-locking value (PLV), and Circular Correlation Coefficient (CCorr). The mathematical equations and references of these metrics are provided in Table 1. In the equations, X(i, t) denotes the analytic signal for subject x at channel i for time point t, and Y(j, t) is that for subject y at channel j for time point t, and the star sign denotes the complex conjugate. The result of each equation is one synchrony value per electrode pair, written as ri, j, and the computation is carried out for all electrode pairs (i, j) where i belongs to subject x 's channels and j belongs to subject y 's channels. Note that we are computing all metrics using the analytic signal from the previous step to streamline the computation. We compute synchrony values from the analytic signals using Hilbert Transform (HT), an alternative analysis to the windowed Fast Fourier Transform (FFT). The analytic signal from HT and the spectra from the windowed FFT both represent the amplitude and phase of the signal in their real and the imaginary parts, respectively, except that the analytic signal is “instantaneous,” while the windowed FFT is an average value over a period (Kovach, 2017). Therefore, while the cross-spectra in coherence is usually the expected value of the product between the two signals' spectra, we used a computation (Table 1) adapted from Equation (1) (Kovach, 2017), i.e., the cross-spectra are expressed as the expectation of the dot product between X(i, t) and the complex conjugate of Y(i, t). Using this formulation is appropriate when investigating the synchronization of signals as it allows us to measure their similarity on a sample-by-sample basis, not just as an average over a relatively long time-window.
Currently, only one metric at a time can be employed, but a user can run multiple instances of the software and thus output multiple metrics simultaneously. The user can then record these streams using LabRecorder (Supplementary Material, Saving Data Through LSL). For visualization, it is possible to differentiate the streams based on the unique source_id in the metadata of the LSL stream, and choose only one to display. However, this feature is not yet developed in our visualization module.
Table 1
| Connectivity metrics | Equation | References |
|---|---|---|
| Coherence | Guevara and Corsi-Cabrera, 1996; Dikker et al., 2017 | |
| Imaginary coherence | Nolte et al., 2004; Dikker et al., 2021 | |
| Envelope correlation | Mehrkanoon et al., 2014; Clerico et al., 2015; Zamm et al., 2018a | |
| where envx = |X(i, t)|, envy = |Y(j, t)| | ||
| Power correlation | Shaw, 1984; Guevara and Corsi-Cabrera, 1996 | |
| PLV | Lachaux et al., 1999; Dumas et al., 2010 | |
| CCorr | Burgess, 2013; Goldstein et al., 2018 | |
| where θx = angle(X(i, t)), θy = angle(Y(j, t)) |
Connectivity metrics adapted from Ayrolles et al. (2021).
The table shows a list of connectivity measures/synchrony metrics implemented in the software. The references column lists relevant papers, with hyperscanning studies adopting the metric marked in bold. The equations are adapted from the first paper in the reference list. The result of the equation is one connectivity value r(i, j) for electrode pair (i, j) in the N time-point window, computed from analytic signals X(i, t) and Y(j, t), as explained in section Connectivity Analysis.
Figure 5
The metrics, which, as mentioned above, are a subset of those implemented in the hyperscanning analysis pipeline HyPyP (Ayrolles et al., 2021), include two variations of correlation (envelope and power correlation) and two of coherence (coherence and imaginary coherence) measures, which are traditional linear methods to estimate brain connectivity, as well as two measures of phase synchrony (PLV and CCorr). While correlation methods are predominantly employed in hyperscanning fMRI studies to characterize joint action and shared attention (Koike et al., 2016), as shown in the case study below, we have validated power correlation as a neurofeedback synchrony signal. Coherence is more commonly used in fNIRS and EEG hyperscanning studies (Liu et al., 2016; Dikker et al., 2017; Miller et al., 2019). Imaginary coherence, i.e., the imaginary part of the coherence, was developed in response to coherence's susceptibility to zero-lagged spurious synchrony, and has been found to reflect personality traits and social closeness in hyperscanning studies (Nolte et al., 2004; Dikker et al., 2021). For phase synchrony, we included PLV, which has been widely used in hyperscanning studies to capture joint action (Dumas et al., 2010), verbal interaction (Perez Repetto et al., 2017), decision-making (Tang et al., 2016) and other tasks. CCorr measures the covariance of phase variance between two data streams and is more robust to coincidental synchrony (Burgess, 2013) compared to PLV, and has been used to investigate touch (Goldstein et al., 2018), learning (Bevilacqua et al., 2019) and language (Perez Repetto et al., 2017) in hyperscanning studies.
In addition to the connectivity metric, users are also able to choose “connectivity type” (Figure 3), which determines the electrode pair combination for connectivity. For each frequency band, the computation is carried out for every possible electrode pair between the participants, and then averaged based on the “connectivity type” parameter. If “connectivity type” is “one-to-one,” only electrode pairs in the matching position are considered (e.g., Fp1 channel of participant A is only paired with Fp1 of participant B and C, etc.); alternatively, if it is set to “all-to-all,” all electrode pairs are considered in v the averaging.
Connectivity Analysis outputs data chunks to LSL as a “Marker” stream under the name “Rvalues.” The size of this data chunk depends on the number of subjects and the number of frequency bands chosen for analysis. For example, if there are 4 subjects and 4 frequency bands, the data chunk will be a vector of length 24 (6 combinations of pairs times 4 frequency bands). Additionally, if the checkbox “sending through OSC” is selected, the same data chunks will simultaneously be transmitted through the OSC protocol with parameters in “OSC IP address” and “OSC port.”
Normalization
As part of Connectivity Analysis, normalization of connectivity values is implemented with two options: manual normalization (labeled as “Manual” in Figure 3) and baselining with a pre-recorded file (labeled as “from file” in Figure 3). With a Min-Max normalization method, the user can use either of the options, or a mixture of both with a weighting factor adjusted by the slider “Weight.” The minimum and maximum limits are then weighted between the “Manual” and “from file” options.
Visualization and Sonification
Visualization
The example visualization protocol provided with the software is based on Mutual Brainwaves Lab (Figure 6), described in Supplementary Material, Mutual Brainwaves Lab. The visualization app was originally built in C++ using the OpenFramework toolkit (https://openframeworks.cc/), a general-purpose framework that wraps together several libraries to assist the creative process. An updated Python3 version makes it easier to deploy the application on different OS (Windows, MacOS, Linux) and more maintainable and extendable for developers, thanks to Python's more accessible syntax and a wide choice of libraries. The application relies on an OSC plugin that listens in real-time for an OSC sender over the network. The GUI of the application is built using OpenFramework for the C++ version and PyQt5 (https://pypi.org/project/PyQt5/) for the Python version. When launched, it presents two avatars representing human heads with a brain icon, and a menu with various options to parameterize the interface. As soon as the OSC receiver starts receiving a stream of data from the Hybrid Harmony running on the same network, the application translates the inter-brain synchrony as the distance between the avatar heads. The user can also set up sessions where participants are encouraged to “score” higher synchrony in a limited timeframe.
Figure 6
Sonification
Sonification of EEG data has been explored in projects such as EEGsynth (https://github.com/eegsynth/eegsynth), which interfaces electrophysiological recordings (e.g., EEG, EMG and ECG) with analog synthesizers and digital devices. Here, we demonstrate that Hybrid Harmony can be easily interfaced with a digital audio workstation (DAW) in real-time through an OSC (Supplementary Material, The Data Transfer Protocol: Open Sound Control) plugin, allowing for the control of audio parameters based on the connectivity values sent through OSC. We describe the protocol for the control of Ableton Live via LiveGrabber (https://www.showsync.com/tools#livegrabber), a set of free Ableton plugins. LiveGrabber receives messages from any OSC sender on the network and uses OSC messages to control track parameters in Ableton.
After specifying the OSC IP address and port in the Hybrid Harmony GUI, the output can be received by the GrabberReceiver plugin in Ableton (part of the LiveGrabber package). The TrackGrabber plugin allows for the control of track parameters (such as volume, reverb, panning, etc.) using the output from Hybrid Harmony in real time. To illustrate a simple sonification example, we have created a soundscape in which the volume of certain musical pitches can be modulated to create alternating moments of dissonance (harmonic tension or unpleasant sounding chords) and consonance (harmonic resolution or pleasant, stable chords). The volume of each pitch is directly controlled by connectivity parameters output through OSC, such that greater connectivity values (moments of increased interpersonal synchrony) correspond to more pleasant, stable sounding chords.
Validation
To validate that Hybrid Harmony can capture socially relevant self-report measures, we used a dataset of 243 dyads participating in the Mutual Wave Machine (Figure 7; see Supplementary Material, Case study: The Mutual Wave Machine for details), during which real-time envelope correlations were recorded and translated into light patterns projected onto the surface of two spheres (Hybrid Harmony parameters: “Frequency bands for analysis”: delta 1–4 Hz, theta 4–8 Hz, alpha 8–12 Hz, beta 12–20 Hz; “Connectivity type”: one-to-one; “Connectivity metric”: envelope correlation; “Window size”: 3 s).
Figure 7
We first asked whether the average envelope correlation reflected dyads' personal distress (Davis, 1980), building on past research where we consistently find a negative relationship between personal distress and inter-brain synchrony (Dikker et al., 2017, 2021; Chen et al., 2021; Reinero et al., 2021). Indeed, we find that pairs' average personal distress was negatively correlated with their neurofeedback synchrony in theta [r(236) = −0.182, pFDR = 0.010], alpha [r(236) = −0.204, pFDR = 0.004] and beta [r(236) = −0.178, pFDR = 0.010].
We then asked whether social closeness (Aron et al., 1992) was positively related to pairs' envelope correlations. We reasoned that this is an interpersonal state measure that might be the target for a social neurofeedback intervention. Indeed, we find significant positive correlations between social closeness and pairs' neurofeedback synchrony in the alpha [r(236) = 0.264, pFDR <0.001] and beta bands [r(236) = 0.210, pFDR = 0.004].
While these results do not speak to the efficacy of Hybrid Harmony as a social neurofeedback tool, they confirm a very important first step, namely that the neurofeedback output is correlated with socially relevant features that might be the target for neurofeedback interventions.
Discussion
We describe Hybrid Harmony, an open-source software that allows researchers to explore interpersonal synchrony in a plug-and-play setup. The project builds on previous work from our group suggesting that incorporating synchrony neurofeedback in naturalistic social interactions may help increase synchrony and interpersonal connectedness, as such raising the possibility that biofeedback may constitute a useful tool to explore meaningful features of social interaction (Dikker et al., 2019, 2021).
As discussed in the Introduction, inter-brain synchrony has been shown to correlate with a range of personal and social characteristics and behaviors, underscoring its relevance in understanding naturalistic social interactions. Interpersonal biofeedback approaches may make it possible to more precisely map such social and psychological factors onto specific neurophysiological processes. For example, testing different types of synchrony in a neurofeedback environment might help inform the field about which metric is the most indicative of social behavior and outcomes in which social contexts. Dyads may be more responsive to multi-brain neurofeedback based on, say, coherence during collaborative tasks, but more responsive to, say, envelope correlations during social sharing. Future findings of such a nature will enrich our knowledge about the social relevance of these metrics and constitute a non-invasive way to probe possible causal links between inter-brain synchrony and social behavior (Moreau and Dumas, 2021; Novembre and Iannetti, 2021). Here we show in a dataset of 243 dyads that social closeness and affective traits can, in fact, be reflected in online synchrony neurofeedback measures, which is an important firspage t step in this direction.
Future Directions and Challenges
Beyond inter-brain coupling, interpersonal synchrony has been examined in more depth in other aspects of behavior, including movement (Oullier et al., 2008; van Ulzen et al., 2008; Varlet et al., 2011), language (Pickering and Garrod, 2004) and physiological rhythms such as heart rate and respiration (Müller and Lindenberger, 2011; Noy et al., 2015). As described above, movement and physiological synchrony may be both cause and effect of inter-brain coupling. In future iterations of Hybrid Harmony, we hope to extend the software to incorporate multiple data streams including physiological and movement data. This would allow users to compare the social relevance of various forms of synchrony, and possibly to tease apart interrelationships between (neuro)physiological and behavioral coupling (e.g., Dumas et al., 2010; Mayo and Gordon, 2020; Pan et al., 2020). Similarly, while we prioritized EEG research in Hybrid Harmony, given the increasingly rich fNIRS hyperscanning literature (Liu et al., 2016; Nozawa et al., 2016; Miller et al., 2019) and recent successful work in fNIRS neurofeedback (Gvirts and Perlmutter, 2020; Kohl et al., 2020) we believe an extension of Hybrid Harmony to include metrics suitable for fNIRS data would be a very welcome and fruitful future direction for the software.
EEG systems' susceptibility to movements and environmental noises can greatly compromise data quality and introduce spurious synchrony in our measure. In controlled lab studies, motion artifacts are often carefully removed manually and through data decomposition (e.g., principal component analysis). In the neurofeedback setting, however, such procedures haven't been widely implemented. In our practice, we tried to address the issue empirically with several solutions. For example, we piloted a version of the Mutual Wave Machine where sudden motion-related fluctuations in the data were removed, but this dramatically influenced the experience: participants often react enthusiastically to a sudden increase in light, only to get “punished” for facial expression, which would discourage them from naturally engaging with each other. We considered patching the data with correlations from non-contaminated stretches of data, but this would lead to arbitrary choices. We therefore instead opted for an alternative solution where participants were told explicitly that because extensive head and facial movements can dramatically affect the EEG signal, what they were seeing could also be caused by synchronous noise or synchronous movement. While this option sufficed for the experiential side of the neurofeedback, it is suboptimal with regard to data fidelity. In future releases of the software, we will incorporate support for online data cleaning procedures such as toolboxes (Mullen et al., 2015) and EEG systems that provide built-in data cleaning options in their software, e.g., the SMARTING system by mBrainTrain (Lee et al., 2020).
In addition to challenges related to data cleaning, the real-time nature of the analysis procedure poses challenges in terms of its interpretation. For instance, data fidelity is much higher when applying filtering and correlation analysis on larger stretches of data, but in the type of analysis employed here, this would compromise the immediacy of the neurofeedback. Therefore, it is important to note that our real-time approach might not be able to characterize those types of synchrony that may not be temporally aligned over short intervals. For example, fMRI studies have suggested there may be delays of up to 8 seconds in inter-brain synchrony between speakers and listeners (Stephens et al., 2010a; Dikker et al., 2014; Misaki et al., 2021). Indeed, while successful joint action is typically associated with the coupling of motor movements (Dumas et al., 2010), being “in sync” or “on the same wavelength” is often taken to imply interactive alignment at the level of mental representations (Garrod and Pickering, 2009; Pickering and Garrod, 2013), usually involving more “abstract” constructs such as sharing viewpoints (Van Berkum et al., 2009). These mental representations may or may not be linked to convergence at the temporal level. In line with this dissociation, in two instances of the Mutual Wave Machine we asked participants to reflect on their “connection strategies.” Pairs who used either eye contact or joint action as a connection strategy (mimicry, laughter, motion coordination) exhibited an increase in inter-brain synchrony over time as measured by Imaginary Coherence and Projected Power Correlation (Dikker et al., 2021). Such an increase in synchrony was not observed for pairs who tried “thinking about the same thing.” While these results validated our approach in capturing synchrony in joint action, they do not exclude the possibility of synchrony on the abstract mental representations, given that such synchrony may entail more complex temporal dynamics.
In future iterations, we hope to incorporate alternatives to the time-aligned approach, such as introducing a temporal delay between the data streams and real-time adaptive normalization. These additions will hopefully increase sensitivity to endogenous synchronizers and facilitate more controlled experimental designs (Stephens et al., 2010b).
Finally, it is worth reiterating that very little is known about the correspondence between different synchrony analysis metrics and socio-psychologically relevant factors. Although some metrics are better than others in theory (e.g., CCorr is more robust to spurious synchrony than PLV), and some are more common in hyperscanning studies than single brain studies (e.g., phase synchrony is more common than coherence/correlation), the exact pros and cons of each metric require further investigation. As more becomes known about the mapping between psychological processes as inter-brain synchrony metrics, we will add guidance for users with respect to the choice of metrics in different situations.
Conclusion
In this study, we describe the background, functionality, and validation of Hybrid Harmony, a multi-person neurofeedback application for interpersonal synchrony. With its user-friendly interface and flexible design, Hybrid Harmony enables researchers to explore the interplay between synchrony as a computational method and the various psychological, cognitive, and social functions potentially associated with it.
Publisher's Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Statements
Data availability statement
The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://github.com/RhythmsOfRelating.
Ethics statement
Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin. Written informed consent was obtained from the individual(s), and minor(s)' legal guardian/next of kin, for the publication of any potentially identifiable images or data included in this article.
Author contributions
SD, MO, and PC contributed to conception and design of the study. MO, DM, SH, MR, GD, and PC developed the software. PC performed the statistical analysis. PC, SD, SH, MR, KS DM, and GD wrote sections of the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.
Funding
This work was supported by Stichting Niemeijer Fonds, The Netherlands Organization for Scientific Research grant #275-89-018 and #406.18.GO.024, Creative Industries Fund NL, TodaysArt, Marina Abramovic Institute, and Fundación Telefónica.
Acknowledgments
We thank Brett Bayes, Laura Gwilliams, and Jean Jacques Warmerdham for their input in the project; and Brain Products, EMOTIV, MUSE, and mBrain Train for their continued support of our work.
Conflict of interest
DM is the co-director of Diademics Pty Ltd company. The handling Editor declared a shared affiliation, though no other collaboration, with one of the authors MR. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnrgo.2021.687108/full#supplementary-material
References
1
AbeM. O.KoikeT.OkazakiS.SugawaraS. K.TakahashiK.WatanabeK.et al. (2019). Neural correlates of online cooperation during joint force production. Neuroimage191, 150–161. 10.1016/j.neuroimage.2019.02.003
2
AronA.AronE. N.SmollanD. (1992). Inclusion of other in the self scale and the structure of interpersonal closeness. J. Pers. Soc. Psychol. 63, 596–612. 10.1037/0022-3514.63.4.596
3
AyrollesA.BrunF.ChenP.DjalovskiA.BeauxisY.DelormeR.et al. (2021). HyPyP: a hyperscanning python pipeline for inter-brain connectivity analysis. Soc. Cogn. Affect. Neurosci. 16, 72–83. 10.1093/scan/nsaa141
4
BabiloniF.AstolfiL. (2014). Social neuroscience and hyperscanning techniques: past, present and future. Neurosci. Biobehav. Rev. 44, 76–93. 10.1016/j.neubiorev.2012.07.006
5
BabiloniF.CincottiF.MattiaD.MattioccoM.BufalariS.De Vico FallaniF.et al. (2006). Neural basis for the brain responses to the marketing messages: an high resolution EEG study, in Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Conference, New York, NY, 3676–3679.
6
BevilacquaD.DavidescoI.WanL.ChalonerK.RowlandJ.DingM.et al. (2019). Brain-to-brain synchrony and learning outcomes vary by student–teacher dynamics: evidence from a real-world classroom electroencephalography study. J. Cogn. Neurosci. 31, 401–411. 10.1162/jocn_a_01274
7
BhayeeS.TomaszewskiP.LeeD. H.MoffatG.PinoL.MorenoS.et al. (2016). Attentional and affective consequences of technology supported mindfulness training: a randomised, active control, efficacy trial. BMC Psychol. 4:60. 10.1186/s40359-016-0168-6
8
BurgessA. P. (2013). On the interpretation of synchronization in EEG hyperscanning studies: a cautionary note. Front. Hum. Neurosci. 7:881. 10.3389/fnhum.2013.00881
9
ChenP.KirkU.DikkerS. (2021). Trait mindfulness predicts inter-brain coupling during naturalistic face-to-face interactions. bioRxiv [Preprint]. 10.1101/2021.06.28.448432
10
ClericoA.GuptaR.FalkT. H. (2015). Mutual information between inter-hemispheric EEG spectro-temporal patterns: a new feature for automated affect recognition, in 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER), Montpellier, 914–917.
11
CzeszumskiA.EustergerlingS.LangA.MenrathD.GerstenbergerM.SchuberthS.et al. (2020). Hyperscanning: a valid method to study neural inter-brain underpinnings of social interaction. Front. Hum. Neurosci. 14:39. 10.3389/fnhum.2020.00039
12
DavisM. H. (1980). A Multidimensional Approach to Individual Differences in Empathy. Available online at: https://www.uv.es/friasnav/Davis_1980.pdf (accessed March 29, 2021).
13
DikkerS.MichalareasG.OostrikM.SerafimakiA.KahramanH. M.StruiksmaM. E.et al. (2021). Crowdsourcing neuroscience: inter-brain coupling during face-to-face interactions outside the laboratory. Neuroimage227:117436. 10.1016/j.neuroimage.2020.117436
14
DikkerS.MontgomeryS.TuncaS. (2019). Using synchrony-based neurofeedback in search of human connectedness, in Brain Art: Brain-Computer Interfaces for Artistic Expression, ed NijholtA. (Springer International Publishing), 161–206.
15
DikkerS.SilbertL. J.HassonU.ZevinJ. D. (2014). On the same wavelength: predictable language enhances speaker–listener brain-to-brain synchrony in posterior superior temporal gyrus. J. Neurosci. 34, 6267–6272. 10.1523/JNEUROSCI.3796-13.2014
16
DikkerS.WanL.DavidescoI.KaggenL.OostrikM.McClintockJ.et al. (2017). Brain-to-brain synchrony tracks real-world dynamic group interactions in the classroom. Curr. Biol. 27, 1375–1380. 10.1016/j.cub.2017.04.002
17
DuanL.LiuW.-J.DaiR.-N.LiR.LuC.-M.HuangY.-X.et al. (2013). Cross-brain neurofeedback: scientific concept and experimental platform. PLoS ONE8:e64590. 10.1371/journal.pone.0064590
18
DumasG.FairhurstM. T. (2021). Reciprocity and alignment: quantifying coupling in dynamic interactions. R. Soc. Open Sci. 8:210138. 10.1098/rsos.210138
19
DumasG.NadelJ.SoussignanR.MartinerieJ.GarneroL. (2010). Inter-brain synchronization during social interaction. PLoS ONE5:e12166. 10.1371/journal.pone.0012166
20
FangZ.RayL. B.OwenA. M.FogelS. M. (2019). Brain activation time-locked to sleep spindles associated with human cognitive abilities. Front. Neurosci. 13:46. 10.3389/fnins.2019.00046
21
Fazel-RezaiR.AllisonB. Z.GugerC.SellersE. W.KleihS. C.KüblerA. (2012). P300 brain computer interface: current challenges and emerging trends. Front. Neuroeng. 5:14. 10.3389/fneng.2012.00014
22
GarrodS.PickeringM. J. (2009). Joint action, interactive alignment, and dialog. Top. Cogn. Sci. 1, 292–304. 10.1111/j.1756-8765.2009.01020.x
23
GoldsteinP.Weissman-FogelI.DumasG.Shamay-TsooryS. G. (2018). Brain-to-brain coupling during handholding is associated with pain reduction. Proc. Natl. Acad. Sci. U.S.A. 115, E2528–E2537. 10.1073/pnas.1703643115
24
GrennanG.BalasubramaniP. P.AlimF.Zafar-KhanM.LeeE. E.JesteD. V.et al. (2021). Cognitive and neural correlates of loneliness and wisdom during emotional bias. Cereb. Cortex31, 3311–3322. 10.1093/cercor/bhab012
25
GuevaraM. A.Corsi-CabreraM. (1996). EEG coherence or EEG correlation?Int. J. Psychophysiol. 23, 145–153. 10.1016/S0167-8760(96)00038-4
26
GvirtsH. Z.PerlmutterR. (2020). What guides us to neurally and behaviorally align with anyone specific? A neurobiological model based on fNIRS hyperscanning studies. The neuroscientist: a review. J. Bring. Neurobiol. Neurol. Psychiatry26, 108–116. 10.1177/1073858419861912
27
HariR.HimbergT.NummenmaaL.HämäläinenM.ParkkonenL. (2013). Synchrony of brains and bodies during implicit interpersonal interaction. Trends Cogn. Sci. 17, 105–106. 10.1016/j.tics.2013.01.003
28
HassonU. (2004). intersubject synchronization of cortical activity during natural vision. Science303, 1634–1640. 10.1126/science.1089506
29
HoehlS.FairhurstM.SchirmerA. (2021). Interactional synchrony: signals, mechanisms and benefits. Soc. Cogn. Affect. Neurosci. 16, 5–18. 10.1093/scan/nsaa024
30
KinreichS.DjalovskiA.KrausL.LouzounY.FeldmanR. (2017). Brain-to-brain synchrony during naturalistic social interactions. Sci. Rep. 7:17060. 10.1038/s41598-017-17339-5
31
KohlS. H.MehlerD. M. A.LührsM.ThibaultR. T.KonradK.SorgerB. (2020). The potential of functional near-infrared spectroscopy-based neurofeedback – a systematic review and recommendations for best practice. Front. Neurosci.14:594. 10.3389/fnins.2020.00594
32
KoikeT.SumiyaM.NakagawaE.OkazakiS.SadatoN. (2019). What makes eye contact special? Neural substrates of on-line mutual eye-gaze: a hyperscanning fMRI STUDY. eNeuro6, 401–412. 10.1523/ENEURO.0284-18.2019
33
KoikeT.TanabeH. C.OkazakiS.NakagawaE.SasakiA. T.ShimadaK.et al. (2016). Neural substrates of shared attention as social memory: a hyperscanning functional magnetic resonance imaging study. Neuroimage125, 401–412. 10.1016/j.neuroimage.2015.09.076
34
KovacevicN.RitterP.TaysW.MorenoS.McIntoshA. R. (2015). My virtual dream”: collective neurofeedback in an immersive art environment. PloS ONE10:e0130129. 10.1371/journal.pone.0130129
35
KovachC. K. (2017). A biased look at phase locking: brief critical review and proposed remedy. IEEE Transac. Signal Proc. 65, 4468–4480. 10.1109/TSP.2017.2711517
36
LachauxJ. P.RodriguezE.MartinerieJ.VarelaF. J. (1999). Measuring phase synchrony in brain signals. Hum. Brain Mapp. 8, 194–208.
37
LeeY.-E.KwakN.-S.LeeS.-W. (2020). A real-time movement artifact removal method for ambulatory brain-computer interfaces. IEEE Transac. Neural Syst. Rehabil. 28, 2660–2670. 10.1109/TNSRE.2020.3040264
38
LiuN.MokC.WittE. E.PradhanA. H.ChenJ. E.ReissA. L. (2016). NIRS-based hyperscanning reveals inter-brain neural synchronization during cooperative jenga game with face-to-face communication. Front. Hum. Neurosci. 10:82. 10.3389/fnhum.2016.00082
39
MayoO.GordonI. (2020). In and out of synchrony-behavioral and physiological dynamics of dyadic interpersonal coordination. Psychophysiology57:e13574. 10.1111/psyp.13574
40
MehrkanoonS.BreakspearM.BritzJ.BoonstraT. W. (2014). Intrinsic coupling modes in source-reconstructed electroencephalography. Brain Connect. 4, 812–825. 10.1089/brain.2014.0280
41
MillerJ. G.VrtičkaP.CuiX.ShresthaS.HosseiniS. M. H.BakerJ. M.et al. (2019). Inter-brain synchrony in mother-child dyads during cooperation: an fNIRS hyperscanning study. Neuropsychologia124, 117–124. 10.1016/j.neuropsychologia.2018.12.021
42
MisakiM.KerrK. L.RatliffE. L.CosgroveK. T.SimmonsW. K.MorrisA. S.et al. (2021). Beyond synchrony: the capacity of fMRI hyperscanning for the study of human social interaction. Soc. Cogn. Affect. Neurosci. 16, 84–92. 10.1093/scan/nsaa143
43
MoreauQ.DumasG. (2021). Beyond correlation versus causation: multi-brain neuroscience needs explanation. Trends Cogn. Sci. 25, 542–543. 10.1016/j.tics.2021.02.011
44
MullenT. R.KotheC. A. E.ChiY. M.OjedaA.KerthT.MakeigS.et al. (2015). Real-time neuroimaging and cognitive monitoring using wearable dry EEG. IEEE Trans. Biomed. Eng. 62, 2553–2567. 10.1109/TBME.2015.2481482
45
MüllerV.LindenbergerU. (2011). Cardiac and respiratory patterns synchronize between persons during choir singing. PLoS ONE6:e24893. 10.1371/journal.pone.0024893
46
NijholtA. (2019). Brain Art: Brain-Computer Interfaces for Artistic Expression. Springer. Available online at: https://play.google.com/store/books/details?id=qC6aDwAAQBAJ (accessed March 29, 2021).
47
NolteG.BaiO.WheatonL.MariZ.VorbachS.HallettM. (2004). Identifying true brain interaction from EEG data using the imaginary part of coherency. Clin. Neurophysiol. 115, 2292–2307. 10.1016/j.clinph.2004.04.029
48
NovembreG.IannettiG. D. (2021). Hyperscanning alone cannot prove causality. Multibrain Stimul. Can. Trends Cogn. Sci. 25, 96–99. 10.1016/j.tics.2020.11.003
49
NoyL.Levit-BinunN.GollandY. (2015). Being in the zone: physiological markers of togetherness in joint improvisation. Front. Hum. Neurosci. 9:187. 10.3389/fnhum.2015.00187
50
NozawaT.SasakiY.SakakiK.YokoyamaR.KawashimaR. (2016). Interpersonal frontopolar neural synchronization in group communication: an exploration toward fNIRS hyperscanning of natural interactions. Neuroimage133, 484–497. 10.1016/j.neuroimage.2016.03.059
51
OppenheimA. V. (1999). Discrete-Time Signal Processing. Pearson Education India. Available online at: http://182.160.97.198:8080/xmlui/bitstream/handle/123456789/489/13.%20Cepstrum%20Analysis%20and%20Homomorphic%20Deconvolution.pdf?sequence=14
52
OullierO.de GuzmanG. C.JantzenK. J.LagardeJ.KelsoJ. A. S. (2008). Social coordination dynamics: measuring human bonding. Soc. Neurosci. 3, 178–192. 10.1080/17470910701563392
53
PanY.DikkerS.GoldsteinP.ZhuY.YangC.HuY. (2020). Instructor-learner brain coupling discriminates between instructional approaches and predicts learning. Neuroimage211:116657. 10.1016/j.neuroimage.2020.116657
54
Perez RepettoL.JasminE.FombonneE.GiselE.CoutureM. (2017). Longitudinal study of sensory features in children with autism spectrum disorder. Autism Res. Treat. 2017:1934701. 10.1155/2017/1934701
55
PérezA.DumasG.KaradagM.DuñabeitiaJ. A. (2019). Differential brain-to-brain entrainment while speaking and listening in native and foreign languages. Cortex111, 303–315. 10.1016/j.cortex.2018.11.026
56
PickeringM. J.GarrodS. (2004). The interactive-alignment model: developments and refinements. Behav. Brain Sci. 27, 212–225. 10.1017/S0140525X04450055
57
PickeringM. J.GarrodS. (2013). Authors' response: forward models and their implications for production, comprehension, and dialogue [Review of Authors' response: forward models and their implications for production, comprehension, and dialogue]. Behav. Brain Sci. 36, 377–392. 10.1017/S0140525X12003238
58
PietromonacoP. R.CollinsN. L. (2017). Interpersonal mechanisms linking close relationships to health. Am. Psychol. 72, 531–542. 10.1037/amp0000129
59
ReindlV.GerloffC.ScharkeW.KonradK. (2018). Brain-to-brain synchrony in parent-child dyads and the relationship with emotion regulation revealed by fNIRS-based hyperscanning. Neuroimage178, 493–502. 10.1016/j.neuroimage.2018.05.060
60
ReineroD. A.DikkerS.Van BavelJ. J. (2021). Inter-brain synchrony in teams predicts collective performance. Soc. Cogn. Affect. Neurosci. 16, 43–57. 10.1093/scan/nsaa135
61
SalminenM.JárveláS.RuonalaA.HarjunenV.JacucciG.HamariJ.et al. (2019). Evoking physiological synchrony and empathy using social VR with biofeedback. IEEE Transac. Affect. Comput. 1. 10.1109/TAFFC.2019.2958657
62
ScholkmannF.HolperL.WolfU.WolfM. (2013). A new methodical approach in neuroscience: assessing inter-personal brain coupling using functional near-infrared imaging (fNIRI) hyperscanning. Front. Hum. Neurosci. 7:813. 10.3389/fnhum.2013.00813
63
ShawJ. C. (1984). Correlation and coherence analysis of the EEG: a selective tutorial review. Int. J. Psychophysiol. 1, 255–266. 10.1016/0167-8760(84)90045-X
64
StephensG. J.SilbertL. J.HassonU. (2010a). Speaker–listener neural coupling underlies successful communication. Proc. Natl. Acad. Sci. 107, 14425–14430. 10.1073/pnas.1008662107
65
StephensG. J.SilbertL. J.HassonU. (2010b). Speaker–listener neural coupling underlies successful communication. Proc. Natl. Acad. Sci. U.S.A. 107, 14425–14430.
66
TangH.MaiX.WangS.ZhuC.KruegerF.LiuC. (2016). Interpersonal brain synchronization in the right temporo-parietal junction during face-to-face economic exchange. Soc. Cogn. Affect. Neurosci. 11, 23–32. 10.1093/scan/nsv092
67
TognoliE.LagardeJ.DeGuzmanG. C.KelsoJ. A. S. (2007). The phi complex as a neuromarker of human social coordination. Proc. Natl. Acad. Sci. U.S.A. 104, 8190–8195. 10.1073/pnas.0611453104
68
Van BerkumJ. J. A.HollemanB.NieuwlandM.OttenM.MurreJ. (2009). Right or wrong? The brain's fast response to morally objectionable statements. Psychol. Sci. 20, 1092–1099. 10.1111/j.1467-9280.2009.02411.x
69
van HoogdalemL. E.FeijsH. M. E.BramerW. M.IsmailS. Y.van DongenJ. D. M. (2020). The effectiveness of neurofeedback therapy as an alternative treatment for autism spectrum disorders in children: a systematic review. J. Psychophysiol. 35, 102–115. 10.1027/0269-8803/a000265
70
van UlzenN. R.LamothC. J. C.DaffertshoferA.SeminG. R.BeekP. J. (2008). Characteristics of instructed and uninstructed interpersonal coordination while walking side-by-side. Neurosci. Lett. 432, 88–93. 10.1016/j.neulet.2007.11.070
71
VarletM.MarinL.LagardeJ.BardyB. G. (2011). Social postural coordination. J. Exp. Psychol. Hum. Percept. Perform. 37, 473–483. 10.1037/a0020552
72
WilliamsN. S.McArthurG. M.BadcockN. A. (2020). 10 years of EPOC: a scoping review of Emotiv's portable EEG device. bioRxiv [Preprint]. 10.1101/2020.07.14.202085
73
WrightM.MomeniA. (2011). Opensound Control: State of the Art 2003. Available online at: http://opensoundcontrol.org/files/Open+Sound+Control-State+Of+The+Art.Pdf
74
YunK. (2013). On the same wavelength: face-to-face communication increases interpersonal neural synchronization [Review of on the same wavelength: face-to-face communication increases interpersonal neural synchronization]. J. Neurosci. 33, 5081–5082. 10.1523/JNEUROSCI.0063-13.2013
75
ZammA.DebenerS.BauerA.-K. R.BleichnerM. G.DemosA. P.PalmerC. (2018a). Amplitude envelope correlations measure synchronous cortical oscillations in performing musicians. Ann. N. Y. Acad. Sci. 1423, 251–263. 10.1111/nyas.13738
76
ZammA.WangY.PalmerC. (2018b). Musicians' natural frequencies of performance display optimal temporal stability. J. Biol. Rhythms33, 432–440. 10.1177/0748730418783651
Summary
Keywords
hyperscanning, neurofeedback, brain-computer-interface, EEG, inter-brain coupling, real-world neuroscience
Citation
Chen P, Hendrikse S, Sargent K, Romani M, Oostrik M, Wilderjans TF, Koole S, Dumas G, Medine D and Dikker S (2021) Hybrid Harmony: A Multi-Person Neurofeedback Application for Interpersonal Synchrony. Front. Neuroergon. 2:687108. doi: 10.3389/fnrgo.2021.687108
Received
29 March 2021
Accepted
16 June 2021
Published
12 August 2021
Volume
2 - 2021
Edited by
Anton Nijholt, University of Twente, Netherlands
Reviewed by
Shigeyuki Ikeda, RIKEN Center for Advanced Intelligence Project (AIP), Japan; Aleksandra Vuckovic, University of Glasgow, United Kingdom
Updates
Copyright
© 2021 Chen, Hendrikse, Sargent, Romani, Oostrik, Wilderjans, Koole, Dumas, Medine and Dikker.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Suzanne Dikker suzanne.dikker@nyu.edu
This article was submitted to Neurotechnology and Systems Neuroergonomics, a section of the journal Frontiers in Neuroergonomics
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.