The sensorimotor basis of subjective experience in social synchronization behavior

The sensorimotor contingency (SMC) theory addresses action and perception as constitutive factors to one another–what is being perceived depends on what we do and vice versa. Accordingly, perception is seen as an active process of probing the environment and receiving feedback from it. These action-effect patterns may also be a predominant factor in social interactions. They can manifest in the phenomenon of synchronization, for example, when applause, gait, or posture in conversations synchronize unintentionally and form the basis of the concept of socially deployed sensorimotor contingencies (socSMCs). In this study, we introduce and compare measures used to study complex systems in order to quantify the information-theoretic basis of socSMCs. Two human participants had to synchronize arm movements in a full body virtual reality (VR) environment with each other. We aimed to evaluate the information-theoretic measures transfer entropy and mutual information in this complex motion synchronization task. Furthermore, in our experiment participants shared a mutual sensation of synchronicity and creativity for their interaction solely based on their movements. These subjective ratings of synchronicity and creativity can be predicted using transfer entropy and mutual information, showing that informational coupling between agents is relevant to subjective experience of interaction as described by the concept of socSMCs.

1 Introduction

Classical cognitive science theories explain cognition and perception as the results of internal representations of the external world. These internal representations then lead to actions. However, recently there has been a shift in cognitive sciences towards more enactive action-oriented approaches to cognition and perception such as the sensorimotor contingency (SMC) theory (O’Regan and Noë, 2001; Noy et al., 2015; O’Regan and Noe, 2011; O’Regan and Noë, 2001). In contrast to classical frameworks, the theory describes perception and cognition as the result of probing the environment with our actions and perceiving its effects. These effects then form the basis for new actions — action and perception are thus influencing each other bidirectionally (Engel et al., 2013; Kaspar et al., 2014; König et al., 2013). According to the SMC theory, perception and cognition are understood as active processes based on these action-effect patterns (O’Regan and Noë, 2001).

1.1 Related work: motion coordination analysis in human interaction

The SMC theory can be extended to social interactions between biological and/or artificial agents. Accordingly, complex social interactions are viewed as being grounded in continuous sensorimotor patterns (Lübbert et al., 2021). These dependencies are described by the concept of socially relevant sensorimotor contingencies (socSMCs), which expands the classic SMC theory by applying it to the domain of social interactions (De Jaegher et al., 2010; Maye and Engel, 2016; Lübbert et al., 2021). Sensorimotor interactions are thought to be the fundamentals that provide knowledge about the intentions, beliefs, and personalities of other agents. As in the SMC theory, the building of social sensorimotor contingencies relies on the interaction between perception and action. These action-effect contingencies can explain behavior in everyday life situations. For example, in conversations we intuitively mirror our counterparts’ movements, facial expressions, or postures. Individuals in groups synchronize their movements in rituals, while dancing, or during applause and often they harmonize unintentionally during joint action.

The socSMCs concept suggests that sensorimotor contingencies are essential for the effective coupling of agents during social interactions (Lübbert et al., 2021). It describes that informational and sensorimotor coupling is the basis of successful social interactions, which can be described at three different levels of coupling between social agents: unidirectional information sharing (check SMCs), bidirectional coupling (sync SMCs), and multidirectional coupling (unite SMCs). Check SMCs involve unidirectional coupling with one agent sharing information with another one. Check SMCs may lead to social entrainment which in turn can result in sync SMCs that are based on bidirectional coupling between both agents sharing information and grounding their decisions and sensorimotor interaction on this shared information. This informational coupling may enable agents to predict the decisions of their counterpart. Unite SMCs implicate coupling between multiple agents, e.g., emergence of joint concepts or group habits, which are not in the focus of the present study.

1.2 Assessing agent coupling using information-theoretic measures

In this study, we investigate sensorimotor and informational coupling by conceptualizing two interacting agents as nodes of a coupled system, and quantifying the extent of their coupling using measures of information transfer (Fusaroli and Tylén, 2016). Specifically, we use transfer entropy as a measure which determines information flow between two processes to quantify the directed interaction between two social agents (Orange and Abaid, 2015) and mutual information, a measure which serves to capture shared information between processes, to quantify shared information between interacting agents.

Recently, these information-theoretic measures received a lot of interest in neuroscience (Lizier, 2010; Vincente et al., 2010). Transfer entropy is typically used to estimate the directional information transfer between two coupled processes (Wibral et al., 2014). Mutual information, on the other hand, is a measure to quantify the uncertainty reduction of one process given information of the coupled process and has been used successfully to measure inter-brain coupling in social interactions (Naeem et al., 2012).

In this study we utilized a motion synchronization task to investigate the information-theoretic basis of socSMCs. Recently, a series of experiments investigated social interactions using the mirror game (Noy et al., 2011), a motion synchronization task based on improvisation in theater and music, in which the degree of synchronization between agents can be quantified. The mirror game serves as a paradigm for investigating synchronization of movements and joint improvisation, in the original study by Noy and colleagues (2011) with simple one-dimensional hand movements. Two participants had the task to synchronize their hand movements, while holding handles and moving them on a given trajectory. Participants were instructed to either lead or follow the movements of their counterpart or to jointly improvise movements without designated leader or follower roles. Noy and colleagues (2011) showed that during improvisation participants produced more complex and synchronous movements. A more recent study investigated the mirror game in a naturalistic and complex two-dimensional scenario and reported the subjective sensation of synchronization to be related to performance parameters derived from participants’ movements (Llobera et al., 2016). Moreover, in a dyadic interaction task in which two partners had to induce rotations of a rope, it was demonstrated that the difference between partners’ individual performances predicted the dyadic performance, suggesting that joint action arises from relative performance differences rather than individual performances (Huys et al., 2018). In a more recent dyadic interaction study involving haptic information exchange, Granger causality analysis revealed that directional information flow increased with task difficulty (Colomer et al., 2023). These findings emphasize a substantial interdependence of action and perception in social interactions. In the current study, we investigate informational and sensorimotor coupling of movement synchronization behavior and hypothesize increased bidirectional coupling during improvisation compared to instructed behavior.

In typical dyadic interaction tasks, participants are instructed to take one of three roles when interacting: leading, following, or jointly improvising. These roles are based on social hierarchies. Interactions can have a hierarchical structure that results in leader and follower roles, but real-life situations often involve joint action and improvisation without a designated leader or follower: for example, when people have to synchronize their actions to jointly lift an object, movement information has to be shared bidirectionally. Hierarchical structures can also emerge in improvisation scenarios and can be dependent on countless observable factors, such as appearance, gender, voice and many more. Visualizing the own and the counterpart’s bodies only as neutral, animated avatars in a virtual environment and thereby reducing the human body to its movements is essential for investigating motion synchronization without confounding factors. For example, factors like perceived gender, age, or even clothing may introduce a bias or additional layers of interpretation that go beyond pure motion synchronisation. We developed a virtual reality (VR) setup which uses 3-dimensional, naturalistic movements, to increase ecological validity and implications for real life compared to one-dimensional mirror games. This study is, to the best of our knowledge, the first to perform a three-dimensional movement synchronization task with movement quantification in VR.

2 Materials and methods

2.1 Participants

A total of 50 participants without any record of psychiatric or neurological illnesses were recruited and randomly assigned to 25 pairs of two players, randomly mixing male and female participants. Participants of each pair arrived independently of each other and did not meet before or during the experiment. One of the 25 pairs had to be excluded from the analysis because one participant interrupted the experiment due to illness. The final sample consisted of 48 participants (aged 19–33 years; mean = 22.79 years, standard deviation (SD) = 2.73; 28 female and 20 male) who were eligible for the analyses. Participants gave written consent and were financially compensated for their participation. The study was approved by the local ethics committee of the Medical Association in Hamburg, Germany, and adhered to the Declaration of Helsinki.

2.2 Hardware

The participants’ motion was tracked by two Xsens Motion Tracking Systems (MTw Awinda, Xsens, Netherlands) with a 60 Hz sampling rate using eleven sensors on the upper body, which were placed unilaterally on head, pelvis, sternum and bilaterally on shoulders, hands, upper and lower arms. Two identical virtual environments were created in Unity3D (Unity Technologies, USA) and connected via TCP sockets. Each environment was screened on an HTC Vive Pro VR-System (HTC/Valve, Taiwan) driven by a GeForce RTX 2080 graphics card (NVIDIA, USA). The visual motion pipeline utilized motion capture data from the Xsens MTw system in combination with real-time rendering in Unity3D. To ensure a natural experience, the pipeline was designed so that participants would not perceive any end-to-end delay. According to the manufacturer, the Xsens MTw system with 10 sensors exhibits a streaming latency of approximately 20 ms. Real-time rendering in Unity at 60 Hz introduces an additional latency of one to two frames (approximately 17–33 ms). This results in a total system latency of about 37–53 ms, which remains well below the perceptual threshold for detecting delays in synchronized movement tasks.

2.3 Software

Motion trajectories were broadcasted in real-time and rendered via Unity3D to each VR-System using the HTC Vive system. Mutual information and transfer entropy were calculated with the Java Information Dynamics Toolkit (JIDT) in Python (Lizier, 2014). The repeated measures correlations calculation of felt synchronicity, creativity, sympathy and focus between both participants were calculated using pingouin (v. 0.3.3) a statistical package for Python. For the generalized linear-mixed model analysis SPSS 27 (IBM, USA) was used.

2.4 Experimental procedure

Participants of each pair were tested in separate rooms and did not meet until the end of the VR experiment. They were not informed in any way about their partner beforehand. Participants received detailed instructions about the procedure both orally and in written form. Before the movement synchronization task started, participants completed the following personality questionnaires: autism quotient (AQ; Baron-Cohen et al., 2001), SPF (Saarbrücker Persönlichkeitsfragebogen für Empathie, German version of the Interpersonal Reactivity Index; Davis, 1983; Paulus, 2009), and NEO-FFI (Costa and McCrae, 1992). The AQ is a self-report inventory aiming to test for traits associated with the autism spectrum. The SPF is a self-report questionnaire to assess individual differences in empathic personality traits. And the NEO-FFI (five factor inventory) is a short version of the NEO PI-R (personality inventory) assessing the Big Five personality traits: neuroticism, extraversion, openness to experience, agreeableness, and conscientiousness. Additionally, participants were asked to complete a customized questionnaire about their current physical wellbeing and regarding their prior experience with VR scenarios, musical instruments, and dancing.

Participants were equipped with the Xsens motion tracking system. After the calibration of the motion tracking system, participants put on the VR goggles. The functionality and calibration of the motion tracking and the VR system were checked by the experimenter. At this moment, participants had the possibility to gain a first impression and get used to the VR environment. Participants were able to see the movements of their avatar in first person (Figure 1A). Movements of both participants were instantaneously projected onto two avatars without noticeable delay. At this point the two avatars were visually separated using a virtual wall and, thus, participants could only see parts of their own avatar. This procedure led to the impression that the avatar was fully controlled by the participant’s own movements. Once both participants were ready, the VR experiment started, in which both participants could see each other.

Figure 1

Figure 1. Virtual reality setup. (A) Perspective of the scene view in the HTC Vive system (white: the own avatar, red: the other participant’s avatar). This is the scene that participants were seeing during the experiment. For both participants the partner was displayed in red. (B) Bird’s eye view of the game (not visible for participants).

In the virtual environment, the two animated avatars stood face-to-face to each other. Each participant experienced an egocentric perspective, perceiving the avatar’s body as their own. Both partners were instructed to synchronize their arm movements. To enable mirror movements in the virtual setup, one partner used their right arm, while the other partner to used their left (Figures 1A,B). There were no restrictions concerning the nature of arm movement. Partners were instructed to focus only on synchronizing their arm movements and to avoid unnecessary movements of the other limbs.

Before each trial, participants were instructed to take one of three different roles. They could either lead (L), follow (F) or jointly improvise (JI) arm movements. The leader was instructed to produce arm movements which the partner should follow. In the joint improvisation task, partners were instructed to synchronize their movements without a designated leader or follower. The experiment consisted of 15 trials lasting 2 min each. The total interaction time was 37 min. The order of conditions was L, F, JI–JI, L, F–L, F, JI–L, F, JI–JI, F, L. Between trials there was a pause of 50 s in which partners were asked to rate their experience during the last trial regarding focus, synchronicity, creativity and sympathy for their counterpart on a continuous scale (range 1–10). A short resting period followed in which they received the instruction for the next trial.

The experiment was followed up with a questionnaire regarding the experience of the interaction. Participants rated their perceived role in the joint improvisation condition on an ordinal scale (range 1–10, 1: Leader; 10: Follower) and whether the interaction consisted of co-leadership or changing leader/follower roles (range 1–10, 1: co-leadership, 10: changing leader/follower roles).

2.5 Data collection and preprocessing

Motion data from 48 participants interacting in leader, follower and joint improvisation conditions was measured (5 trials per condition, each lasting 2 min) resulting in 360 trials in total (24 pairs acting in 15 trials each, 240 trials leader/follower, 120 trials joint improvisation). The eleven sensors on the upper body provided data in the form of quaternions describing the relative orientations. Quaternions are 4-dimensional representations of rotations in 3-dimensional space (w, x, y, z) and represent the rotation by an angle (α) around a unit axis vector v = (x, y, z). From this the quaternion is derived as:

$\mathrm{X}=\mathrm{x}\times \sin \left(0.5\times \mathrm{\alpha }\right)$

$\mathrm{Y}=\mathrm{y}\times \sin \left(0.5\times \mathrm{\alpha }\right)$

$\mathrm{Z}=\mathrm{z}\times \sin \left(0.5\times \mathrm{\alpha }\right)$

$\mathrm{W}=\cos \left(0.5\times \mathrm{\alpha }\right)$

The quaternions are mapped to a uniform avatar independent of the participants body dimensions in Unity. Therefore, further analysis was independent of the participants body dimensions. The origin of the coordinate system was the pelvis center. Further analyses were limited to the three-dimensional trajectories of the active hand. The data was recorded at 60 Hz. The data was filtered to remove any jitter in the movement using a Savitzky-Golay filter (polynomial of third degree, window length of 17, which represents 0.28 s at a sampling frequency at 60 Hz). Only one of 360 trials was excluded due to technical failure (n = 1).

2.6 Transfer entropy

Transfer entropy (Schreiber, 2000; Barnett et al., 2009) is a non-parametric measure, which represents the directional transfer of information between two processes by calculating the reduction of uncertainty with the current information of one system in predicting the future state of the other system. Transfer entropy is measured in bits, the basic unit of information.

Here we use transfer entropy to calculate the flow of information from one participant’s hand movements to the other participant’s hand movements. In a movement-based leader-follower scenario, information from the leader is transferred to the follower, influencing the follower’s future movements. If information of the leader’s movement can be used to predict the future movements of the follower, transfer entropy from leader to follower is larger than transfer entropy from follower to leader. Accordingly, this method can be used to classify leaders and followers in corresponding scenarios as well as in open-end improvisation. In contrast, an improvisation task should require a more balanced exchange of information between both parties. Here, transfer entropy may determine leader and follower tendencies.

We computed transfer entropy on continuous data using the Kraskov-Stögbauer-Grassberger transfer entropy estimator (Kraskov et al., 2004) and searched for the maximum transfer entropy in a search space of 1,500 ms on each individual axis of movement (x, y, z) per trial (Figure 2). We tested the transfer entropy between conditions using paired t-tests (Table 1). Furthermore, given the directional nature of transfer entropy, from leader to follower, TE (LF), and from follower to leader, TE (FL), we calculated net transfer entropy (netTE):

$\text{netTE}=\text{TE}\left(\text{LF}\right)-\text{TE}\left(\text{FL}\right)$

Figure 2

Figure 2. Example data of leader (red line) and follower (blue line) trajectories showing synchronized movements in three axes (x, y, z) in a leader/follower trial.

Table 1

Table 1. Statistical evaluation of transfer entropy between follower and leader (F), leader and follower (L), joint improvisation (JI). Only the comparison between JI and F conditions shows significant results for the y-axis (df = 23; p < 0.05, Bonferroni corrected).

Based on the assumption that information flow from leader to follower should be greater than vice versa, netTE should be positive in leader-follower scenarios. We tested this netTE from leader to follower using t-tests against a normal distribution with a mean of zero (Nakayama et al., 2017). To investigate the influence of transfer entropy on subjective motion ratings we first computed tranfer entropy on each of the three axes (x,y,z) of the motion tracker data. We then reduced dimensionality of these data by applying principal component analysis (PCA), retaining only the first principal component.

2.7 Mutual information

Mutual information is a symmetric statistic that quantifies the amount of information that two processes have in common. Mutual information represents the reduction of uncertainty about one process by learning from the other process and vice versa. In a motion synchronization task mutual information represents the information both participants base their consecutive movements on. In improvisation scenarios, participants have to rely on their shared information of the process to conclude further action. In contrast, leader-follower scenarios do not rely on the bi-directional exchange of information as heavily as improvisation tasks. To calculate mutual information, we used the Java Information Dynamic Toolkit’s (JIDT) implementation of mutual information on continuous data, also utilizing the Kraskov-Stögbauer-Grassberger estimator (Kraskov et al., 2004). We first computed mutual information on each of the three axes (x, y, z) of the motion tracker data. To reduce dimensionality, we then applied principal component analysis (PCA), retaining only the first principal component.

2.8 Task performance parameters

Similar to the original mirror game (Noy, et al., 2011) we calculated the motion velocity and velocity error. Motion velocity was assessed by calculating the mean of the three-dimensional Euclidean distance per time point of both participants in 3D space as:

$v=\sqrt{{\left(x^{\prime }\right)}^2+{\left(y^{\prime }\right)}^2+{\left(z^{\prime }\right)}^2}$

and mean velocity as:

$\overline{v}={\displaystyle \sum _{i=1}^n}\frac{1}{n}{v}_i$

The velocity error was assessed as the absolute difference in velocities per time point.

$dv=\left|v1–v2\right|$

and mean velocity error as:

$\overline{dv}={\displaystyle \sum _{i=1}^n}\frac{1}{n}{dv}_i$

2.9 Statistical methods

Using the GENLINMIXED package in SPSS, we deployed a generalized linear mixed effect model (GLMM) to determine whether net transfer entropy, transfer entropy, mutual information and velocity error are predictive of the ratings of synchronicity and creativity after each trial. A hierarchical linear model was built with trials on the lowest level, participants on the second level and pairs on the third level. To reduce the number of features for this model, we applied principal component analysis (PCA) on mutual information and transfer entropy data to reduce the three axes of these measurements to the first principal component. In the linear mixed-model either synchronicity or creativity were used as outcome variable. Subject ID and Pair ID were considered as random effects, Trial ID as the repeated measure variable. A backward elimination procedure was used for feature selection. In the backward elimination process, the model was first fitted with all predictors (net transfer entropy, transfer entropy, mutual information and velocity error). Step-by-step the predictor with the highest p-value was removed from the model and the model was fitted again until only significant predictors and interactions remained.

3 Results

Participants performed movements varying in amplitude and frequency in the three axes (Figures 2A–C) and synchronized with each other. The motion tracker provided spatial position data as person-centered coordinates, originating at the centre of the pelvis, with the three axes oriented as follows: x = left/right, y = up/down, z = forward/backward. Unlike in the original mirror game (Noy et al., 2011), we did not observe the characteristic jitter of the follower’s motion velocity around the leader’s motion velocity, but instead a smoother trajectory lagging behind the leader’s movement.

3.1 Task performance

Motion velocity was significantly lower in JI than in L/F (t (23) = - 4.8870, p < 0.001). As a measure of synchronization, we calculated the velocity error (see Noy et al., 2011), which was not significantly different in JI compared to L/F trials (t (70) = - 0.86409, p = 0.396455). Mutual information in all axes was significantly higher in JI than in L/F trials (x-axis: t (23) = - 3.577, p = 0.002; y-axis: t (23) = - 3.285, p = 0.003; z-axis: t (23) = - 3.684, p = 0.001), indicating that participants based their consecutive movements more on shared information in the JI than in the L/F condition.

3.2 Leader- and followership

In order to classify leader- and followership in JI without a designated leader we calculated transfer entropy as a per-trial-measure of leader- and followership (Table 1; Figure 3). Transfer entropy can be calculated from leader to follower (L - > F), follower to leader (F - > L) and between the participants in joint improvisation (JI). Transfer entropy was only significantly different between conditions in the y-axis, showing smaller values for joint improvisation compared to follower condition (t (23) = 3.0941, p = 0.0033).

Figure 3

Figure 3. Transfer entropy [bits] from follower to leader, leader to follower, and under joint improvisation (JI) conditions across the three spatial axes. Plotted values represent medians with interquartile ranges; outliers are indicated by circles.

Net transfer entropy in L/F was positive in 77,1%, 79,2%, 72,9% (x-axis, y-axis, z-axis) of participant pairs (Figure 4) and differed significantly from zero between leader and follower (x-axis: t (23) = - 6.597, p < 0.001, y-axis: t (23) = - 5.6825, p < 0.001, z-axis: t (23) = - 7.1751, p < 0.001; Figure 4).

Figure 4

Figure 4. Net transfer entropy [bits] (leader - follower) across the three spatial axes. Plotted values represent medians with interquartile ranges.

3.3 Ratings

After each trial we assessed focus (Lead: 8.14 +- 1.33, Follow: 8.19 +- 1.51, JI: 8.29 +- 1.48), experienced synchronicity (Lead: 7.32 +- 1.73 Follow: 7.19 +- 1.61, JI: 7.27 +- 1.56), creativity (Lead: 7.20 +- 1.78, Follow 6.55 +- 1.73, JI: 6.85 +- 1.88) and sympathy (Lead: 8.31 +- 1.51, Follow: 8.38 +- 1.41, JI: 8.55 +- 1.33) We calculated the repeated measures correlation of synchronicity (r = 0.15, p = 0.002), sympathy (r = 0.076, p = 0.197), creativity (r = 0.116, p = 0.042), and focus (r = 0.11, p = 0.047) between pairs of participants.

We used a generalized linear mixed model to further investigate the relationship between the movement parameters and the ratings of perceived synchronicity and creativity of the participants. After backward elimination of non-significant factors, the GLMM revealed that mutual information of the two participants is predictive of the perceived synchronicity (F (1,613) = 6.485, p = 0.011, β = 0.021; results before backward elimination, see Supplementary Tables S1, S2). Furthermore, condition (F (2,608) = 3.322, p = 0.037) and the interaction of trial and condition are predictive of synchronicity (F (2,578) = 3.702, p = 0.025). Creativity ratings were predicted by the factors transfer entropy (F (1,431) = 10.65, p = 0.001, β = −0.022), and condition (F (2,527) = 15.305, p < 0.001).

In the post experimental inquiry, participants reported on their experience regarding follower- and leadership in the joint improvisation condition - whether the improvisation felt like switching between leader- and followership or a co-leadership in which participants were not aware of a designated leader. Participants rated on a scale from 1 (switching) to 10 (co-leadership) using a visual analogue scale (M = 4.12, SD = 2.87). When in a state of switching between leader- and followership, participants also reported whether they perceived themselves predominantly in the role of a follower or leader (rated on a scale from 1 (follower) to 10 (leader), M = 5.10, SD = 1.29). These results suggest that participants tend to perceive the improvisation condition as a switching between follower and leader instead of a co-leadership.

4 Discussion

This study has focused on the investigation of informational and sensorimotor coupling of human agents synchronizing their arm movements in a three-dimensional VR imitation task. In addition, the predictive value of these coupling measures on subjective experience of sensorimotor interactions was assessed. We instructed participants to interact in different scenarios and analyzed different information-theoretic measures, such as transfer entropy and mutual information. We showed that participants have a shared sensation, which can be partly predicted by coupling measures of their motion data. These results pave the way for future research investigating more open-ended, uninstructed interactions with dynamic coupling in social contexts. What is more, this study is the first to deploy human movement synchronization tasks in VR. For future studies, we propose this paradigm is suitable to investigate dynamic embodied interactions in more naturalistic scenarios.

4.1 The mirror game

Noy and colleagues (2011) introduced the mirror game as a motion synchronization task using one-dimensional sliders, in which participants were assigned leader or follower roles or instructed to improvise. Over the years, this experimental design has been refined, with a trend towards more naturalistic ecologically valid synchronization behavior. Recent, advances led to the development of a two-dimensional mirror game design (Llobera et al., 2016). Building on this trajectory, we have now advanced the mirror game into a three-dimensional version within a VR setup, utilizing motion tracking in combination with VR headsets.

In the original mirror game, movements on one-dimensional sliders have shown a characteristic jitter of the follower around the leader’s trajectory. In our study this characteristic jitter was absent, potentially caused by the increased difficulty of the task compared to previous mirror game versions. Moreover, comparing improvisation and leader/follower conditions showed that motion velocity in improvisation was generally slower with smaller errors in velocity, in contrast to the original mirror game, in which participants' interactions were faster with smaller velocity error (Noy et al., 2011). This may be due to the difference that Noy and colleagues studied experts in improvisation and used a less complex one-dimensional task in comparison to the free arm movements participants were able to perform in our experiment.

4.2 Socializing sensorimotor contingencies

Building on the concept of socially deployed sensorimotor contingencies (socSMCs) Lübbert and colleagues (2021) distinguish three different stages of human interactions: sync SMCs, check SMCs and unite SMCs. Of these stages, only sync SMCs and check SMCs are relevant in this study. Firstly, check SMCs involve unidirectional coupling: agents determine how their counterpart reacts and can form predictions about their counterpart’s actions. Secondly, sync SMCs involve bidirectional coupling, which can lead to entrainment of behavior and shared action goals. Here we lay the foundation of quantifying behavioral coupling by comparing information-theoretic measures in instructed leader/follower role-taking mimicking check SMCs and improvisation mimicking sync SMCs.

To this end, we applied two information-theoretic measures for the analysis of human movement interactions. Firstly, we were able to quantify the information flow between leaders and followers using transfer entropy and net transfer entropy in instructed leader-follower interactions. While transfer entropy itself did not show significant differences between conditions, net transfer entropy was significantly different from zero, correctly classifying follower- and leadership. Thereby, we substantiate findings that leadership can be predicted by the direction of transfer entropy, as shown previously in studies involving leader-follower scenarios in animals (Sun et al., 2014; Orange and Abaid, 2015; Crosato et al., 2018) and speaker-to-listener scenarios in humans (Dumas et al., 2014). Furthermore, our findings on net transfer entropy substantiate findings of more restricted human interaction tasks to classify leader- and followership (Nakayama et al., 2017).

Secondly, we deployed mutual information, which is sought to quantify shared information, upon which the actions of the two participants are based. Mutual information values were significantly higher in the joint improvisation task condition than in conditions with well-defined leader and follower roles, indicating that the process of interacting is more strongly based on shared information in dynamic role-taking than in instructed role-taking.

4.3 Subjective experience

Furthermore, we investigated whether participants’ shared sensation of synchronicity and creativity may be grounded in motion parameters and informational coupling in the movement synchronization task. Our results suggest that participants had a shared feeling of synchronicity and creativity that was essentially based on movement perception and generation as there was no verbal communication between participants possible. Therefore, we aimed to predict the sensation of synchronicity and creativity in our task from the information-theoretic measures using a generalized linear mixed model. Results show that mutual information between participants is a significant predictor for the sensation of synchronicity. We interpret mutual information as the information both participants share before making a decision about their next move. High mutual information values were predictive of high synchronicity ratings. Velocity error on the other hand, a directly motion-dependent measure of synchronicity in the task, showed no predictive value.

For the sensation of creativity, the measure of transfer entropy showed significant predictive power. Our results revealed that higher transfer entropy is associated with one’s own leadership. This suggests that a stronger adoption of the leadership role by oneself is accompanied by a higher sensation of creativity. Other studies have explored the role of social relationships on interpersonal movement coordination, revealing associations between social factors and movement coordination (Latif et al., 2014; Tschacher et al., 2014; Cornejo et al., 2018). For example, cross-correlation analysis of movements during dyadic conversations revealed that friends exhibited stronger mirror-like bodily coordination than strangers (Cornejo et al., 2018). The authors conclude that both form and intensity of interpersonal coordination are highly sensitive to the social context. The present study extends these findings by showing that, even when social factors are minimized–by using a virtual reality setup with neutral avatars–shared sensations of synchronicity and creativity can still emerge from the underlying movement coordination between participants.

4.4 Limitations of the study

Three-dimensional motion synchronization is a complex task for the participants to perform and the analysis requires simplification. Thus, only hand movements were considered here, excluding movements of other body parts to simplify the analysis. Even though we tried to keep the VR scenario as naturalistic as possible, participants may act differently compared to natural in-person synchronization. The structured nature of the mirror game focuses on a specific form of social interaction, namely, dyadic movement coordination within defined constraints. The mirror game, while widely used as a paradigm to study joint action and synchrony, captures only a subset of the rich and varied spectrum of social interactions that occur in everyday life. Our intention was to highlight the potential of VR as a tool to study certain aspects of social interactions under controlled and replicable conditions. We fully acknowledge that our findings are mostly informative for understanding interactive motor synchronisation and do not generalize to the entire complexity of social interactions, which often involve verbal communication, spontaneous behavior, nuanced social cues, and diverse social contexts. Additionally, most participants were young undergraduate students, thus future work should focus on a more diverse study sample.

5 Conclusion

Our study demonstrates that embodied motor synchronization as a form of social interaction can be studied in VR with animated avatars. We used motion tracking in a two-person mirror game allowing 3-dimensional movements. The findings of our study support the notion that dyadic movement coordination is grounded in modes of sensorimotor and informational coupling. These coupling modes, which are based on low-level sensorimotor interactions, can serve as a tool to predict subjective experience of dyadic movement coordination. Transfer entropy has proven to successfully distinguish leader- and followership in this complex movement task. Transfer entropy and mutual information were predictors of the sensation of synchronicity and creativity, showing that shared information as well as the exchange of information are relevant for the experience of interaction. Furthermore, these findings have potential implications for the understanding of neuronal coupling in the human brain, which could be investigated using non-invasive methods such as electroencephalography (EEG). The neuronal foundations of social cognition and especially predictive coding could be investigated by combining the VR setup with the investigation of neuronal coupling between brain areas and between agents based on EEG signals.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Medical Association Hamburg, Germany. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

JB: Data curation, Software, Investigation, Writing – review and editing, Conceptualization, Writing – original draft, Formal Analysis, Visualization, Methodology. MK: Data curation, Conceptualization, Methodology, Writing – review and editing, Software, Investigation. FG: Writing – review and editing, Methodology. AM: Investigation, Writing – review and editing, Methodology, Project administration, Conceptualization. AE: Conceptualization, Resources, Funding acquisition, Writing – review and editing, Supervision, Project administration. TS: Methodology, Conceptualization, Investigation, Writing – review and editing, Supervision, Formal Analysis.

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. This work was supported by grants from the EU (project “socSMCs”, H2020-641321) and the DFG (SFB “Cross-Modal Learning”, TRR169-261402652-B1/Z2). We acknowledge financial support from the Open Access Publication Fund of UKE - Universitätsklinikum Hamburg-Eppendorf

Acknowledgements

We would like to thank Karin Reimann, Dept. of Neurophysiology and Pathophysiology, UKE Hamburg, for participant recruitment.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Generative AI statement

The author(s) declare that no Generative AI was used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2025.1619710/full#supplementary-material

References

Barnett, L., Barrett, A. B., and Seth, A. K. (2009). Granger causality and transfer entropy are equivalent for gaussian variables. Phys. Rev. Lett. 103, 238701. doi:10.1103/PhysRevLett.103.238701

PubMed Abstract | CrossRef Full Text | Google Scholar

Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., and Clubley, E. (2001). The autism-spectrum quotient (AQ): evidence from asperger syndrome/high-functioning autism, males and females, scientists and mathematicians. J. Autism Dev. Disord. 31, 5–17. doi:10.1023/a:1005653411471

PubMed Abstract | CrossRef Full Text | Google Scholar

Colomer, C., Dhamala, M., Ganesh, G., and Lagarde, J. (2023). Granger geweke causality reveals information exchange during physical interaction is modulated by task difficulty. Hum. Mov. Sci. 92, 103139. doi:10.1016/j.humov.2023.103139

PubMed Abstract | CrossRef Full Text | Google Scholar

Cornejo, C., Hurtado, E., Cuadros, Z., Torres-Araneda, A., Paredes, J., Olivares, H., et al. (2018). Dynamics of simultaneous and imitative bodily coordination in trust and distrust. Front. Psychol. 9, 1546. doi:10.3389/fpsyg.2018.01546

PubMed Abstract | CrossRef Full Text | Google Scholar

Costa, P. T., and McCrae, R. R. (1992). “Professional manual: revised NEO personality inventory (NEO-PI-R) and NEO five-factor inventory (NEO-FFI),”, 61. Odessa, FL: Psychological Assessment Resources.

Google Scholar

Crosato, E., Jiang, L., Lecheval, V., Lizier, J. T., Wang, X. R., Tichit, P., et al. (2018). Informative and misinformative interactions in a school of fish. Swarm Intell. 12, 283–305. doi:10.1007/s11721-018-0157-x

CrossRef Full Text | Google Scholar

Davis, M. H. (1983). Measuring individual differences in empathy: evidence for a multidimensional approach. J. Pers. Soc. Psychol. 44, 113–126. doi:10.1037/0022-3514.44.1.113

CrossRef Full Text | Google Scholar

De Jaegher, H., Di Paolo, E., and Gallagher, S. (2010). Can social interaction constitute social cognition? Trends Cogn. Sci. 14, 441–447. doi:10.1016/j.tics.2010.06.009

PubMed Abstract | CrossRef Full Text | Google Scholar

Dumas, G., de Guzman, G. C., Tognoli, E., and Kelso, J. A. S. (2014). The human dynamic clamp as a paradigm for social interaction. Proc. Natl. Acad. Sci. U. S. A. 111, E3726–E3734. doi:10.1073/pnas.1407486111

PubMed Abstract | CrossRef Full Text | Google Scholar

Engel, A. K., Maye, A., Kurthen, M., and König, P. (2013). Where’s the action? The pragmatic turn in cognitive science. Trends Cogn. Sci. 17, 202–209. doi:10.1016/j.tics.2013.03.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Fusaroli, R., and Tylén, K. (2016). Investigating conversational dynamics: interactive alignment, interpersonal synergy, and collective task performance. Cogn. Sci. 40, 145–171. doi:10.1111/cogs.12251

PubMed Abstract | CrossRef Full Text | Google Scholar

Huys, R., Kolodziej, A., Lagarde, J., Farrer, C., Darmana, R., and Zanone, P.-G. (2018). Individual and dyadic rope turning as a window into social coordination. Hum. Mov. Sci. 58, 55–68. doi:10.1016/j.humov.2017.12.015

PubMed Abstract | CrossRef Full Text | Google Scholar

Kaspar, K., König, S., Schwandt, J., and König, P. (2014). The experience of new sensorimotor contingencies by sensory augmentation. Conscious Cogn. 28, 47–63. doi:10.1016/j.concog.2014.06.006

PubMed Abstract | CrossRef Full Text | Google Scholar

König, P., Wilming, N., Kaspar, K., Nagel, S., and Onat, S. (2013). Predictions in the light of your own action repertoire as a general computational principle. Behav. Brain Sci. 36(3), 219–220. doi:10.1017/S0140525X12002294

PubMed Abstract | CrossRef Full Text | Google Scholar

Kraskov, A., Stögbauer, H., and Grassberger, P. (2004). Estimating mutual information. Phys. Rev. E—Statistical, Nonlinear, Soft Matter Phys. 69, 066138. doi:10.1103/PhysRevE.69.066138

PubMed Abstract | CrossRef Full Text | Google Scholar

Latif, N., Barbosa, A. V., Vatikiotis-Bateson, E., Castelhano, M. S., and Munhall, K. G. (2014). Movement coordination during conversation. PloS One 9, e105036. doi:10.1371/journal.pone.0105036

PubMed Abstract | CrossRef Full Text | Google Scholar

Lizier, J. T. (2014). Jidt: an information-theoretic toolkit for studying the dynamics of complex systems. Front. Robot. AI 1, 11. doi:10.3389/frobt.2014.00011

CrossRef Full Text | Google Scholar

Lizier, J. T., Heinzle, J., Horstmann, A., Haynes, J. D., and Prokopenko, M. (2010). Multivariate information-theoretic measures reveal directed information structure and task relevant changes in fMRI connectivity. J. Comput. Neurosci. 30, 85–107. doi:10.1007/s10827-010-0271-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Llobera, J., Charbonnier, C., Chagué, S., Preissmann, D., Antonietti, J.-P., Ansermet, F., et al. (2016). The subjective sensation of synchrony: an experimental study. PLoS One 11, e0147008. doi:10.1371/journal.pone.0147008

PubMed Abstract | CrossRef Full Text | Google Scholar

Lübbert, A., Göschl, F., Krause, H., Schneider, T. R., Maye, A., and Engel, A. K. (2021). Socializing sensorimotor contingencies. Front. Hum. Neurosci. 15, 624610. doi:10.3389/fnhum.2021.624610

PubMed Abstract | CrossRef Full Text | Google Scholar

Maye, A., and Engel, A. K. (2016). “Extending sensorimotor contingencies to cognition,” in The pragmatic turn. Toward action-oriented views in cognitive science. Editors A. K. Engel, K. J. Friston, and D. Kragic (Cambridge, Massachusetts: The MIT Press), 175–191. doi:10.7551/mitpress/9780262034326.003.0011

CrossRef Full Text | Google Scholar

Naeem, M., McGinnity, T. M., Watson, D., Wong-Lin, K., Prasad, G., and Kelso, J. A. S. (2012). “Inter-brain mutual Information in social interaction tasks,” in Second international workshop on pattern recognition in NeuroImaging (PRNI). London, UK, 25–28. doi:10.1109/PRNI.2012.32

CrossRef Full Text | Google Scholar

Nakayama, S., Ruiz Marín, M., Camacho, M., and Porfiri, M. (2017). Plasticity in leader-follower roles in human teams. Sci. Rep. 7, 14562. doi:10.1038/s41598-017-14851-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Noy, L., Dekel, E., and Alon, U. (2011). The mirror game as a paradigm for studying the dynamics of two people improvising motion together. Proc. Natl. Acad. Sci. U. S. A. 108, 20947–20952. doi:10.1073/pnas.1108155108

PubMed Abstract | CrossRef Full Text | Google Scholar

Noy, L., Levit-Binun, N., and Golland, Y. (2015). Being in the zone: physiological markers of togetherness in joint improvisation. Front. Hum. Neurosci. 9, 187. doi:10.3389/fnhum.2015.00187

PubMed Abstract | CrossRef Full Text | Google Scholar

Orange, N., and Abaid, N. (2015). A transfer entropy analysis of leader-follower interactions in flying bats. Eur. Phys. J. Spec. Top. 224, 3279–3293. doi:10.1140/epjst/e2015-50235-9

CrossRef Full Text | Google Scholar

O’Regan, J. K., and Noë, A. (2001). A sensorimotor account of vision and visual consciousness. Behav. Brain Sci. 24, 939–973. doi:10.1017/s0140525x01000115

PubMed Abstract | CrossRef Full Text | Google Scholar

Paulus, C. (2009). Der Saarbrücker Persönlichkeitsfragebogen SPF(IRI) zur Messung von Empathie: psychometrische Evaluation der deutschen Version des Interpersonal Reactivity Index. Saarbrueck Personality Quest. Empathy Psychometric Eval. Ger. version Interpers. React. Index. doi:10.23668/psycharchives.9249

CrossRef Full Text | Google Scholar

Schreiber, T. (2000). Measuring information transfer. Phys. Rev. Lett. 85, 461–464. doi:10.1103/PhysRevLett.85.461

PubMed Abstract | CrossRef Full Text | Google Scholar

Sun, Y., Rossi, L. F., Shen, C.-C., Miller, J., Wang, X. R., Lizier, J. T., et al. (2014). Information transfer in swarms with leaders. Available online at: http://arxiv.org/abs/1407.0007 (Accessed December 16, 2021).

Google Scholar

Tschacher, W., Rees, G. M., and Ramseyer, F. (2014). Nonverbal synchrony and affect in dyadic interactions. Front. Psychol. 5, 1323. doi:10.3389/fpsyg.2014.01323

PubMed Abstract | CrossRef Full Text | Google Scholar

Vincente, R., Wibral, M., Lindner, R., and Pipa, G. (2010). Transfer entropy—a model-free measure of effective connectivity for the neurosciences. J. Comput. Neurosci. 30, 45–67. doi:10.1007/s10827-010-0262-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Wibral, M., Vicente, R., and Lindner, M. (2014). “Transfer entropy in neuroscience,” in Directed information measures in neuroscience. Understanding complex systems. Editors M. Wibral, R. Vicente, and J. Lizier (Berlin, Heidelberg: Springer). doi:10.1007/978-3-642-54474-3_1

CrossRef Full Text | Google Scholar