Development and validation of an art-inspired multimodal interactive technology system for a multi-component intervention for older people: a pilot study

1 Introduction

The World Health Organization (WHO) gives the term “health” the following definition: “a state of complete physical, mental and social wellbeing and not merely the absence of disease”. It's in this bio-psycho-social dimension that the WHO itself recognizes the existence of a significant body of evidence on the benefits of the arts on health. Specifically, the effectiveness of cultural activities with respect to individual health is emphasized, considering a range of aspects from physical wellbeing to quality of life, and from the perspective of social and community impact (Fancourt and Finn, 2019).

Cultural welfare is precisely based on a model that places visual arts, performing arts and cultural heritage at the service of the wellbeing of individuals and communities, through collaboration and integration of knowledge with other areas of expertise, in particular medical sciences and social policies.

The peculiarity of cultural welfare projects and activities is also the ability to intercept, at the same time and in a single context, different demands from the point of view of health promotion, involving, for example, physical work as a mental exercise component, thus being more effective than the separate implementation of the same prescriptions (Fancourt and Finn, 2019).

The possible relationships between body movement and artistic components have been investigated at length over time, increasingly framing movement as a non-verbal language of universal character. Leading artistic and choreographic projects emerged in the last decades aiming at stimulating an active participation of audience members, characterized by a collective, physical, cognitive, and emotional experience of artistic content (e.g., visual arts, performing arts): examples include the artistic projects of Bill Viola, of Peter Greenaway, of the Batsheva Dance Company, and of Virgilio Sieni's “Academy of the Art of the Gesture”.

In these approaches, cultural heritage (a museum or a historical-artistic site) takes on the function of a framework for an activity supporting health treatments: for example, Dance Well—movement research for Parkinson's (Cicerchia and Seia, 2020) and Musei Toscani per l'Alzheimer (De Luca, 2017; Cicerchia and Seia, 2020) promote health not by substituting museums to medical environments, but by exploiting their strengths in supporting memory stimulation and emotional experience. In other cases, cultural content is exploited in healthcare settings—such as hospital wards, outpatient clinics and nursing homes—to support treatment and mitigate or slow down certain pathological conditions, with a particular focus on degenerative ones (Cicerchia and Seia, 2020).

Emerging technologies are pushing a disruptive change of paradigm in the design of interactive digital tools and services, involving the users to engage in full-body, physical, emotional experiences. Advanced multimedia and interactive technology can contribute to the emerging innovative paradigms of cultural welfare, to enhance the active experience of users in confronting artistic and cultural content. Recent interaction design approaches propose design techniques to develop successful interactive systems characterized by a smooth, intimate correspondence between users' individual, as well as joint, actions and system response. Present multidisciplinary approaches include Slow Computing and Calm Technology (Hallnäs and Redström, 2001), focusing on the interaction in everyday exposure to technology, and Somaesthetic Interaction Design (Höök, 2018; Loke and Schiphorst, 2018), considering the embodied first-person engagement in the interaction design process.

In 2003 (Camurri et al., 2003) an approach and an interactive system was proposed in hospital to support physical exercises for Parkinson patients, based on the paradigm of aesthetic resonance: the full-body movement of the patient causes the emergence and real-time manipulation of interactive soundscapes and visual feedback: the system measures the fluidity in the full-body movement of the patient, and a high degree of fluidity enables the patient to “paint” using his/her own body as a “brush” on the surrounding walls, and at the same time to create and develop naturalistic soundscapes (e.g., the forest and the sea): this only when moving fluidly, overcoming the typical hesitating movement in Parkinson. The body silhouette was like a brush to paint on the surrounding walls in a dedicated room in the hospital. Usually, frustrating physical rehabilitation exercises are here transformed into a sort of artistic installation, enabling to interactively create aesthetically resonant visual and auditory feedback. A similar approach was adopted in the exergames designed for supporting the cognitive and motor rehabilitation of children in the ARIEL-Augmented Rehabilitation Joint Lab of Casa Paganini-University of Genoa with the G. Gaslini children hospital.¹ In the European Horizon 2020 ICT DANCE project,² Casa Paganini collaborated with the choreographer Virgilio Sieni in a series of public events³ involving over 120 (non-dancer) citizens participating to the (re)discovery of their own full-body expressive qualities in a series of public collective performances (Niewiadomski et al., 2019). In DANCE, the real-time measures and sonification of fragility and lightness expressive qualities of full-body movement of one of the participants (the leader) had the goal to induce the other participants to entrain and express the same qualities in their full-body joint movements. In Singh et al. (2016), interactive sonification of human movement qualities was proposed in rehabilitation from chronic pain disease: the interactive auditory stimuli have the role to enhance fluid and safe movements, and to “distract” the patient from the attention to chronic disease problems. In the EU Horizon 2020 ICT WHOLODANCE project,⁴ Casa Paganini developed interactive sonification of human movement qualities, grounded on cross-modal correspondences (Spence, 2011) to support the learning process of full-body movement and actions in dance and in other scenarios.

The above examples are grounded on our approach for the modeling of full-body multimodal gestural expressiveness in individuals as well as in small groups and in cross-modal correspondence of movement qualities to visual and sonic stimuli (Camurri et al., 2003, 2005, 2016; Volpe and Camurri, 2011), resulting in our Slow Mood and Aesthetic Resonance paradigm (Cera et al., 2022). Originally developed for the interaction with the environment that stimulates active listening, our multilayer conceptual framework, together with multi-temporal scales and representational structures for affect processing, expressiveness, inter-personal synchronization and entrainment, our approach is structured in four layers, ranging from physical signals to high-level qualities of full-body individual as well as small group movement.

2 Materials and methods

2.1 The DanzArTe system and protocol

DanzArTe is the result of about 2-year user-centered iterative interaction design and software engineering development process. It aims to support an active full-body experience with a traditional visual art painting, as a collective, autonomous aesthetically resonant amateur practice, supporting memory and physical treatment. We focus on two main categories of users: older people living in nursing homes with residual cognitive and physical capacities sufficient to take part in the program, and community-dwelling healthy older people (over 65 years old) at risk of frailty. DanzArTe is a treatment protocol and an interactive system to support the active experience and real-time processing of visual arts content (classical religious paintings, image manipulation and 3D modeling) and interactive sonification, to support full-body physical activity and cognitive exercise of memory training.

The DanzArTe project aims to create a replicable treatment protocol for multidimensional frailty (Pilotto et al., 2020) grounded on an innovative low cost and scalable technological platform. The treatment protocol consists of four different sessions. Each session is attended by groups of 4/5 participants.

Visual art content functions as a guide to execute full-body physical activity and is a reference for memory training. Interactive sonification provides cues to nudge (Thaler and Sunstein, 2021) participants' movement toward the desired movement qualities: fluidity of movement and entrainment/synchronization among users. Small groups of 4/5 participants in turn interact (individually or in dyads) with the visual content, which is displayed by a video projection visible only to one or two of the group: the other participants give their back to the video projection and imitate the active participants in their affective interaction with the visual stimuli.

DanzArTe interactive system integrating real-time manipulation of visual art and interactive sonification content takes the role of a guide and stimulus for the users, as a sort of surrogate of artistic or clinical personnel. A supervisor of the system plays only the role of mediator and facilitator of the emotional experience among the participants.

All participants are engaged in a form of slow and safe movement sequence, inspired by a prolonged, gradual observation and discovery of a traditional religious painting and related interactive sonification of the movement qualities of fluidity (Niewiadomski et al., 2019) and entrainment (Varni et al., 2010) in joint actions.

One of the most important goals of the project is to create a self-sufficient and automatic experience that, as soon as the system that forms its physical support is launched, does not require any significant intervention other than that of the participant. At the same time, an effort has been made to limit verbal content as much as possible—such as text notes indicating which movements to perform or the corresponding body parts—privileging affordance, intuitiveness, and naturalness of interaction, along with immersion and fluidity. The intent is to counter fragility, characterized by broken and hesitant movements, by “nudging” toward an aesthetically resonant experience characterized by slow and continuous movement.

2.2 The DanzArTe experience

The DanzArTe experience requires a room equipped with a video projector, a screen (or white wall) of at least 2 × 3 m, a computer, movement sensor(s), and a stereo sound system. A mediator/facilitator (e.g., personnel of a residency for older people, or personnel from the museum or cultural institution) supervises the experience and facilitates the social interaction among the participants. The overall DanzArTe protocol is composed by a preliminary introductory session, followed by four sessions (one or two sessions per week). Each session has a duration of about 1 h. A preparatory session (Session 0) is performed the same day of Session 1. Sessions 1 and 2 are designed with one active participant, while Sessions 3 and 4 have two active participants interacting with each other to perform joint actions together: the focus here is on synchronization and entrainment of the affective gesture of the dyad. Each group is formed by 4/5 participants, who exchange their role, to alternate roles of leading (active) and following participants. Active participants are in front of the video projection, while the others observe, imitate the active participants, and may interact with suggestions. For example, Session 1 starts with the video projection of the painting by Luca Cambiaso “The Mother with the Child”. The active user observes the painting for a few seconds, then the painting tends to disappear, except for the body of the Mother, which then loses the color, and fades out, replaced by a virtual agent in the same posture. The virtual agent is a very essential and geometric 3D model derived by the preparatory drawings of the painter Luca Cambiaso. Now the painting has completely disappeared, and the agent starts to move from the posture of the Mother sitting with the Child on her lap, and slowly repeats a first component of the movement evoking the affective gesture of the Mother: the movement of the left arm to embrace the Child on her lap, then the other arm, the head inclination, one leg a little forward. If the participant imitates the agent correctly, the agent disappears and the body part of the painting's subject, evoked by the movement, starts to appear and gradually gets the original color from repetitions of the correct movement. An incorrect quality of the movement (e.g., not fluid) causes the agent to return and show it again. The agent movements are based on motion capture recordings of performances by two dancers, who performed sequences generated from the postures depicted in the paintings, following guidelines from experts in science of movement and geriatrics. After the participant has completed each component of the movement, the agent shows the whole affective gesture, in a sort of final dance with all the parts of the body engaged. The participants start this crescendo of movement repetitions and this gives life back to the original painting, that receives from the participants the “energy” in terms of light of small candles that at the end illuminate the whole painting.

After the first sessions in collaboration with the Museo Diocesano di Genova, feedback from participants were on the lines of “I could see the painting with new eyes”, or “now when I look at a painting, I see more on it”.

The participants are initially instructed (Session 0) to imitate the movements of the agent and learn that their movement is causing sounds. So they understand that if they start moving correctly, the agent slowly disappears, and a fragment of the painting emerges, as if it were lit by a candle, corresponding to the part of the body moved by the participant. This part of the painting regains its color by repeating the movement. The participants know that the picture will be “reconquered” according to the way they will be moving, imitating the affective gesture in the painting. It is important to notice that the system does not measure the posture exactly, but it mainly focuses on the following movement features: (i) the correct part of the body; (ii) Fluidity and Quantity of Movement of a part of the body; and (iii) level of synchronization among two participants. The interactive storyboard is controlled by the DanzArTe EyesWeb component: each session corresponds to a timed script depending on time and on the behavior of the users, thus determining the evolution of the interaction by comparing the participant's movements with the reference model of movement required at each moment of the storyboard.

2.3 Embodied interaction with visual art paintings

The paintings were selected according to several criteria, including the evoked affective gesture of the main characters depicted in the painting, and the type of implied movement of such characters, taking also into account movement science and geriatrics perspectives. We focused on traditional religious paintings, since they are familiar to the selected users' population, and avoid the risk to induce possible psychological issues in users. Paintings were chosen also in relation to the affective and dynamic movement they evoke, avoiding paintings characterized by static or difficult postures which may cause physical injury. The presence of the painting in the local museums and churches in the territory was also considered, to encourage participants to visit cultural institutions or to bring the cultural content of such institutions to the residencies of older people where the system is adopted. We individuated a series of images that can induce familiarity and confidence, and at the same time present movements suitable to be reproduced by a person with frailty without causing health hazards or embarrassment. For these reasons, the preference turned to sacred iconography, easily readable as part of a cultural substratum which the target audience, for generational reasons, identifies with. The works were also screened taking into consideration the opportunity to generate a network of relations between the paintings present in the experience and, by affinity, those exhibited in the museum institutes of the Ligurian and Piedmontese territory, in order to favor the dissemination and replicability of the project. The decision to focus on the work of Luca Cambiaso naturally stemmed from the need to have subjects with great gestural expressiveness and scenes containing multiple characters, features that constituted research material for the Genoese painter throughout his artistic career, as evidenced by the famous sketches through which he studied poses and compositions. The movements, once identified, were verified and approved by the geriatric experts in our team. The selected movements for each session are the following:

• Session 1.1, “La Sacra Famiglia, San Giovannino e Angeli (Riposo durante la fuga in Egitto)” (Figure 2):

1. Forward extension of the left leg.

2. Folding of the left arm toward the torso, slight arching of the back.

3. Positioning of the right arm to form a loop with a welcoming gesture, without exceeding the vertical axis of the head.

These are the constituent movement fragments of the affective gesture of a mother hugging her newborn son: the objective of the sessions is not to replicate the exact gestures, but to imitate and empathize with the affective gesture in the painting. The same criterion is followed also in the other sessions described below.

• Session 1.2, “Vergine e il Battista che intercedono presso Cristo in gloria tra i santi”:

1. Slight divarication of the lower limbs and subsequent translation of the right leg outward.

2. Lifting of the left arm until the elbow is at shoulder level, subsequent shifting back of the same arm with respect to the axis represented by the shoulders.

3. Opening the right arm outward to form a descending curve, twisting the head to the right about 90°.

• Session 2, “San Gioacchino e Sant'Anna”:

1. Left leg steps forward and goes to support the weight of the body. Slight forward arching of the back.

2. Left arm folded toward chest, right arm raised with elbow bent and right hand touching chin.

3. Step to the side with rotation of the trunk as a trigger for movements related to st. anne's.

4. Left leg step back, right leg pivoting.

5. Left arm folded at chest level, right arm slightly arched along hip.

• Session 3, “Baptism of Christ” (Figure 3):

• Participant A:

1. Forward step of right leg with weight shift, foot slightly tilted inward.

2. Joining hands slightly to the right in relation to the central axis of the body, head tilted to the left.

• Participant B:

1. Forward step of the left leg with weight shift.

2. Raising left arm above head, hand extended forward, right arm contracted along hip.

• Session 4, “Pentecoste”:

• Participant A:

1. Forward step of left leg with weight shift, left arm extended forward at shoulder height.

2. Seated.

3. Tilt of the torso and head to the right, joining hands on the lap.

• Participant B:

1. Step forward of right leg with weight shift, right arm extended forward at pelvis height.

2. Seated.

3. Arching of the back and head, joining hands on the lap.

• Participant A and Participant B:

1. 1.1. Left arm folded on chest, right arm on lap.

Movement qualities automatically measured in sessions 1 and 2 include fluidity (Camurri et al., 2016; Piana et al., 2016), measured on the involved body parts. Sessions 3 and 4 add the automated measure of the inter-personal synchronization (Varni et al., 2010) of full-body movements of the two participants: in session 3 the synchronization of the body sway and of the hands; in session 4, the synchronization of trunk, hands, heads, and of the joint stand-up and sit-down on a chair.

2.4 The DanzArTe system architecture

The DanzArTe system is a scalable, low-cost, technological platform to support real time interaction with visual content (visual art paintings) and interactive sonification (see Figure 1).

Figure 1

Figure 1. The DanzArTe–Emotional Wellbeing Technology system architecture.

The system architecture is the result of a long (2 years) user-centered iterative interaction design process. The main software modules, described in Figure 1, are the following:

- Expressive Movement Analysis: this module extracts in real-time the full-body expressive movement qualities from a video signal obtained from a Kinect 2 depth camera sensor: following previous research results (Camurri et al., 2016; Niewiadomski et al., 2019) we included low-level movement qualities such as Quantity of Movement, Body Sway, Smoothness of arm movement, as well as mid-level features, including fluidity (Piana et al., 2016) and inter-personal synchronization (Varni et al., 2010). These features are measured in real-time by the Expressive Movement Analysis software module, developed and implemented in our EyesWeb software platform available online.⁵

- Interactive Storyboard and Sync Manager: this Python software module implements the interactive narrative structure resulting from the interaction design development process, structured in two phases: learning of the affective movement (Phase 1) followed by the final “dance” of the whole learned movements (Phase 2) and conclusion (see Section 2.5). This module receives the movement data from the EyesWeb Expressive Movement Analysis module and coordinates the response of the system in terms of real-time interactive visual and sonification outputs.

- Interactive Sonification Generation and Processing: This software module (developed in Max-MSP) receives movement data and information on the storyboard and generates the interactive sonification aiming to induce the participants toward the desired movement quality and experience (Cera et al., 2022). The design of the interactive sonification in DanzArTe follows the following requirements: (i) to facilitate a state of serene concentration; (ii) to enhance the perception of fluidity and slowness in the performed movements (and of the synchronization of movements in sessions involving two participants); and (iii) to avoid unnecessary startling, intrusive, or annoying effects that would destroy the flow and the liminality of the experience. The sound environment is composed by a “background” (ambient soundscape based on processed recordings of natural environments), and a “foreground” (real-time sonification of the movements qualities of smoothness/fluidity and synchronization, created with granulators based on pre-produced soundfiles whose timbral content is characterized by fluidity and continuity, to create a smooth, ceaseless sound).

- Video Content Processing: This module integrates two main modules, corresponding to the management of the blending of the mannequin overlapped and emerging from the visual art painting, and a module which manages the emergence of the painting as it were lit by a growing number of virtual candles that gradually enable the active participants to experience the painting.

All these modules are coordinated by the Interactive Storyboard and Sync Manager, which includes the real-time management of the multiple temporal scales that form the whole experience, from immediate responses to slower adaptations and changes of states due to the evolution of the behavior of the active participants. All system modules run on a Windows 11 personal computer (Intel i7 CPU), connected to a Kinect 2 depth camera, an external audio board, two Genelec loudspeakers placed laterally to the active users, and a video projector (see Figures 2, 3).

Figure 2

Figure 2. Structure of the interaction in session 1, involving one active participant: the system reacts to the correct imitation of the right arm's movement of the mannequin (A); The correct movement causes the gradual fade of the virtual agent and emergence of the painting's subject, displaying the same pose (B). If the active participant runs repeatedly the correct movement, the correspondent body part in the painting is highlighted by an emerging candle (C).

Figure 3

Figure 3. Excerpt of the interaction with two active participants simulating the “Batism of Christ by St. John” by Luca Cambiaso (Session 3 of DanzArTe): the two participants play the roles of Christ and of St. John, respectively. The system reacts to the correct imitation and synchronization of the movements displayed by the virtual agents (A), causing the emergence of painting characters (B), and the body parts involved in the movements are highlighted one after the other, until the whole painting unveils itself completely thanks to the activation of all the candles (C). See https://entimement.dibris.unige.it/events/46-danzarte for a sample video excerpt of this session.

The expressive movement qualities are analyzed in real-time by EyesWeb. The visualization of the movement of the mannequin is based on pre-recorded database of short movement fragments obtained starting Qualisys motion capture system recording sessions (see Figure 1), while the processing the visual art painting static image is based on a real-time software module that controls the colorization and the dynamic appearance/disapperance of “virtual candles” simulating the illumination of parts of the painting (see Figures 2, 3). Movement features are mapped in real-time on the parameters of the granulators according to the following steps:

- 1) The EyesWeb movement analysis module verifies if the movement quality is achieved by the correct part of body (fluidity/smoothness) of user(s), and the synchronization of hands and head movements in the case of a dyad. This determines the real-time sonification, that induces (“nudges”) the user(s) to continue with their correct movement quality. In case of correct movement quality, the measured movement qualities are sent as inputs to the sonification engines module and to the visualization module. In case of correct movement quality, the mannequin disappears and a crossfade with the portion of the visual art painting showing the part of the body involved, as it were illuminated by a virtual candle, appears and slowly increase to full color;

- 2) The sonification engine maps the kinetic energy on the output volume, the smoothness (and the degree of synchronization in case of a dyadic joint movement) controls the cutoff frequency of a parallel low-pass filter and other spectral features. Both parameters are treated in order to smooth out any undesired fast peaks or valleys, and to create slow shapes at temporal scales of 0.5 s and 3–5 s.

In a successful session, the “foreground” part of the sonification, initially rare and sparse, becomes gradually continuous, until it almost blends with the “background”. This effect of fusion of “foreground” and “background” signals the correct expressive body movement. The “background” soundscape is not static and continuous. In order to stimulate user attention, it appears/disappears following extremely slow dynamic curves: it fades in a slower temporal scale ranging between 5 and 20 s. In the case of detection of error in an important sequence, these sounds disappear/appear with a faster fade out/in curve (1,000 ms). These subtle modifications in the background have the effect of semi-consciously enhancing the attention and vigilance of the user(s) without being intrusive, as a sort of salience effect by subtraction.

More details on the system platform and the whole DanzArTe protocol are described in a paper in preparation.

2.5 The virtual tutor: Cambiaso's mannequin

The main medium of Cambiaso's spatial and graphic experimentation are his sketches, found in numerous examples and even duplicates. The analysis of the peculiar characteristics of these sketches, combined with the recovery of previous studies (Magnani, 2015), guided the design and realization of the digital model that performs the function of a virtual tutor in DanzArte, executing those movements that are implicit in the works and represented in their most essential form.

In the preparatory drawings made by Cambiaso, the movement of the mannequin is represented in a static form, a moment crystallized in time that keeps all its gestural and affective content within the bounds of the graphic sign. In contrast, the virtual mannequin created for Danzarte explores all the facets of movement, breaking it down and repeating it, in the course of a complex and fluid animation, in order to represent a guide for user interaction. Given these important differences, the study and development phase was largely dedicated to the most effective transformation of the static representation of movement into a dynamic one.

One of the main problems was to establish unambiguous proportions for the body of the mannequin, so that on the one hand they did not betray Cambiaso's original sketches and paintings, and on the other hand they did not deviate too much from anatomical correctness and harmonious appearance. Initially, anthropometric tables were used, and reference was made the original drawings and notes by Luca Cambiaso. At a later stage, however, an inconsistency emerged between the proportions of the mannequin and the movements of the animation, causing an impression of imbalance and disharmony in the gestures. This led to the decision to base the virtual model on the measurements of the bodies of the choreographers who performed the recorded movements, thus allowing for maximum adherence between the Motion Capture recording and the animation reported by the virtual model, while remaining consistent with Cambiaso's work.

The movements of the mannequin have been recorded with our motion capture system Qualisys (16 cameras full-body marker-based recordings) with the help of two dancers. The movement fragments were further elaborated and refined in order to optimize the communication of the movement qualities to the users. This resulted in a flexible and parametrized movement dataset (see Figure 1). Details on this module and dataset are available in a paper in preparation.

2.6 The pilot study

The experimental phase of the DanzArTe project aimed to test the feasibility of the DanzArte program in two separate settings: nursing home and community. To this end, an exploratory study was carried out involving subjects residing in the Galliera “Residenza Cardinal Minoretti” nursing home in Genoa (nursing home group) and subjects residing at home (community-dwelling group) and recruited at Museo Diocesano di Genova. The protocol scheduled, both in the nursing home and at Museo Diocesano, 10 cases and 10 controls: however, a larger number of participants was recruited in order to prevent possible “drop-out” cases. Forty-five participants were recruited (21 at the nursing home and 24 community-dwelling) randomly divided between treatment and control as follows:

• Nursing home Galliera “Residenza Cardinal Minoretti”: 11 treatments and 10 controls

• Museo Diocesano di Genova: 12 treatments and 12 controls

The program involved 4 sessions (2 per week for a total duration of 2 weeks) lasting ~90 min. Small groups of 4 or 5 persons participated in each session. Upon inclusion in the study, all subjects were assessed using the following rating scales:

• Multidimensional Prognostic Index (MPI)—Multidimensional Frailty Scale including:

1. Basic activities of daily living (ADL)

2. Instrumental activities of daily living (IADL)

3. Nutritional status through mini-nutritional assessment short form (MNA-SF)

4. Mobility status (using exton smith scale—ESS- or barthel mobility index—MOB)

5. Cognitive state (short portable mental status questionnaire—SPMSQ- or test your memory—TYM)

6. Cohabitation status

7. Comorbidity: as measured by the comorbidity index of the cumulative illness rating scale (CIRS)

8. Number of medications

• Resilience Scale (RES-14 items)

• Client Satisfaction Questionnaire (CSQ-8)

• Subjective psychological wellbeing evaluating 6 different Health-Related Quality of Life domains through the Psychological General Well-Being Index—Short (PGWBI-S)

• Short Physical Performance Battery (SPPB)

• Hand grip (HG)

Participants who did not undertake the DanzArTe program (controls) performed all scales except for the CSQ-8, as this assessed satisfaction with the program. Furthermore, only participants from nursing homes underwent the physical performance assessment with SPPB and HG study.

2.7 Statistical plan

For the baseline assessment, socio-demographic and clinical characteristics if continuous were reported using means (M) and Standard deviations (SD) for the total sample and the two sub-groups (treatment vs. control). Categorial variables together with the MPI's domains were summarized as frequencies and percentages. The differences between the two groups in terms of gender and age were analyzed using, respectively, the chi-square test and the t-test. All subsequent analyses will be non-parametric because, according to the Shapiro-Wilk test, all clinical variables—aside from the PGWBI-S, CSQ-8, and age—were not normally distributed (all p < 0.019). The comparison at the baseline was carried out using t-tests or the correspondent non-parametrical Mann-Whitney test and χ²-tests for frequencies of the risk categories. Wilcoxon's signed-ranks tests were used to compare pre- and post-intervention results. The effect size was computed as Cohen's d. Finally, a p-value of 0.05 or less was considered statistically significant for all two-tailed statistical tests. All statistical analyses were performed using SPSS (version 21.0) for Mac and Jamovi.

3 Results

3.1 Study population

Forty-five participants were involved in this case-control pilot study. The sample was composed of 23 elderly people who participated in the DanzArTe program (Mean age = 78.3 ± 8.2 years) and 22 elderly people who did not (Mean age = 77.3 ± 7.8 years). The average age of the participants was 77.8 (±7.96 years; range, 58–102) and 60% of them were females (27 individuals, 15 in treatment group and 12 in control one). The two subgroups do not differ in terms of gender (p = 0.465) or age (p = 0.683).

3.2 Baseline characteristics

Table 1 shows the baseline data of the entire sample, the treatment group, and the control group. All the recruited participants completed the baseline assessment.

Table 1

Table 1. Means, standard deviations, frequencies for the total sample and the two sub-groups for each dimension and for the MPI domains.

As can be seen at the baseline, the two groups were substantially comparable except for the differences in CIRS-CI (p < 0.001), the cognitive domain (p = 0.015), and SPPB (p = 0.038). Thus, the treatment group reported less comorbid diseases, better physical performance and worst cognitive status than the control group.

3.3 Post-intervention results

One participant didn't complete the post-intervention examination thus the treatment sample at T2 consisted of 22 subjects. A statistically significant improvement in health-related quality of life was observed between the baseline assessment and the post-intervention assessments (t = −2.344; p = 0.029, Cohen's d = −0.500). The control group didn't show the same trend for the PGWBI-S (p = 0.389), thus the DanzArTe program could have improved this dimension (Table 2).

Table 2

Table 2. Results of the comparison between the pre-intervention measurement (T1) and the post-intervention one (T2) for the treatment group (n = 22) for all the scales.

Gender differences in the post-intervention data were found only in the CIRS-CI (p = 0.04) with females reporting an average score of 1.429 ± 0.9 vs. males 2.286 ± 0.8.

3.4 User satisfaction

After the completion of the DanzArTe program, all the participants completed the CSQ-8 for the satisfaction about the activity. As a whole, the sample showed a mean of 25.5 ± 3.3 (range population from 20 to 31, scale range 8–32). Specifically, nursing home residents reported a mean of 23.3 ± 1.5 whilst the community-dwelling group had a mean of 27.6 ± 3.3, highlighting a high appreciation for the program.

4 Discussion

The study conducted on the DanzArTe protocol and system showed a significant improvement in the subjective psychological wellbeing of participants when comparing the baseline assessment and the post-intervention assessments. The control group, on the other hand, did not exhibit a comparable trend in their self-perceived psychological health and wellbeing.

The community-dwelling group's high satisfaction with the DanzArTe program added weight to the positive effects observed. This may indicate that the program was appreciated and had a beneficial impact on participants' general sense of wellbeing.

From a qualitative standpoint, several observations were gathered from the program's personnel, offering some insight into the elements that might have influenced the successful outcomes. Groups from residential care facilities, which are used to an established schedule, were said to particularly like taking part in the DanzArTe project. They looked forward to and enjoyed the sessions. Participants acknowledged the performers' inspiring and entertaining qualities as well as those of the group performing.

From the remarks made by participants, some suggestions for the future were drawn. Increased exposure to the artworks and a more complete description of the psycho-physical objectives of the DanzArTe program were advised. These recommendations show a deep interest and are a stimulus for future research of this project.

Overall, the research shows that participants' subjective psychological wellbeing was improved by the DanzArTe program, positively judged as a creative, inspiring, and engaging experience by the participants. Results suggest that the program improves self-perceived psychological health and wellbeing among people taking part in DanzArTe sessions.

This study, however, has certain limitations. The experiment specifically took place in the post-pandemic period, but there were still COVID-19 epidemic flare-ups that slowed down the trial duration, particularly in nursing homes. Since the current validation project is a pilot study, the experiment also included a relatively small sample of beneficiaries, which may have impeded the achievement of more statistically significant results. In order to consolidate the outcomes obtained and presented here and maybe extend them to other domains from a multimodal and componential intervention perspective, it may be helpful to increase the program's meeting frequency and overall duration. Alongside the pilot study, we tested the protocol and the system with over 200 healthy users, as a means to increase awareness in the comprehension of artworks: for example, to teach museum visitors to see an artwork “through the body”, or “with different eyes”. The involvement of subjects unaffected by frailty showed DanzArTe's suitability for serving as an introductory approach to art and culture, as underlined by the participants' comments.

Another noteworthy aspect is the success in promoting inter-generational interaction and community building, as reflected by the many participants who reported a special enjoyment in reinforcing, through the shared interactive experience, their relationships with younger family members and peers.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by University of Genoa Ethics Committees (Protocol n.2022/11). The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study. Written informed consent was not obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article because the persons depicted signed informed consent for participation in the study, were photographed from behind, and were wearing a mask, thus not being recognizable in any way.

Author contributions

ACa: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Software, Supervision, Writing – original draft. ES: Data curation, Investigation, Methodology, Software, Writing – original draft. WM: Data curation, Formal analysis, Methodology, Writing – original draft. CCa: Conceptualization, Investigation, Methodology, Software, Writing – review & editing. NF: Conceptualization, Methodology, Writing – review & editing. SG: Data curation, Investigation, Methodology, Software, Writing – review & editing. ACe: Conceptualization, Methodology, Software, Writing – review & editing. PC: Methodology, Software, Writing – review & editing. MB: Data curation, Investigation, Writing – review & editing. GP: Data curation, Formal Analysis, Writing – review & editing. IN: Investigation, Supervision, Writing – review & editing. CCo: Investigation, Supervision, Writing – review & editing. BS: Investigation, Supervision, Writing – review & editing. AP: Conceptualization, Methodology, Project administration, Supervision, Writing – review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This research has received funding from the European Union's Horizon 2020 Research and Innovation Program under Grant Agreement No. 824160 (FET PROACTIVE EnTimeMent Project, entimement.dibris.unige.it).

Acknowledgments

The partners of the DanzArTe-Emotional Wellbeing Technology project are the following: (i) University of Genoa, Casa Paganini—InfoMus Research Center (Coordinator) who originated and designed the conceptual framework and the interactive system, the design of expressive motion analysis, real-time sonification, and the choice of the artworks and visual content; (ii) E. O. Ospedali Galliera, Department of Geriatric Care, Orthogeriatrics and Rehabilitation; dealt with the definition of treatment protocols, clinical evaluation during the interaction design phases and supervision within the nursing homes where the experience took place; (iii) Lavanderia a vapore, Casa Europea della Danza, Choreographic Residency Center: contributed to the motion capture recording sessions, and to the experimental and dissemination sessions and public events in the Piemonte area; (iv) Museo Diocesano in Genova, contributed to the choice and digitalization of the chosen artworks, and hosted experimentation in their museum spaces, to contribute to the validation of the DanzArTe project. We thank Vittorio Gallese, Tutor of the DanzArTe project on behalf of the Compagnia di San Paolo, and Pier Luigi Sacco, President of the Scientific Committee of Compagnia di San Paolo, for their precious suggestions and discussions during the whole duration of the project. We thank Beatrice Sarosiek, project manager of Compagnia di San Paolo, Sandra Aloia, program manager in the Cultural Innovation Department, Compagnia di San Paolo, ASVAPP, and all the team at Compagnia di San Paolo for the support. We thank Babette Dijk, geriatrician at ASL4 Chiavari, for her contribute from a clinical perspective; Francesca Cola, Eugenia Coscarella, Mara Forcelli, Debora Giordi, Mara Loro, and Carlotta Pedrazzoli from Lavanderia a Vapore; Paola Martini and the volunteer personnel from Museo Diocesano; Maria Tolaini for her support in the initial part of the project. The DanzArTe project is promoted and supported by the foundation Compagnia di San Paolo as part of Well Impact (https://www.compagniadisanpaolo.it/it/csp-play/well-impact/).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^Ariel Augmented joint Lab Università di Genova (2014), ariel.unige.it.

2. ^The DANCE Project (EU ICT H2020 Project). (2020), dance.dibris.unige.it.

3. ^L'atlante del gesto_Genova. (20 January – 26 March 2017), https://it-it.facebook.com/atlantedelgestoGenova/.

4. ^WhoLoDance (Whole-Body Interaction Learning For Dance Education) (2016), wholodance.eu.

5. ^Casa Paganini-InfoMus https://casapaganini.unige.it/eyesweb_bp [Accessed July 20, 2023].

References

Camurri, A., Mazzarino, B., Volpe, G., Morasso, P., Priano, F., and Re, C. (2003). Application of multimedia techniques in the physical rehabilitation of Parkinson's patients. J. Visualiz. Comp. Anim. 14, 269–278. doi: 10.1002/vis.324

Crossref Full Text | Google Scholar

Camurri, A., Volpe, G., De Poli, G., and Leman, M. (2005). Communicating expressiveness and affect in multimodal interactive systems. IEEE Multimedia 12, 43–53. doi: 10.1109/MMUL.2005.2

Crossref Full Text | Google Scholar

Camurri, A., Volpe, G., Piana, S., Mancini, M., Niewiadomski, R., Ferrari, N., et al. (2016). “The dancer in the eye: towards a multi-layered computational framework of qualities in movement,” in Proceedings of 3rd International Symposium on Movement and Computing MOCO (Thessaloniki: ACM).

Google Scholar

Cera, A., Canepa, C., Ferrari, N., Pilotto, A., Coletta, P., Ghisio, S., et al. (2022). “Interactive sonification of expressive gesture: the DanzArTe - emotion wellbeing technology project,” in Proc XXIII Colloquium on Musical Informatics, Ancona, 25–28.

Google Scholar

Cicerchia, A., and Seia, C. (2020). Fondazione Symbola—Unioncamere, Io sono Cultura—Rapporto 2020, I Quaderni di Symbola. Roma.

Google Scholar

De Luca, M. (2017). La “cura dell'arte”: arti visive e malattia. Econ. Della Cult. 2, 305–312. doi: 10.1446/87285

Crossref Full Text | Google Scholar

Fancourt, D., and Finn, S. (2019). What is the Evidence on the Role of the Arts in Improving Health and Well-being? A Scoping Review. Copenhagen: WHO Regional Office for Europe; (Health Evidence Network (HEN) synthesis report 67).

Google Scholar

Hallnäs, L., and Redström, J. (2001). Slow technology – designing for reflection. Pers. Ubiquit. Comput. 5, 201–212. doi: 10.1007/PL00000019

Crossref Full Text | Google Scholar

Höök, K. (2018). Designing With the Body: Somaesthetic Interaction Design. Cambridge: MIT Press; Massachusetts Institute of Technology. doi: 10.7551/mitpress/11481.001.0001

Crossref Full Text | Google Scholar

Loke, L., and Schiphorst, T. (2018). The somatic turn in human-computer interaction. Interactions. 25, 54–5863. doi: 10.1145/3236675

Crossref Full Text | Google Scholar

Magnani, L. (2015). Luca Cambiaso: dalla ricerca storica al virtuale. Genova: De Ferrari.

Google Scholar

Niewiadomski, R., Mancini, M., Cera, A., Piana, S., Canepa, C., and Camurri, A. (2019). Does embodied training improve the recognition of mid-level expressive movement qualities sonification? J. Multim. User Interf. 13, 191–203. doi: 10.1007/s12193-018-0284-0

Crossref Full Text | Google Scholar

Piana, S., Alborno, P., Niewiadomski, R., Mancini, M., Volpe, G., and Camurri, A. (2016). “Movement fluidity analysis based on performance and perception,” in Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 1629–1636.

Google Scholar

Pilotto, A., Custodero, C., Maggi, S., Polidori, M. C., Veronese, N., and Ferrucci, L. (2020). A multidimensional approach to frailty in older people. Ageing Res. Rev. 60, 101047. doi: 10.1016/j.arr.2020.101047

PubMed Abstract | Crossref Full Text | Google Scholar

Singh, A., Piana, S., Pollarolo, D., Volpe, G., Varni, G., Tajadura-Jiménez, A., et al. (2016). Go-with-the-flow: tracking, analysis and sonification of movement and breathing to build confidence in activity despite chronic pain. Hum. Comp. Interact. 31, 335–383. doi: 10.1080/07370024.2015.1085310

Crossref Full Text | Google Scholar

Spence, C. (2011). Crossmodal correspondences: a tutorial review. Atten. Percept. Psychophys. 73, 971–995. doi: 10.3758/s13414-010-0073-7

Crossref Full Text | Google Scholar

Thaler, R. H., and Sunstein, C. R. (2021). Nudge: the final edition (Final edition.). Penguin Books. Penguin Random House LLC.

Google Scholar

Varni, G., Volpe, G., and Camurri, A. (2010). A system for real-time multimodal analysis of nonverbal affective social interaction in user-centric media. IEEE Trans. Multim. 12, 576–590. doi: 10.1109/TMM.2010.2052592

Crossref Full Text | Google Scholar

Volpe, G., and Camurri, A. (2011). A system for embodied social active listening to sound and music content. ACM J. Comput. Cult. Herit. 4, 2–23.

Google Scholar