The I-Ork project: a technological approach to inclusive music making and therapy

The “Inclusive Orchestra” project (I-Ork) examines the issues concerning inclusive music making and therapy from a multidisciplinary perspective, involving music technology, human-computer interaction, customized music instrument design, music therapy practices, community music making, music education and musical creativity. The core objective is to foster the creation of performance groups composed of individuals with diverse abilities, including those with physical and/or cognitive impairments, as well as participants with no formal musical background. The research team iteratively designed several digital musical instruments and performance setups, tested during sessions involving individuals with various disabilities. A specific observational protocol based on the Individualized Music Therapy Assessment Profile (IMTAP) was developed and applied to validate the effectiveness of the study. This protocol allowed for a structured assessment of participants' behavior across functional, emotional, and musical domains. Preliminary results indicate that the digital instruments promote high levels of participation, elicit positive emotional responses, stimulate curiosity, and support exploratory behavior. The findings suggest that technology-enhanced musical tools can play a significant role in inclusive music practices by enhancing accessibility, engagement, and emotional expression. The IMTAP-based evaluations provided valuable insights into how different instruments align with users' needs, supporting further refinement of both the tools and intervention strategies.

1 Introduction

According to the World Health Organization “Disability is part of being human and it is integral to the human experience.” This vision enhances diversity as an element of enrichment of the human experience, contributing to understanding and participating in the perspectives of people with disabilities. Again, WHO affirms that disability “…results from the interaction between health conditions such as dementia, blindness, or spinal cord injury, as well as environmental and personal factors” (WHO, 2020). This implies that physical factors alone do not determine a person's disability status and places emphasis on the need to remove negative attitudes, and limited social support in the effort of building inclusive environments to foster task accomplishment and positive social interaction. The great number of disabled people (more than one-quarter of the EU population aged 16 years and over had a disability in 2022, according to Eurostat (2023)) has influenced the policies of the educational systems in various European countries, leading to a set of recommendations. In order to overcome current gaps and barriers, ANED, the Academic Network of European Disability, recommends that the current school system be reorganized based on a shared educational concept of disability (Ebersold et al., 2011).

Music playing, as one of the most universally accessible and inclusive human activities (Cross, 2009), represents the perfect field for applying such recommendations. Its engaging power applies to all ages and is known to provide benefits even in terms of non-musical skills (Costa-Giomi, 2004; Stensaeth, 2013). However, music is still not easily accessible for people with disabilities (Magee, 2006). In particular, traditional acoustic instruments, due to the complexity of their interfaces and playing techniques may be unsuited for persons with physical disabilities of different types and degrees, e.g., with partially able limbs or with quadriplegia. Experiences in this sense are based on the use of facilitated performance techniques, on the assistance of operators, and on the participation of musicians capable of supporting the performance environment with their musical contribution (Rubino, 2022). Digital musical instruments conversely have the potential for augmented accessibility as they allow for new, non-conventional modes of interaction. These instruments generate sound digitally, with the performer producing it through physical actions captured by sensing devices. Exponential increases in available computational resources, miniaturization, and sensors are enabling the development of digital music instruments that use non-conventional interaction paradigms and interfaces (Miranda and Wanderley, 2006). This scenario opens up new opportunities and challenges in the creation of accessible music instruments to include people with disabilities in music practice. Building on these foundations, the “I-Ork” project explores the theme of inclusion in music activities from a multidisciplinary perspective. It examines what factors contribute to an effective musical environment for the expressive use of digital instruments and for fostering the participation of individuals with disabilities.

This contribution starts with a short review of the use of music technologies in the various fields of music therapy practices. Section 2.3 presents and analyzes the experiences of the “Drake Music” and AUMI projects, which serve as significant and exemplary references for shaping the “I-Ork” project. These initiatives are particularly relevant because they integrate instrument design with their musical application and therapeutic dimensions–an approach central to the “I-Ork” project as well. In Section 3 the characteristic of “I-Ork” are outlined, with respect to its three main domains: the technological domain for instrument design, the musical domain for composition and sound design, and the therapeutic domain for instrument-centered approaches and community music therapy. In the first domain, authors introduce the characteristics of the digital instruments used in the activities with various clients, and describe their functioning and range of skills necessary for their use. In the second domain, authors examine musical compositions and conducting strategies designed for organizing performances by an all-abilities group of performers exclusively using digital musical instruments. The third domain focuses on the IMTAP protocol, employed as a tool to evaluate music therapy interventions involving these instruments. To assess their impact, authors conducted observation sessions with 10 participants of varying ages and disabilities.

Section 4 details the observation protocol used to gather data, as well as the methodologies applied to analyze the results. These include an exploration of the primary themes identified through qualitative analysis (Section 5.1), an evaluation of participants' experiences with ADMIs (Section 5.2), and a final assessment of the instruments themselves (Section 5.3). Conclusions and directions for future work are provided in Section 7.

2 Related work

Interventions in music therapy settings are defined as receptive when the client is asked to respond somehow to a musical stimulus, or active when some process of music making is involved (Wheeler, 2015, p. 8). In both domains, music technology plays an important role, as evidenced by the work of many researchers and practitioners. Musical listening, fundamental in receptive activities, was greatly facilitated first by the ease and portability of MP3 players, and more recently by the Internet and music sites (Whitehead-Pleaux and Spall, 2013). Simple response to a musical stimulus can be helped by the Disklavier¹, which can provide a number of repetitions of the musical excerpt, leaving the therapist free to offer further assistance to the client (Roth, 2014). Kubicek et al. (2011) offer a survey of the technologies employed by music therapists in various contexts and with a broad group of users ranging from 8 to 71 years of age. These include switches used for speech messages activated by hand pressure, chin or head,² or for music performance. In addition, computer-based applications such as Garageband³ can be used for creative experiences involving many sounds and musical styles, improving cognitive functions and communication skills (Street and Magee, 2014).

Alongside these more traditional uses of technology, music technology offers many other possibilities. Since 2004 (Ellis, 2004) has highlighted the benefits of his “Vibroacoustic Sound Therapy” (VAST) based on the use of sound technologies to elicit what he calls “aesthetic resonation,” a deep moment of connection between the sound vibration and the client's emotional response. To stimulate this feeling, he employs a microphone and a digital sound processor for changing a vocal input, which is transformed through reverberation, echo, and pitch shifting.

2.1 Digital music instruments in music therapy

The spread of low-cost sensors and the evolution of human-computer interaction led to the design of various digital music instruments (DMI). These instruments are characterized by a separate control interface and a sound production engine, with some important consequences for the instrument and its musical applications. In fact, the separation between control interface and sound production allows the design of instruments that exploit all the synthesis possibilities of the computer, combining them with a great variety of musical gestures (Miranda and Wanderley, 2006). And even if such a rich range of possibilities does not seem until now to have given rise to significant artistic productions (Jordà, 2004), DMI's have nevertheless proved to be very useful for both musical and therapeutic activities of people with disabilities.

Because of their characteristics that enable them to be customized to meet the needs of musicians with disabilities, more than 150 instruments have been classified as ADMIs (Accessible Digital Music Instruments) by Frid (2019). Based on their control interface, these instruments are separated into nine types, with tangible and touchless instruments being the most significant. Among these, there is the “Soundbeam”⁴, which employs ultrasound to generate music from physical movements. According to Ellis (2004) the “Soundbeam” helps to develop physical control, listening abilities and elicits exploration and self-expression. Many scholars and practitioners recognize the therapeutic value of ADMIs. Hunt et al. (2004) report benefits from the use of instruments such as “MidiGrid”⁵ and “MIDIcreator”⁶ because they allow an immediate control on sound output for people with movement limitations, and provide great richness and variety of sounds. Magee (2006), Magee and Burland (2008a), and Johnston et al. (2018) outline many advantages in the use of ADMIs, particularly for clients whose physical abilities limit access to acoustic instruments. These advantages include the ability to connect with the clients and motivate them to actively participate in therapy. Moreover, for the therapists they offer ways to improve their work also through cooperation with others, and of giving clients independence to expand their musical perspectives (Burland and Magee, 2014). Particularly Partesotti et al. (2018) highlight how the ease of use and variety of musical output of digital instruments promote expression, artistic freedom, and emotional engagement. They are also important for the development of proprioception and of creative empowerment, that is, the full control achieved by the client over technology that allows her/his full musical expression (Partesotti, 2016).

2.2 Drawbacks and limitations

However, there are some drawbacks to these positive aspects, such as the lack of technological training of the therapists, who seldom do not know how to access digital resources (Farrimond et al., 2011). Technology is also considered too invasive by some music therapists, so much so that it inhibits the relationship with the patient (Magee and Burland, 2008b). Finally, electronic instruments are also attributed with strong aesthetic limitations, which would make them less attractive than acoustic instruments (Magee, 2006). Partesotti et al. (2018) provide a thorough analysis of benefits and contraindications of the use of ADMIs in therapeutic practice. The main contraindications are the possible overstimulation of the user, distraction due to the presence of visual feedback, aesthetic difference and loss and physical experience compared to acoustic instruments. Farrimond et al. (2011) identify a series of barriers to engagement with technology for music therapists. Instead of igniting curiosity, the abundance of digital materials makes therapists feel incapable of making decisions and determining which tools are ideal for their requirements. Similar results are reported by Hahna et al. (2012) who, continuing the research started by Magee (2006), published a study about the use of technology by therapists in Australia, Canada, UK, and USA. Results indicate that 71% of the surveyed therapists have used technology in their practice, mainly with adolescents with developmental disabilities. The remainder of the sample declare not to use music technology mainly because they do not have access or don't know how to use it, or because it is too expensive. But the main problem seems the lack of training in the use of technologies, as the 61% of the sample report of being self-taught or having needed assistance from colleagues during the practice.

2.3 The “Drake Music” and “AUMI” projects

Many ADMIs have been developed by researchers with the aim of linking the design of the musical tools with specific needs of the users or particular contexts (Förster and Schnell, 2024; Ward, 2023; Davanzo and Avanzini, 2020; Förster et al., 2020; Stensaeth, 2013). This less generic and more user-centered approach certainly helps to address some technology adaptation issues highlighted above, improving accessibility and usability of music making tools. However, employing ADMIs or in general digital devices in therapeutic setting and in inclusive music making is an activity that requires a multidisciplinary approach and takes into account multiple points of view. For this reason, it is important to examine projects that address the problem from a holistic and operational point of view, trying to combine skills and practices that come from sectors that are also very distant from each other. For example, “Drake Music”⁷ is a British organization that employs accessible music technology to promote music making for a wide category of users, ranging across various ages and abilities. In 20 years of activity, the association has supported the creation of the “Accessible Musical Instrument Collection,” a series of modified or designed from scratch instruments made available for musicians with any type of disability. The richness of the collection enables the musicians to choose the instrument that best suits her/his needs, making the musical experience easier and more engaging. But the search for the highest level of adaptability to the needs of the disabled musician is not the sole component of the “Drake Music” project. As Samuels (2019) outlines, it is not only the accessibility of instruments that makes music performance possible and inclusive, but rather the level of meaning and emotion that the musical experience is able to arouse in the performers. The “Drake Music” project's activities are organized in workshops where the musicians are called to participate directly in the compositional experience. With the help of a music technology tutor, the musical project is discussed and structured after improvisation and critical listening sessions. At the end, a public event is organized for the social sharing of the experience. Thus, the “Drake Music” project not only provides bespoke technology for disabled musicians, but also considers the musical quality of the experience as a founding element of the therapeutic approach. Another meaningful experience is the “AUMI” project⁸, started in 2006 following the artistic and philosophical premises of the electronic music composer Pauline Oliveros's. According to Oliveros (1974), no special skills are necessary for a fulfilling musical experience. The most important thing is to build a musical environment where participants are equally valued and where they can gain confidence and mutual musical appreciation. Based on these ideas, the AUMI software interface (Adaptive Use Musical Instrument) has been developed in various stages over the years and currently includes Windows, MacOS and IOS versions.⁹ AUMI employs data from embedded or external cameras to track the movements of the head, the hand or the whole body. It is endowed with various user interfaces which allow moving a cursor on a grid to trigger the connected sounds. AUMI is a freely available ADMI, very easy and cheap to use. However, it requires a shift in the way of conceiving music and its performance. As witnessed by the research carried out by Lindetorp et al. (2023), music formalized through notes, rhythms, and written scores represents a strong obstacle to inclusive music making. Music formalized through notes, rhythms, and written scores represents a strong obstacle to inclusive music making. The sake of conformity and predictability linked to these musical styles prevents the practice of many other less preclusive ways of organizing music events which, thanks to greater freedom and creativity, can allow a much greater number of people to make music based on their feelings and possibilities (Ciufo et al., 2024, p. 52). Another important reflection concerns the concept of control, which is very important in musical performances. AUMI is not always precise and requires different strategies for musical realization. In AUMI improvisation, there are not “wrong” or “right” notes, but musical interventions adaptable to different circumstances and situations (Ciufo et al., 2024, p. 28). The very interesting idea here is that instead of adapting instruments to music making, with AUMI is music making that tries to find new ways of expression depending on how the instrument is used. This requires openness of mind, creativity, and full awareness of the potential that this technology can offer. On the website of the AUMI Editorial Collective book Ciufo et al. (2024) there are many examples of such approaches¹⁰. Finally, it is important to mention the work of Agres et al. (2021) who try to put the basis for a roadmap that takes into account the multidisciplinary aspects of inclusive music making and therapy with technologies. The framework is composed by three main fields: music therapy, music information retrieval and music psychology. Music information retrieval, through the use of computational analysis methods, provides useful information about structures such as melodies, rhythms, and harmonies preferred by the patients also in relation with their needs and therapeutic goals, while music psychology contributes in understanding the effects of musical interventions with a wide range of users, affected by the most varied pathologies ranging from newborns to the elderly. The study also takes into account other fields, such as music composition, performance, and human-computer interaction.

3 The I-Ork project

The I-Ork (Inclusive Orchestra) project also starts from the idea that it is necessary to address the problem of inclusive musical practices by combining operational attitudes and points of view from various disciplines. Particularly, I-Ork originates from the interaction of at least three domains:

• the technological domain, including human computer interaction for the design of ADMIs

• the musical domain, including creativity, composition, conduction techniques, and music technology knowledge for musical output

• the therapeutic domain, including music therapy approaches, community music making, and participation among all-abilities performers, possibly aimed at the realization of public musical events.

According to the social model of disability (Barnes, 2019), people with physical or more complex impairments experience isolation and fewer opportunities than other individuals not for their condition but for barriers and negative attitudes found in the society where they live. Translated into musical terms, this means that not only the instruments, but the entire system of musical expression must be questioned. As previously highlighted, traditional modes of expression require precise timing and high level of control of musical structures. On the other hand, the recent history of electronic music provides many alternative ways of expression, based on improvisation, the search for new sounds, emotional involvement and a relative freedom — if not randomness — in the succession of events (Butler, 2014). Authors believe that these elements are precious for the building of the “… welcoming musical environment.” (Ciufo et al., 2024, p. 28) so important in the eyes of the AUMI collective. Based on these premises, the I-Ork has the following objectives:

1. To provide disabled musicians with tools that allow them to making music by aligning with their possibilities and musical preferences;

2. To create friendly musical environments for collective music making where timing constraints and event sequences are freely programmable and adaptable to the needs of the musicians;

3. To create musical groups where everybody plays only ADMIs to avoid discrimination and frustration in choosing between traditional and accessible tools;

4. To expand the social experience of disabled performers by building mixed groups, made up of performers with different abilities, even without previous musical experience;

For the I-Ork project, authors used instruments that were constructed from both self-assembled and commercial parts. Authors experimented with ways of expression potentially attractive to the public and capable of arousing emotions in the performers employing only ADMIs and, finally, authors tested the tools in experimental sessions involving clients with various disabilities.

3.1 The technological domain: ADMIs in the I-Ork project

The instruments currently employed in the project are partially based on commercial hardware and software, and partially built employing cheap basic hardware components such as microcontrollers and MIDI modules. This is a compromise solution while waiting to be able to independently create all the instruments necessary for the musical practice of clients with various disabilities.

Following the classification of ADMIs proposed by Frid (2019), the instruments currently in use for the project can be subdivided into three categories: tangibles, touchless, and mouth-operated. All the instruments don't allow on-board sound synthesis and manipulation, and require the use of the computer for sound production. The sounds are produced employing the MIDI protocol¹¹ or data sent on serial monitor¹², and are synthesized using MAX¹³ and Ableton Live¹⁴. Numerical data coming from user's interaction are analyzed, processed and adapted to musical parameters to obtain sound output. Two main strategies are employed. The first is a one-to-one relationship, producing a single event from a single action. The second is a one-to-many relationship, which, from a single action, produces musical phrases or complex and rich musical events (Hunt and Kirk, 2000). All design decisions are aimed at creating devices that are easy to use, possibly cable-free, robust, accessible and adaptable, with the objective of ensuring a pleasant, engaging and above all aesthetically pleasing musical experience. Moreover, the instruments are designed also taking into account the motor skills, control, balance, and proprioception necessary for execution. In Table 1 the nine instruments currently in use in the project are listed, divided by category and with indications of the hardware and software used for their creation. Their design, accessibility, hardware, and software characteristics are described in the following Sections.

Table 1

Table 1. Table of the nine ADMIs employed in the I-Ork project, sorted by category with the indication of related hardware and software.

3.1.1 Tangibles

This category of instruments is built using commercial components belonging to the I-CUBEX system¹⁵. Tangible instruments require physical contact between user and tool, and therefore both gross and fine motor skills¹⁶ are involved in the interaction. The mapping strategy employed is one-to-many, so that simple gestures correspond to complex and structured events. By doing this, authors hope to support and encourage their use, particularly for clients who have motor impairments, who might thus gain from physical rehabilitation as well.

Push2D requires fine motor skills for arm and fingers control. As depicted in Figure 1 left side, the Push2D is a mechanical push button, connected to the I-CUBEX USB-micro-Dig¹⁷ via a rather long cable that allows for easy mobility and adaptability to different clients and contexts.

Figure 1

Figure 1. The hardware of the three tangible ADMIs of the I-Ork project by I-CUBEX. From left to right the Push2D, the TapMat and the TapTile.

TapMat (in the middle of Figure 1), uses a pressure sensor included inside a square of soft fabric. It does not require great fine mobility of the fingers, but only of the arms. However, it introduces the need for the musician to know how to effectively dose the pressure of her/his hands on a surface. The production of sound events is triggered by the physical and tactile exploration of the instrument, with the aim of developing and enhancing gross motor skills and the sense of touch.

Finally, TapTile (on the right of Figure 1) uses a weight sensor integrated into a 40 cm square of wood placed on the floor. The user is asked to move their weight on the device in order to activate the sound production. As a result, it presents the necessity of using many abilities, including balance, proprioception, and step control. All three tangible ADMIs can trigger audio samples or random MIDI notes. MIDI notes are associated with a granular synthesis algorithm with a chain of effects that enriches the timbre and prolongs the duration of the event over time. Note that the MIDI note is in this case used only to change the general frequency of the sound, and it is not intended from a tonal perspective. At each activation, audio samples can also be randomly selected from multiple versions of the same musically structured event. This increases the richness of the resulting sound, making it more interesting and engaging for the musicians¹⁸. Sound Creator is a tool based on the hand tracking algorithms made available by Monica Lim ¹⁹ and uses the same “Google MediaPipe” algorithms described above. The hand tracking data are sent via OSC protocol²⁰ to a MAX patch for sound production. The user is required to have the same skills as the “Elastic,” with the addition of the rotation movement of the wrists and the opening of the palms, used to activate different sound production processes. The relative position of the hands in relation to the screen allow thickening or thinning out the sound events and managing the dynamics. These characteristics make the “Sound Creator” a more complex instrument to use compared to the “Elastic.”²¹

AUMI (Adaptive Use Musical Instrument) is a tool based on motion tracking algorithms written in various stages by AUMI developers. While it was designed for users who can only move their head, AUMI actually responds to object, body, and mouse movements as well. AUMI is composed of various tracking and sound modules, which provide many solutions for interaction and sound production. Based on the position of a cursor relative to a grid and linked to head movements, AUMI plays either audio samples or MIDI notes. In the experiments, authors used the desktop version with a basic two zone grid associated to Midi notes production or audio files (see Figure 2, left side).

Figure 2

Figure 2. In the left the AUMI desktop interface with the two zones vertical grid employed in the experiments. In the middle, the Mismo and in the right the Glove.

3.1.2 Wearable/prosthetic

The instruments belonging to this category can be worn, grabbed or tied to the body. As such, they can be considered as real physical extensions, providing a sense of power and great satisfaction in sound production. They allow a variety of movements and sound feedback and – like touchless tools – require movement control, proprioceptive abilities, as well as the ability of moving in midair and without reference points.

Mismo (Modular Inertial System for Multimodal Operations) is a modular system based on inertial sensors such as accelerometer, gyroscope, and magnetometer that allow real-time monitoring of the device's orientation in space. The hardware is developed around the ESP32 microcontroller²², data exchange with computers is wireless, and power is provided by a rechargeable battery. Thanks to the lack of cables and to the dimension of the hardware, the users can grab the device in their hand and move it freely (see Figure 2, middle side). If the hand lacks grip, Mismo can be applied with an elastic band to one of the limbs or to the head. The data processed by the microprocessor are converted into MIDI notes. Clockwise rotation on the azimuthal plane (Yaw)²³ of the device produces high notes, vice versa low notes. Rotating Mismo upwards or downwards (Pitch) allows for control over dynamics, while lateral rotation (Roll) activates sound manipulation processes.

Glove is an instrument that follows the same design as Mismo but adds independent motion sensors for each individual finger. Unlike Mismo, which is handheld, Glove is a fully wearable instrument: its body is 3D printed using parametric techniques, allowing it to be adapted and customized according to the musician's physical needs (Figure 2, right side). Data from wrist rotation is converted into MIDI information for pitch and dynamics, while finger movements are encoded into Control Change (CC) messages for timbral and/or spectral manipulation.

3.1.3 Mouth-operated

Currently, in this category there is only the Anemometer, a breadth-based instrument, which works via rotary device with reed switch, and generative musical algorithms. The hardware is developed around the ESP32 microcontroller, data exchange with computers occurs via cable with MIDI protocol. The user is required to have a good ability to sustain air flow, although the Anemometer can also be activated with the hands. The sound output consists of various MIDI notes played on different channels. The user can interact with the generative algorithm by choosing to rotate the sensor more or less quickly, thus producing events with greater or lesser density²⁴

3.2 The musical domain: composition, improvisation and conduction techniques

The second objective of the I-Ork project is the building of inclusive and friendly musical environments, where every participant can express him/herself with as few limitations as possible. For “friendly musical environment” authors essentially mean:

• Soft or free timings of musical events. This is possible thanks to the organization of the composition in macro areas that group together musically related events. Once in the area, it is not important when exactly the event will be played, but rather that only the selected events appear regardless of the order. This informal music practice, theorized by the avant-garde composers of the post-war period of the 20^th century, implies a certain amount of indeterminacy in the musical result and involves improvisation in the performance (Adorno, 2004). Informal music then flowed into the compositional methods of electronic music, exploiting the enormous possibilities for sound manipulation that it offers (Roads, 2015). For the “I-Ork” project, informal music represents a good strategy for managing musical events in order to maintain a certain level of organization without too stringent time constraints. Authors also work on devices designed to resist involuntary stresses that could damage their operation. The purpose of these measures is to make the use of technology functional, efficient and as non-invasive as possible. The experience of producing sounds through gestures that belong to everyday life is a precious element for both disabled and non-disabled performers. However, it should not be forgotten that for some of them these gestures are not at all simple and that sometimes their practice is an achievement to be pursued.

• Rewarding and meaningful musical result. The mapping strategies, especially the one-to-many employed in the ADMIs are aimed at a sound design that is as expressive and engaging as possible. The use of sound in rehabilitation helps the achievement of motor tasks, making exercise repetition easier and more motivating (Bevilacqua et al., 2018). This is why authors need to use refined sound design techniques and generative music algorithms that, following a simple gesture, offer a rich and engaging musical result. Furthermore, the possibility of including all these elements within a shared performative event only increases the expressive and communicative meanings of the experience of using ADMIs.

• Equity in roles and tools assignment. As pinpointed by Samuels (2019, p. 154), equity may be an issue in an all-abilities group of performers, where some instruments may be perceived as simplified with respect to others. To avoid these differences, the I-Ork project employs only ADMIs in its performances. This implies a shift in the composition practice, as the ADMIs usually do not produce single notes but rather complex musically structured events.

3.2.1 Musical styles and therapy

Discussion of the impact that musical languages and sound qualities have on music therapy patients is very scarce in the literature (Frid, 2019, p.11), probably because most of the therapists don't even consider the possibility of using languages other than the ones they know. And yet already in 2004 Ellis had considered the great potential of the use of electronic vocal sound treatments for music therapy (Ellis, 2004). His idea of “aesthetic resonation” as a way of improving vocal expression, range of phonemes and listening skills is deeply linked to the emotional impact that electronic sounds can evoke in patients. As already outlined in Section 2.3 other authors Partesotti et al. (2018); Förster and Lepa (2023); Ward (2023) consider very positively the great possibilities offered by ADMIs in terms of self-expression, inclusion, and involvement. The musicians of the AUMI collective offer important reflection about the concept of control, mainly bound to the possible technical failures of the AUMI motion tracking software. Their criticism against the use of traditional music in therapeutic settings highlights the unique qualities of ADMIs: “[AUMI] … unleashes artistic impulses not bound by physical and social conventions that have long defined Western art music. It expands access to the palette of sounds from which music can be made.” (Ciufo et al., 2024, p. 51). However, the importance of the musical use not only of ADMIs but of digital musical instruments in general remains largely underestimated (Jordà, 2004). And yet, it should not be ignored by designers and programmers that musical instruments are not separate from the expressive and social use that is made of them (Bates, 2012), and that therefore the two areas should grow in relation to each other. Another important musical issue is musical conduction, which has been defined by Veronesi (2019) as the practice of communicating through conventional signs with musicians in improvisational contexts with no written scores. The signs are performed by a conductor and regulate the entrances and exits of the musicians, the musical form and the content of the various interventions. Although it originated in jazz, the practice was soon extended to other musical genres including electronic music, also using other forms of communication in addition to gestures (Veronesi, 2017). As such, conduction may help to manage a musical event with musicians who cannot read a score or even have difficulties in responding to a prompt or in relating to others. Important recommendations in this area can be found in educational practices, which provide numerous instances of conduct intended to help kids create music even if they are not familiar with musical notation. These include:

• informal scores, where musical events are annotated in the form of graphical elements (Orff, 1977, vol. 1, p. 20)

• fairy tales narratives, where certain musical events are associated with the text (Orff, 1977, vol. 1, p. 39)

• nursery rhymes to be accompanied by sound gestures, melodies, and mimics (Orff, 1977, vol. 1, p. 44)

3.2.2 “In the Deep Blue”: an example of multi-modal musical conduction

To experiment with these issues and to demonstrate the musical validity of a performance with only accessible instruments, authors produced an event with an ensemble of only ADMIs²⁵. In order to create a conduction potentially accessible even to the blind, authors chose a multi-modal approach, providing visual and auditory cues through the use of a video and voice guide, which told the story of a curious bluefish exploring the depths of the ocean. The musical storytelling was divided into three sections, each enhanced by specific instruments and sounds that personified the characters and described the underwater environment:

1. A curious little bluefish swims joyfully along with his friend, a goldfish. The Anemometer represents the blue fish and its lively spirit with melodic tones. AUMI plays the role of the goldfish with suitable sounds to create a harmonious duet with the protagonist. Sound Creator adds texture to the scene, evoking the rich palette of corals through shifting sounds. While exploring, the two friends pass through air bubbles, produced by Push2D, and jellyfishes, represented by TapMat (Figure 3, left side).

2. The visual and musical setting changes when the bluefish ventures alone into the mysterious depths of the ocean. Here, the Glove helps to create tension by introducing the appearance of a shark, represented by TapTile. Fortunately, the goldfish helps the bluefish escape away from the shark (Figure 3, middle side).

3. The story ends when the two friends return to swim safely among the coral. The visual and musical elements recall those in the first section and bring back a feeling of calm and quietness. Mismo contributes in the creation of a magical musical ambience (Figure 3, right side).

Figure 3

Figure 3. Three screenshots of the video used for the conduction of “In the Deep Blue.”

This strategy of conduction may allow performers to follow the storytelling and participate more easily in musical improvisation. The integration of visual storytelling and auditory cues is effective for people with disabilities, encouraging sensory and emotional involvement.

3.3 The therapeutic domain: music therapy approaches and community music making

The third and fourth objective of the I-Ork project is integrating musical performance in the music therapy practices with the aim of:

• contributing to the improvement of the patient's motor skills through musical exercises with the ADMIs;

• fostering social, cognitive and communicative attitudes by organizing collective music performance sessions with all-abilities musicians, including external audiences.

This last objective has been theorized in a new context of music therapy studies called “community music making”. Ruud (2004) has defined community music making as a way of thinking about music therapy that is not directed at the clients, but rather aimed at changing the musical system around them. This implies a redefinition of the traditional model of music therapy interventions in terms of places, objectives, and therapist's attitude. While in the traditional model the place of practice must be reserved and protected from external influences, in community therapy the place can be anywhere you can make music, in a room, in an auditorium or even outdoors. The role of music also undergoes a profound change. From a tool for therapy, it becomes the glue of a healing community where social relationships are a founding element of the activity. Finally, the role of the therapist also evolves toward new values. It moves away from a figure similar to a psychotherapist who has a private relationship with the patient to take on the role of musical animator, instrumentalist, arranger, etc. (Caneva and Mattiello, 2018). However, conventional model and community music therapy can also coexist, or rather complement each other. According to Wood et al. (2004) the transition can take place through successive and gradual phases that progressively introduce the client to the new environment. This shift aligns with the objectives of the I-Ork project, where music serves as the medium for both therapeutic and creative interventions. But to fully understand how this can happen, authors need tools for observing client's behavior when using the ADMIs, and for measuring their progress from a motor and cognitive point of view. This comes from the work of Baxter et al. (2007), who, thanks to the participation of a team of music therapists, produced the Individualized Music Therapy Assessment Profile (IMTAP), an assessment procedure for various areas of operation in improvisational music therapy interventions. IMTAP is based on the profiling of 374 skills subdivided into 10 different domains and various subdomains all listed in Table 2.

Table 2

Table 2. The 10 domains of IMTAP with description.

The Fine motor domain is mainly devoted to describing abilities related to the use of the most common instruments in music therapy, such as guitar, piano, or percussion. For instance, the ability FM E.I (coordinates both hands) is listed in the subdomain “E. Piano.” However, in this research authors are not interested in the ability of “Client uses both hands simultaneously to play isolated keys.” (Baxter et al., 2007, p. 48), but rather in the ability of hand coordination when moving in front of a camera and within the camera space. According to IMTAP's classification, wrists rotation and finger articulation should belong to the Fine motor domain, but they are not listed, probably because these movements do not belong to traditional instrument practice. Also, the ability of sustaining airflow for a while it is not present in the domain of Oral motor skills, subdomain B. (Air production). This analysis highlights the potential of ADMIs to expand the gestural repertoire compared to traditional instruments, outlining their rehabilitative capacity.

4 Methods

The I-Ork project originates from the interplay of three fundamental domains: technological, musical and therapeutic. As such, it is a complex project whose development will take place through various progressive phases. To begin testing some ideas and principles set out in Section 3, authors started with observing reactions, behaviors, and barriers or difficulties in the use of their ADMIs from clients of various ages and disabilities. The observation aims at collecting data about:

1. motor and sensory skills observable during the use of the ADMIs;

2. social, cognitive, and musical skills fostered by the musical activities with ADMIs;

3. clients' involvement and emotional impact during the use of the ADMIs.

For collecting observations, authors involved two music therapy centers. One is the “Accademia di Franciacorta” in Bornato (Brescia, Italy), the other “Dandelion APS” in Monselice (Padova, Italy). The observation sessions took place in February and April 2024, respectively. Each session was dedicated to observe the behavior of five clients per therapy center for a duration of 45 min for each individual. Six observers took part in each of the two observation sessions. In the Bornato session, observers were, beyond the three authors, an ADMI's programmer and two music therapists. In the Monselice session, beyond the three authors, an ADMI's programmer, a music therapist, and a disability counselor. Each session was led in turn by a different person, from now referred as the therapist. Video clinical records were used as a secondary source of data in all experimental sessions.

4.1 Participants

There were 10 clients in total (4 female, 6 male) who had a range of diagnoses which were mostly congenital in nature. Four of the clients were children aged 8–12 years, four were adolescents aged 13–17 years, and two were adults aged 18 and 22 years. Some of the participants presented autism spectrum disorders, one was blind, two were in wheelchairs, and the rest of them presented mixed impairments. Most of them had various experiences as music therapy clients without technological devices. A written consent to participate in the observation studies, data collection, and video recording of the clients was signed by the parents or by the client him/herself if of age. Table 3 summarizes all the specific information about them.

Table 3

Table 3. Specific information about the clients.

4.2 IMTAP for the evaluation of musical activities with ADMIs

Authors employed the IMTAP to define specific domains and subdomains related to musical activities with their ADMIs, also if they are quite different from instruments traditionally used in music therapy. This required some modifications or integrations with respect to the IMTAP original abilities. With regard to the abilities necessary to play their ADMIs, authors identified the 12 skills, reported in Table 4 where, following the conventions used by the authors, domains are expressed by acronyms, subdomains by capital letters and abilities by Roman numerals. The Fine motor domain is mainly devoted to describing abilities related to the use of the most common instruments in music therapy, such as guitar, piano, or percussion. For instance, the ability FM E.I (coordinates both hands) is listed in the subdomain “E. Piano.” However, in this study authors are not interested in the ability of “Client uses both hands simultaneously to play isolated keys.” (Baxter et al., 2007, p. 48), but rather in the ability of hand coordination when moving in front of a camera and within the camera space. According to IMTAP's classification, wrists rotation and finger articulation should belong to the Fine motor domain, but they are not listed, probably because these movements do not belong to traditional instrument practice. Also, the ability of sustaining airflow for a while it is not present in the domain of Oral motor skills, subdomain B. (Air production). This analysis highlights the potential of ADMIs to expand the gestural repertoire compared to traditional instruments, outlining their rehabilitative capacity.

Table 4

Table 4. The 12 IMTAP skills related to the use of ADMIs.

4.3 The observation protocol

Each session is organized according to an observation protocol subdivided into 5 phases. For each phase, domains, subdomains, and a total of 23 skills are defined.

1. Greeting (3 skills)

2. Exploring ADMIs (5 skills)

3. Use of ADMIs (8 skills)

4. Choice of preferred ADMI (3 skills)

5. Performance (4 skills)

Observers express qualitative judgments for each skill, according to the complete Table 5.

Table 5

Table 5. The 23 IMTAP skills tested during observation sessions.

4.3.1 Greeting

In this phase, the domain of social abilities is evaluated. These include greeting the therapist (SOC A.VI), using appropriate eye contact (SOC A.VIII), and interacting appropriately with the therapist (SOC A.V).

4.3.2 Exploring the ADMIs

In the testing activities, authors excluded the use of Glove due to its complexity. Thus, only 8 of the 9 tools introduced in Section 3.1 are used. The client is encouraged by the therapist to explore the 8 instruments placed on a table in front of them. The therapist intervenes only if necessary. In this phase, social, emotional and musical skills are assessed.

4.3.3 Use of ADMIs

The 12 skills related to the use of ADMIs have been analyzed in Table 2. However, in the observation protocol they have been reduced to 8 for brevity and simplicity for observers. The skills belong to gross motor, sensory, fine motor and oral motor domains. Head movements, wrist rotation, and ability to sustain air flow have no IMTAP code, because they are not present in the IMTAP protocol.

4.3.4 Choice of preferred ADMI

In this phase, the client is required to indicate the instrument which they like the most. This requires social and cognitive skills such as demonstrating interest in the activity (SOC A.III), confidence (SOC A.XII), and the ability of making a choice (COG B.V).

4.3.5 Performance

In the final phase, the therapist chooses an instrument and proposes a musical duet with the client. In this phase, social skills are put into play such as proper interaction with the therapist, understanding of rules, and waiting for turn (SOC A.V, SOC A.X, SOC C.II). Enjoyment of musical interaction in the form of proposal/answer is also evaluated (MUS A.II). Observers deliver their judgement for each of the 23 tested skills in a free textual form, completing a customized Google form²⁶. Authors made this choice because in this study authors were more interested in collecting observers' impressions and opinions rather than quantitative evaluations.

4.4 Data analysis

The Google Form grids, completed by observers, were analyzed with textual judgements for each of the 23 skills of the observation protocol's phases. The tool used for this purpose is Lexicool²⁷, which provides statistical information on the repetition of phrases and keywords. Additionally, words and short phrases with information about the client behavior were manually extracted from text, labeled as sub-themes, and their occurrences reported phase by phase for each client. In this way, for each client, authors obtained a dataset consisting of 5 rows (each for every observation phase) and 2 columns. In the first column, authors have a score reporting the percentage of positive occurrences related to the observed skills obtained by each subject during the 5 observation phases. In the second column, authors have a list of keywords or short phrases used to describe the client's IMTAP abilities, as well as the frequency with which they occur. Based on these data, authors perform a quantitative as well as a qualitative analysis.

4.4.1 Quantitative analysis

The authors analyzed each observer's responses textually, assigning 1 point for each explicitly positive observation and 0 points for other different responses. The points for each skill within a given observation phase were then summed and converted into a percentage, using the hypothetical maximum score (i.e., all observations being positive) as a reference. For example, in the “Greeting” phase, the maximum score was 18 (1 point for each of the 3 skills, evaluated by 6 observers). In the case of Client C, the total score for the “Greeting” phase was 5, which yields a percentage of 27.8% (5/18 × 100). This approach allowed the authors to standardize the data and visualize the results more clearly. To gain a comprehensive overview of the entire music therapy session and to compare the experiences of different clients, the authors summed all phases scores to obtain the overall score (TOT). The results for each client are reported in Figure 4.

Figure 4

Figure 4. Scores obtained by clients in each of the 5 observation phases.

4.4.2 Thematic analysis

Keywords and short phrases in column 2 of the dataset are summarized in Table 6, where 26 labels are organized in sub-themes (listed in parentheses), themes (with code) and 6 overarching themes:

• A. Guidance and mediation includes both the need for physical or behavioral guidance and the mediation and support role played by parents or other reference figures.

– A1 Parental mediation outlines the need of some clients to overcome the confusion resulting from dealing with a new figure and new musical activities.

– A2 Assistance highlights the need of many clients to be supported in the proposed activities both vocally and physically, and also the need to be encouraged and stimulated to face new experiences. Assistance include guidance, physical support, physical contact and stimulation.

• B. Communication and emotional interaction concerns the modalities of verbal and non-verbal expression, positive emotions such as joy, but also aspects related to negative emotions such as rejection and fear.

– B1 Vocalization is a typical behavior of people with autism or other disabilities, such as paralysis who may exhibit repetitive vocalizations, echolalia (repetition of words or phrases), or other nonverbal sounds as a form of sensory stimulation or communication.

– B2 Positive emotions has many sub-themes describing how participants express emotions while using ADMIs, their joy for music, and their degree of comfort in the proposed activities (optimal interaction).

– B3 Negative emotions has many sub-themes too, describing shyness, unsafety, discomfort, and even fear while navigating the different phases of the protocol and interacting with the therapist and with the ADMIs.

– B4 Non-verbal communication includes facial expressions and even a final bow at the end of the musical performance.

– B5 Positive attitudes refer to the client's general attitude and their propensity to participate in the proposed activities.

– B6 Negative attitudes include refusal and contact avoidance, a behavior very often detected already in the “Greeting” phase.

• C. Interaction with ADMIs summarizes the attitudes and reactions recorded during the instrumental activities. They include:

– C1 Discovery denotes a desire to explore and experiment not only with the single instrument, but in general with the entire musical environment set up for the experiment.

– C2 Improvisation refers to the fact that some clients spontaneously invented small musical behaviors, such as proposal/response games, during the exploration of the instruments.

– C3 Interest testifies to the curiosity that ADMIs have aroused in many clients.

– C4 Short interaction concerns the very short time in which some participants interacted with ADMIs, as if they were simultaneously attracted and scared. This was noted in conjunction with shyness (B3), negative attitudes (B6) and the absolute need for parental mediation (A1).

– C5 Turn respect it is the IMTAP ability SOC C.II observed in the “Performance” phase, and it is a good predictor of the capacity of playing in coordination with other clients.

– C6 Passivity is the opposite behavior of C3.

– C7 Need for more time reflects the difficulties experienced by some clients in adapting to an unknown environment in the restricted time of the experiment.

– C8 Alternative use of instruments shows the critical spirit of many users who, where the instrument allowed it, used alternative and easier ways to play it.

– C9 Conditioning shows the dependence of the choices of instrument or type of sounds or music on previous experiences.

• D. Imitation and repetitive behaviors concerns motor conducts related to repetition, imitation, and automatism. It focuses on mechanical and stereotyped actions, which reflect a way of learning or responding to the environment through repetitive patterns.

– D1 Emulation is linked to the presence of a known person (the therapist or a parent) who proposes to the client interaction models with the ADMIs.

– D2 Stereotyped behavior is a characteristic of people with autism and comprehends automatic and mechanical movements.

• E. Commitment and ability to concentrate focuses on the degree of commitment and concentration of the client. It can concern both the ability to focus on a task and the difficulties related to poor concentration.

– E1 Fatigue is an issue particularly important for people affected by pathologies that severely limit their movements.

– E2 Detachment leads to distraction and superficiality, due in any case to poor involvement or selective attention.

– E3 Commitment, on the contrary, leads to determination and concentration in the proposed activities.

• F. Responses to stimuli collects behaviors and reactions related to visual feedback and other sensory stimuli. It includes the ability to respond to what happens in one's surrounding, as well as the difficulty/interest in specific stimuli.

– F1 Visual. Some clients were drawn to the rich visual feedback produced by Elastic and Sound Creator, as well as the image of their hands moving on the computer screen.

– F2 Sound source. The separation between digital instruments and the actual sound source is an issue in electronic music that proved to be particularly important for some clients, who were listening and searching all the time for the point from which the sound vibrations actually came.

– F3 Tangible instruments expresses the preference for instruments that can be manipulated by touch.

– F4 Environmental influence relates to the difficulties of some clients to feel comfortable in an unfamiliar environment and in the presence of many unknown people.

Table 6

Table 6. Overarching themes, themes and sub-themes extracted from textual responses, with occurrences for each client.

5 Results

In this section, authors analyze in general the main themes emerged from qualitative analysis, trying to contextualize them within the observation activity carried out during the five phases of the experiment. Then authors focus on single clients with a brief personal profile aimed at delivering a more precise idea of their relationship with ADMIs and musical interaction. Finally, authors will attempt an overall evaluation of their ADMIs.

5.1 Main themes from qualitative analysis

The first overarching theme is Guidance and mediation, with the Assistance (A2) labels obtaining the major number of occurrences. In most cases, the need for the active presence of the therapist emerges, with specific assistance depending on the diagnosis, and the necessity to adapt activities according to individual abilities and preferences (clients F, I, and J). Furthermore, the presence of family members and reference therapists (A1) has proven to be important as social mediators in activities (clients A, B, C, and G). This is probably due to the impromptu nature of the intervention, which did not give clients time to adapt to new people and new environments.

While interacting with ADMIs positive emotions are more numerous than negative ones (B2), thus creating the optimal conditions to be able to enjoy the benefits and pleasure of making music offered by ADMIs. Also important is the result of Interest and Discovery (C3 and C1), which outline the importance of an open attitude, curiosity, desire to experiment, and the great interest that ADMIs arouse in most clients.

Theme D2 (Stereotyped behavior) highlights the presence in some clients of automatic and mechanical movements, which, however, are a characteristic linked to their pathologies and which has not disturbed the experience with ADMIs too much. For instance, in the case of client F stereotyped behavior combined with commitment and determination (E3) made him an excellent ADMIs' scout.

Among sensory stimuli, the visual output produced by Elastico and Sound Creator is the most remarkable. Clearly, multimedia productions where sound and image are coordinated and both depend on the client's movement have a great attractive capacity that can contribute to increasing the therapeutic benefits of the interaction.

5.2 Client's experience with ADMIs

In this section authors describe how participants deal with specific motor challenges, interact with tools and express their musical preferences and skills. These evaluations are referenced through the label codes reported in Table 6. Authors also relate the individual behaviors to the results of the quantitative analysis reported in Figure 4, focusing on the observation phases where the client obtained the highest score.

• Client A with his highest score in the “Choice of preferred ADMI” phase (66.6%) shows a selective interest for Mismo having barely interacted if not openly rejected all other tools. Client A demonstrates good hand grip and a genuine interest in finding the source of the sound generated by Mismo (F2). As he gradually gains confidence, he moves more and more in the environment and creates a kind of game alternating moments of sound with moments of silence in which he places the Mismo in the hands of the therapist (C2).

• Client B has a good coordination while playing alternately Push2D, TapMat and TapTile. He ignores AUMI and Sound Creator but pays some attention to Elastic and Mismo showing at the beginning indecision and then making a clear choice for Mismo (88.8% in the “Choice of preferred ADMI” phase). When playing it, he produces many vocalizations (B3) and moves a lot in the environment. In musical interaction tends to emulate the therapist (D1) rather than respect his turn.

• Client C shows a good balance while playing TapTile and a good coordination while playing alternately Push2D, TapMat and TapTile, obtaining her maximum score in the ”Use of instruments” phase (58.3%). With all other tools, she has a very short interaction (C4) due to extreme shyness (B3). She tends to avoid contact with the therapist (B6) and therefore it is impossible to experience any musical interaction.

• Client D obtains his maximum score in the “Greeting” phase (94.4%) more or less in line with the scores obtained during the other phases (always higher or equal to 75%). Although with somewhat mechanical gestures (D2) client D tests all tools successfully, with a good hand grip in Mismo and good wrist rotation in Sound Creator. He is also interested in the visual feedback of Elastico. Due to previous experience with drums (C9), he prefers percussive sounds played with AUMI not with head movements, but with the mouse (C8). In the musical interaction, he is able to respect his turn only after repeated requests from the therapist.

• Client E already from the “Greeting” phase refuses any type of contact with the therapist (no score). He only walks around the table where the instruments are placed in time to the sound of a song used during his music therapy sessions (C9). This behavior prevented the implementation of all the following phases of the protocol.

• Client F as a blind shows some difficulties in getting balanced while stepping on TapTile. However, balance improves after some trials. With Elastico, AUMI, and Sound Creator she seems a bit uncertain due to the lack of visual feedback which prevents her from calibrating hands movements. She also shows little interest in the other tools, giving her preference to TapTile (100% in the “Choice of preferred ADMI” phase) probably due to conditioning from previous positive experiences with this tool (C9). She is very positive in turn respect (C5) and in musical interaction with the therapist (92.5% in the “Performance” phase).

• Client G with a score of 27.7% in the “Greeting” phase is completely dependent from parental mediation (A1), and appears very reluctant to accept relationships with other people (B6). Moreover, she seems disturbed by the environment around her and by the presence of many people (F4). Among the various tools she interacts briefly only with Anemometer moving it only with hands (C8). No musical interaction has been experimented.

• Client H explores all the tools carefully, for a long time and without difficulty (C1), also accompanied by some vocalizations (B1). Shows good hands coordination and interest for the visual output of Elastico (F1). He appreciates also Sound Creator, experimenting for a long time with many gestures (union of fists, opening and closing of hands, and movements of the torso), thus discovering autonomously the timbre changes produced by wrist rotation. He is one of the few clients who experimented the Anemometer with the breath instead of just with the hands. He has a score of 91.6% in the “Use of instruments” phase. He is more interested in exploration (C1) than in choosing a particular instrument. The musical interaction with the therapist is positive only after vocal guidance (A2).

• Client I already from the “Greeting” phase (95.2%) shows great availability and openness toward the new experience. At the time of the experiment, she is sitting in a wheelchair and therefore has serious impediments in exploring many of the ADMIs. However, she manages to make some attempts with the help of the therapist (A2). She tries Elastico and AUMI in both cases with some difficulties in coordinating hands movements. She has even greater difficulty when she experiments with Sound Creator and with Anemometer. The use of Mismo demonstrates a weak and ineffectual hand grasp, necessitating the use of a rubber band as support. Despite the difficulty in holding the instrument, as soon as she connects the production of sound to movement, she concentrates in the attempt to control the movement of her arm (E3). All this activity is accompanied by clear expressions of joy (B2), but also by fatigue (E1). The musical interaction with the therapist is characterized by free performance (C2) with many vocalizations (B1) and expressions of satisfaction.

• Client J due to his illness is sitting in a wheelchair and shows severe limitations of movement. He also benefits from an auxiliary pump for breathing. He tries with difficulty to coordinate hand and head movement respectively with Elastico and AUMI (E3), and discovers how to move the Anemometer using the power of his air pump. He also tries to move Mismo by rotating the device with his fingers (100% in both the “Choice of the preferred ADMI,” and “Performance” phases). He is able to interact with the therapist and to engage in musical discourse by respecting his turn. Despite the fatigue during the experiment, client J managed to overcome all difficulties thanks to his determination and positive intelligence (B5).

5.3 Final evaluation of ADMIs according to thematic analysis

Regardless of individual choices, thematic analysis obtained from all 23 observed skills suggests that Mismo, Elastic and AUMI are the most effective tools to facilitate musical interaction and participation. The analysis shows that instruments that require less effort to play autonomously, or that offer immediate and effective feedback, tend to hold the client's interest for longer.

With regard to Anemometer and TapTile clients may be initially uninvolved, but involvement grows with success in exploring and using ADMIs, especially with the help of the therapist.

Sound Creator arouses in general less involvement and interest due to the challenging motor mechanism needed to control the instrument. However, in some clients with excellent hand coordination Sound Creator provides ample space for musical exploration and control of sounds. TapMat and Push2D prove to be less interesting in the long run due to lack of control over sound production and limited interaction possibilities.

The use of touchless ADMIs with visual feedback generation (Elastic and Sound Creator) leads to positive and satisfying interaction for most participants, some of which tend to actively explore the instrument (clients H, I). However, wrist rotation in Sound Creator is limited, and the movement is not performed fully or consistently in particular by clients F, I, and J. In the case of Anemometer tactile stimulus and manual control are more rewarding and manageable than blowing. In fact, most clients, end up moving Anemometer with their hands because it is easier and more satisfying to modulate movement and sound in this way. The same may be said for client C while using AUMI with the mouse instead with the head (C8).

6 Discussion

In this study, the authors explored client interactions with ADMIs, focusing on motor, sensory, social, cognitive, and musical skills, as well as engagement and emotional impact. Using both quantitative and qualitative methods, they aimed to create a comprehensive, objective view. Their observations highlighted the strengths and limitations of ADMIs in meeting diverse client needs. Overall, the authors found that ADMIs evoke positive emotions, spark curiosity, and help some clients overcome challenges in engaging with them. Positive experiences like these can support motor development and musical interaction. Notably, social skills observed in the “Greeting” phase were good predictors of later engagement, though clients with lower initial scores can still benefit over time. The study also emphasized the importance of visual output, which not only captures attention but aids in hand coordination. Future use of multimedia devices combining audio and visual cues could enhance these benefits. Despite positive findings, the study also offers important critical insights.

6.1 Participant selection

The authors did not have the opportunity to select the participants based on their diagnosis, but accepted the clients who were willing to participate in the experiment. Although participants presented different disabilities and had varying degrees of music therapy experience, this diversity has not appeared as a limitation, but rather it offered an opportunity to observe the effectiveness of the instruments across a broad range of different attitudes. On the other hand, a selection of participants based on the various disabilities would probably not have been very significant, since the same disability can be accompanied by many others, so different from each other as to make the clinical picture difficult to classify. For example, see the case of clients E, F, G, all three affected by visual impairment. All three clients have different other disabilities, which determine large differences between them, despite the common visual impairment (see Table 3).

6.2 ADMIs

At the starting point of the “I-Ork” project, the authors were inspired by their previous experience with different ADMIs. Some of these were designed using commercial sensors or libraries freely available online, others were built from scratch by the second author. Once again the authors chose not to select among the available ADMIs but to propose them all during the observation phases. On the other hand, the instruments themselves were explicitly designed according to “Universal Design Principles” (Savidis and Stephanidis, 2004), ensuring they would be accessible and usable by individuals with different needs. However, while this approach has proven to be very useful for collecting data on the usability of the ADMIs, the same cannot be said for the clients who in some cases found themselves a bit lost in front of so many possibilities of musical interaction. This was for instance the case of client G who was particularly sensitive to the environment and the presence of many people around her. It is also important to note that ADMIs may have a strong sensory impact. While this can result stimulating for some individuals, it can represent a significant obstacle for others. Therefore, in the future it will be necessary to plan sessions with a fewer ADMIs in order to help customers stay focused and reduce their overall sensory impact. Another important issue concerns the audio setup used in the experiments. Some clients appeared disoriented by the lack of connection between the instrument and the sound source (see F2 Sound source in Section 4.4.2). In fact, the authors set up a stereo output that was the same for all the ADMIs, without considering that some of them were positioned far from the speakers. Although this can be easily solved by using small monitors placed near each ADMI, rather than a central sound diffusion system, it remains a problem for instruments like Mismo that can be played while moving freely in space.

6.3 Assessment

As explained in Section 4.3 authors designed the evaluation grid not for collecting quantitative ratings but rather to stimulate observers in defining with their own words the behavior of the clients. An evaluation form based on the Likert scale would have been immediately interpretable and would have allowed for easier data collection. However, to express a reliable evaluation, it is necessary to have in mind previously tested and a well-defined rating models, which was impossible in the case of this research. So authors opted for an open response form. The data extracted from the evaluation grids were then analyzed employing textual analysis tools (see Section 4). During the analysis, all three authors worked together concurrently and adopted the “Think Aloud Method” to discuss and interpret the data, thereby minimizing potential biases (Ericsson and Simon, 1984). While the authors have used different methods to ensure that the evaluations are as objective as possible, however, the absence of a professional music therapist in the research team may represent a limitation both in the planning and in its evaluation.

6.4 Musical expression

In Section 3.2.2 authors presented a musical prototype of a possible performance made with only ADMIs. The realization of this experience is important because it provides a concrete idea of the point of arrival of this and other future experiments. In fact, from the point of view of a project like “I-Ork” it is not enough to deal with the technological and therapeutic aspects of ADMIs – which are of fundamental importance – but it is absolutely necessary to offer an idea of the musical context where these tools can be employed. Actually, during the experiments all the tangible tools (Push 2D, TapMat, and TapTile) played the sounds used in “In Deep the Blue” (see Section 3.2.2). These are sounds designed to make the underwater environment of the story as attractive as possible. During the experiments, observers noticed that some clients returned repeatedly to tangible ADMIs. However, it is unclear whether this behavior stemmed from the ease of interaction or simply from the enjoyment of the interesting sounds the instruments produced. This is an aspect that should certainly be investigated more thoroughly in the future.

7 Conclusion and further work

The aim of the “I-Ork” project is to promote groups of performers with different abilities, all playing only accessible digital instruments (ADMIs) in the spirit of community music making. The present study represents the first step of the project, where authors focus their interest on the use of ADMIs. The results show that Mismo, Elastic, and AUMI are the instruments which have aroused the greatest interest among the participants in the experiment, even though the other ADMIs have also demonstrated good potential at least for some clients. Thanks to both quantitative and qualitative analysis, authors were able to outline many important themes connected to the use of ADMIs, as well as a personalized profile for each of the clients (Section 5.2). However, the perspective of the I-Ork project is not limited to the design and implementation of prototypes, but also extends to the use of ADMIs in various musical performances. During sessions, authors encouraged the clients not only to explore the ADMIs, but also to use them musically while interacting with the therapist. It must be said that only a few of the clients spontaneously demonstrated the ability to understand rules and structures as stated in the IMTAP protocol (SOC A.X). This indicates that in order to proceed from a mere exploration toward a musical performance, whatever its nature it is, a lot of pedagogical effort needs to be done. In fact, it is necessary to build a pedagogy of ADMIs that sets clear objectives commensurate with the potential of the client. The musical example given in Section 3.2.2 serves as a point of arrival for future experiments, as it is just one of the many possibilities waiting to be put to the test.

Data availability statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://github.com/TeAchMusicEdu/Iork.

Ethics statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent for participation in this study was provided by the participants' legal guardians/next of kin. Written informed consent was obtained from the individual(s), and minor(s)' legal guardian/next of kin, for the publication of any potentially identifiable images or data included in this article.

Author contributions

MM: Conceptualization, Formal analysis, Investigation, Methodology, Supervision, Writing – original draft, Writing – review & editing. GB: Data curation, Formal analysis, Investigation, Software, Writing – original draft, Writing – review & editing. SV: Data curation, Formal analysis, Investigation, Writing – original draft, Writing – review & editing.

Funding

The author(s) declare that no financial support was received for the research and/or publication of this article.

Acknowledgments

Authors thank “Accademia di Franciacorta” in Bornato (Brescia, Italy), “Braccio di Ferro” Onlus (Padova, Italy), and “Dandelion APS” in Monselice (Padova, Italy) for their cooperation in the organization of the experiments. In addition, the authors would like to thank ADMIs programmer Maurizio Botti, all the clients, music therapists, and disability counselor who participated in the observation phases and contributed to the collection of data used in this research. And, of course, a big thank you to all the parents who have dedicated their time allowing their daughters and sons to participate in this experience.

Conflict of interest

The authors declare that the research was conducted in the absence of commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declare that no Gen AI was used in the creation of this manuscript.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^The Disklavier is an acoustic piano equipped with electronic sensors, which allow the movement of keys, hammers, and pedals. These respond to MIDI messages to provide an automatic performance of a music composition.

2. ^VOCA (Voice Output Communication Aids) are devices designed to support people with disabilities in personal communication https://www.hpft.nhs.uk/services/learning-disability-services/useful-resources/switches-for-communication/.

3. ^Garageband is a software for music production included in the operating systems macOS and IOS https://it.wikipedia.org/wiki/GarageBand.

4. ^http://www.soundbeam.it/

5. ^“MidiGrid” allows the control of a synthesizer by the activation via mouse of the musical content stored in a cell matrix https://www.adsrsounds.com/product/software/midigrid/.

6. ^“MIDIcreator is made of a transparent resin filled with sensors that transform pressure, tap or hits into MIDI messages for music production.

7. ^https://www.drakemusic.org/

8. ^http://aumiapp.com/aumi.php

9. ^Windows is the current operating system for personal computers. MacOS is the operating system for Mac desktop and laptop computers. IOS is the operating system for Apple iPhones and iPad.

10. ^https://www.fulcrum.org/concern/monographs/cf95jf39j?locale=en

11. ^The MIDI protocol (Musical Instruments Digital interface) is the standardized way of communication among electronic instruments.

12. ^The serial monitor allows the control of data input and output from the computer to a microcontroller and vice versa.

13. ^MAX is one of the most popular programming environment for sound synthesis and processing https://cycling74.com/.

14. ^Ableton Live is a powerful software for music creation and performance https://www.ableton.com/en/.

15. ^https://infusionsystems.com/catalog/index.php

16. ^Gross motor skills often involve the whole body with large movements; fine motor skills involve smaller parts such as wrists, hands, and fingers.

17. ^https://infusionsystems.com/catalog/product_info.php/products_id/204

18. ^A video of a musical performance with tangible ADMIs is available at https://youtube.com/shorts/2pPpIIC54qc?si=5S6PBQgt9FMVBEHR.

19. ^https://www.monicalim.online/

20. ^Open Sound Control (OSC) provides communication among various applications for music or multimedia production.

21. ^An example of performance with the “Sound Creator” is available at https://youtube.com/shorts/GtB8KxDGFU8?si=m5v_-rTB-aI0wHSz.

22. ^ESP32 is a series of low-cost, low-power system on a chip microcontroller, with integrated Wi-Fi and dual-mode Bluetooth.

23. ^The azimuthal plane is the angle that lies on the horizontal plane of a spherical coordinate system.

24. ^An example of performance with the Anemometer can be found at https://youtube.com/shorts/Eod1145yMdM?si=F2Vosb4So7e-c1VX.

25. ^https://youtu.be/_Ji6JaB_7Gs

26. ^https://forms.gle/Rh7CAm3SkqjKWLQc7

27. ^https://www.lexicool.com/text_analyzer.asp?IL=2

References

Adorno, T. W. (2004). Immagini Dialettiche: Scritti Musicali: 1955-65, ed. G. Borio (Torino: Einaudi), 322.

Google Scholar

Agres, K. R., Schaefer, R. S., Volk, A., Van Hooren, S., Holzapfel, A., Dalla Bella, S., et al. (2021). Music, computing, and health: a roadmap for the current and future roles of music technology for health care and well-being. Music Sci. 4:2059204321997709. doi: 10.1177/2059204321997709

Crossref Full Text | Google Scholar

Barnes, C. (2019). “Understanding the social model of disability: past, present and future,” in Routledge Handbook of Disability Studies, 2nd ed., eds. N. Watson, A. Roulstone, and C. Thomas (Routledge), 14–31. doi: 10.4324/9780429430817-2

Crossref Full Text | Google Scholar

Bates, E. (2012). The social life of musical instruments. Ethnomusicology 56, 363–395. doi: 10.5406/ethnomusicology.56.3.0363

Crossref Full Text | Google Scholar

Baxter, H. T., Berghofer, J. A., MacEwan, L., Nelson, J., Peters, K., and Roberts, P. (2007). The Individualized Music Therapy Assessment Profile: IMTAP. Jessica Kingsley Publishers.

PubMed Abstract | Google Scholar

Bevilacqua, F., Peyre, I., Segalen, M., Pradat-Diehl, P., Marchand-Pauvert, V., and Roby-Brami, A. (2018). “Exploring different movement sonification strategies for rehabilitation in clinical settings,” in Proceedings of the 5th International Conference on Movement and Computing (Association for Computing Machinery), 1–6. doi: 10.1145/3212721.3212881

Crossref Full Text | Google Scholar

Burland, K., and Magee, W. (2014). Developing identities using music technology in therapeutic settings. Psychol. Music 42, 177–189. doi: 10.1177/0305735612463773

PubMed Abstract | Crossref Full Text | Google Scholar

Butler, M. J. (2014). Playing With Something That Runs: Technology, Improvisation, and Composition in DJ and Laptop Performance. Oxford University Press. doi: 10.1093/acprof:oso/9780195393613.001.0001

Crossref Full Text | Google Scholar

Caneva, P. A., and Mattiello, S. (2018). Community Music Therapy. Itinerari, principi e pratiche per un'altra musicoterapia. Milano: Franco Angeli.

Google Scholar

Ciufo, T., Dvorak, A. L., Haaheim, K., Hurst, J., Leu, G. S.-e., Miller, L., et al. (2024). Improvising Across Abilities: Pauline Oliveros and the Adaptive Use Musical Instrument. University of Michigan Press. doi: 10.3998/mpub.11969438

PubMed Abstract | Crossref Full Text | Google Scholar

Costa-Giomi, E. (2004). Effects of three years of piano instruction on children's academic achievement, school performance and self-esteem. Psychol. Music 32, 139–152. doi: 10.1177/0305735604041491

Crossref Full Text | Google Scholar

Cross, I. (2009). “The nature of music and its evolution,” in The Oxford Handbook of Music Psychology, eds. S. Hallam, I. Cross, and M. Thaut (Oxford University Press), 3–18.

Google Scholar

Davanzo, N., and Avanzini, F. (2020). Hands-free accessible digital musical instruments: conceptual framework, challenges, and perspectives. IEEE Access 8, 163975–163995. doi: 10.1109/ACCESS.2020.3019978

Crossref Full Text | Google Scholar

Ebersold, S., Schmitt, M. J., and Priestley, M. (2011). Inclusive education for young disabled people in europe: trends, issues and challenges. A synthesis of evidence from ANED country reports and additional sources. Leeds: Academic Network of European Disability experts. Available online at: http://www.disability-europe.net/content/aned/media/ANED (accessed May 10, 2017).

Google Scholar

Ellis, P. (2004). “Vibroacoustic sound therapy: case studies with children with profound and multiple learning difficulties and the elderly in long-term residential care,” in Medical and Care Compunetics 1, eds. L. Bos, S. Laxminarayan, and D. O. Hervé (IOS Press), 36–42. doi: 10.3233/978-1-60750-946-2-36

PubMed Abstract | Crossref Full Text | Google Scholar

Ericsson, K. A., and Simon, H. A. (1984). Protocol Analysis: Verbal Reports as Data. Cambridge, MA: MIT Press.

Google Scholar

Eurostat (2023). Population with Disability. Available online at: https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Population_with_disability (accessed April 26, 2024).

Google Scholar

Farrimond, B., Gillard, D., Bott, D., and Lonie, D. (2011). Engagement With Technology in Special Educational and Disabled Music Settings. Youth Music Report, 1–40.

Google Scholar

Förster, A., Komesker, C., and Schnell, N. (2020). “SnoeSky and SonicDive: Design and evaluation of two accessible digital musical instruments for a SEN school,” in Proceedings of the 5th International Conference on New Interfaces for Musical Expression (NIME 2020) (Birmingham City University), 83–88. doi: 10.5281/zenodo.1176378

Crossref Full Text | Google Scholar

Förster, A., and Lepa, S. (2023). Digital musical instruments in special educational needs schools: requirements from the music teachers' perspective and the status quo in germany. ACM Trans. Access. Comput. 16, 1–34. doi: 10.1145/3616015

Crossref Full Text | Google Scholar

Förster, A., and Schnell, N. (2024). Designing accessible digital musical instruments for special educational needs schools–a social-ecological design framework. Int. J. Child-Comput. Interact. 41:100666. doi: 10.1016/j.ijcci.2024.100666

Crossref Full Text | Google Scholar

Frid, E. (2019). Accessible digital musical instruments–a review of musical interfaces in inclusive music practice. Multimodal Technol. Interact. 3:57. doi: 10.3390/mti3030057

Crossref Full Text | Google Scholar

Hahna, N. D., Hadley, S., Miller, V. H., and Bonaventura, M. (2012). Music technology usage in music therapy: a survey of practice. Arts Psychother. 39, 456–464. doi: 10.1016/j.aip.2012.08.001

Crossref Full Text | Google Scholar

Hunt, A., and Kirk, R. (2000). “Mapping strategies for musical performance,” in Trends in Gestural Control of Music, eds. M. M. Wanderley, and M. Battier (Paris: IRCAM – Centre Pompidou), 231–258.

Google Scholar

Hunt, A., Kirk, R., and Neighbour, M. (2004). Multiple media interfaces for music therapy. IEEE Multimed. 11, 50–58. doi: 10.1109/MMUL.2004.12

Crossref Full Text | Google Scholar

Johnston, D., Egermann, H., and Kearney, G. (2018). Innovative computer technology in music-based interventions for individuals with autism moving beyond traditional interactive music therapy techniques. Cogent Psychol. 5:1554773. doi: 10.1080/23311908.2018.1554773

Crossref Full Text | Google Scholar

Jordà, S. (2004). “Digital instruments and players: part i—efficiency and apprenticeship,” in Proceedings of the 2004 Conference on New Interfaces for Musical Expression (Hamamatsu: NIME), 59–63. doi: 10.5281/zenodo.1176619

Crossref Full Text | Google Scholar

Kubicek, L., Martino, L., and Zigo, J. (2011). Using music technology in music therapy with populations across the life span in medical and educational programs. Music Med. 3, 146–153. doi: 10.1177/1943862111403005

Crossref Full Text | Google Scholar

Lindetorp, H., Svahn, M., Hölling, J., Falkenberg, K., and Frid, E. (2023). Collaborative music-making: special educational needs school assistants as facilitators in performances with accessible digital musical instruments. Front. Comp. Sci. 5:1165442. doi: 10.3389/fcomp.2023.1165442

Crossref Full Text | Google Scholar

Magee, W. L. (2006). Electronic technologies in clinical music therapy: a survey of practice and attitudes. Technol. Disabil. 18, 139–146. doi: 10.3233/TAD-2006-18306

PubMed Abstract | Crossref Full Text | Google Scholar

Magee, W. L., and Burland, K. (2008a). An exploratory study of the use of electronic music technologies in clinical music therapy. Nord. J. Music Ther. 17, 124–141. doi: 10.1080/08098130809478204

Crossref Full Text | Google Scholar

Magee, W. L., and Burland, K. (2008b). Using electronic music technologies in music therapy: opportunities, limitations and clinical indicators. Br. J. Music Ther. 22, 3–15. doi: 10.1177/135945750802200102

Crossref Full Text | Google Scholar

Miranda, E. R., and Wanderley, M. M. (2006). New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. Volume 21 of the Computer Music and Digital Audio Series. AR Editions, Inc.

Google Scholar

Oliveros, P. (1974). Sonic Meditations. Smith Publications, Baltimore.

PubMed Abstract | Google Scholar

Orff, C. (1977). Music for Children: Orff-Schulwerk American Edition. Volume 1: Pre-school. Schott Music Corporation.

Google Scholar

Partesotti, E. (2016). L'interazione nelle tecnologie musicali di realtà mista: Il prototipo e-mocumu come esempio multimodale con propositi terapeutici [Tesi di dottorato, Università di Valladolid]. UVaDOC. doi: 10.35376/10324/16874

Crossref Full Text | Google Scholar

Partesotti, E., Pe nalba, A., and Manzolli, J. (2018). Digital instruments and their uses in music therapy. Nord. J. Music Ther 27, 399–418. doi: 10.1080/08098131.2018.1490919

Crossref Full Text | Google Scholar

Roads, C. (2015). Composing Electronic Music: A New Aesthetic. Oxford University Press: USA. doi: 10.1093/acprof:oso/9780195373233.001.0001

Crossref Full Text | Google Scholar

Roth, E. A. (2014). “Music technology for neurologic music therapy,” in Handbook of Neurologic Music Therapy, eds. M. H. Thaut and V. Hoemberg (Oxford University Press), 12–23. doi: 10.1093/acprof:oso/9780199695461.003.0002

Crossref Full Text | Google Scholar

Rubino, G. (2022). “Tailoring scores for an inclusive group,” in Teaching a Musical Special Educational to Pupils With Instrument Needs: Inclusion in the Italian School Model, eds. L. Rizzo (Milano: Franco Angeli).

Google Scholar

Ruud, E. (2004). Community Music Therapy. Pavilicevic e G. Andell (orgs): London.

Google Scholar

Samuels, K. (2019). The drake music project northern Ireland: providing access to music technology for individuals with unique abilities. Soc. Incl. 7, 152–163. doi: 10.17645/si.v7i1.1706

Crossref Full Text | Google Scholar

Savidis, A., and Stephanidis, C. (2004). Unified user interface design: designing universally accessible interactions. Interact. Comput. 16, 243–270. doi: 10.1016/j.intcom.2003.12.003

PubMed Abstract | Crossref Full Text | Google Scholar

Stensaeth, K. (2013). Musical co-creation? exploring health-promoting potentials on the use of musical and interactive tangibles for families with children with disabilities. Int. J. Qual. Stud. Health Well-being 8:20704. doi: 10.3402/qhw.v8i0.20704

PubMed Abstract | Crossref Full Text | Google Scholar

Street, A., and Magee, W. (2014). “Using garageband™ music software with adults with acquired brain injury at headway east london: identity communication and executive function,” in Music Technology in Therapeutic and Health Settings, ed. W. Magee (Jessica Kingsley Publisher), 217–234.

Google Scholar

Veronesi, D. (Ed.). (2017). The Art of Conduction: A Conduction^® Workbook. New York, NY: Karma.

Google Scholar

Veronesi, D. (2019). “Gestures and language in improvised music: The example of Conduction^®,” in Du signe à la performance: La notation, une pensée en mouvement, eds A. Journeau, Z. Chueke, and B. Vassileva (Paris: L'Harmattan), 355–375.

Google Scholar

Ward, A. (2023). The development of a modular accessible musical instrument technology toolkit using action research. Front. Comp. Sci. 5:1113078. doi: 10.3389/fcomp.2023.1113078

Crossref Full Text | Google Scholar

Wheeler, B. L. (2015). Music Therapy Handbook. Guilford Publications.

Google Scholar

Whitehead-Pleaux, A., and Spall, L. (2013). “Innovations in medical music therapy,” in Music Technology in Therapeutic and Health Settings, ed. W. Magee (Jessica Kingsley Publishers), 133–145.

Google Scholar

Wood, S., Verney, R., and Atkinson, J. (2004). “From therapy to community: making music in neurological rehabilitation,” Community Music Therapy, 48–62.

Google Scholar

World Health Organization. (2020). Global Health Estimates 2020: Disease Burden by Cause, Age, Sex, by Country and by Region, 2000–2019. Geneva: WHO.

Google Scholar