“Below 58 BPM,” involving real-time monitoring and self-medication practices in music performance through IoT technology

Merendino, Nicoló; Rodà, Antonio; Masu, Raul

doi:10.3389/fcomp.2024.1187933

ORIGINAL RESEARCH article

Front. Comput. Sci., 21 February 2024
Sec. Human-Media Interaction
Volume 6 - 2024 | https://doi.org/10.3389/fcomp.2024.1187933

“Below 58 BPM,” involving real-time monitoring and self-medication practices in music performance through IoT technology

Nicoló Merendino¹^*

Antonio Rodà¹

Raul Masu²

¹CSC - Department of Information Engineering, University of Padova, Padua, Italy
²Computational Media and Arts, Hong Kong University of Science and Technology (GZ), Guangzhou, China

The project presented in this paper illustrates the design process for the development of an IoT system that monitors a specific bio-metric parameter (heart rate) in real time and provides feedback for an opera singer, as well as adding effects that manipulate the sounds emitted by the body during a self-healing practice. This allows the singer to rest and alternate opera singing techniques (which is very demanding) with other less demanding singing techniques and even a self-healing session in case of necessity during a performance. The case study presented in this paper has been developed with and for Eleonora Amianto, an opera singer who suffered from a carotid aneurysm. We performed an idiographic design process, closely collaborating with Eleonora, and developed a wearable IoT that suited her health and artistic needs. In the design of the system, we explore the intersection between self-healthcare and performative arts, focusing on the use of an Internet of Musical Things (IoMusT) system to implement medical prevention and treatment practices in an art performance. The system is developed using open-source tools, allowing for easy replication and improvement, as well as reducing risks of obsolescence and costs of updating. We complement a formal evaluation session with field notes collected during the design phase. We could observe a positive effect of the system on Eleonora's practice and its potential applications within different performative scenarios.

1 Introduction

Musical performance activities such as singing and instrument playing can be demanding in terms of physical effort to the point that performers might decide to assume substances to repress physical stress manifestations such as trembling and muscle contraction (Patston and Loughlan, 2014). This aspect of music performance renders pursuing an artistic career nearly impossible for people who have been through traumatic medical events or were born with health issues or impediments. Throughout the history of Western music, we can observe that music compositions have been written in different variations according to idiosyncratic characteristics or limits of specific performers, a notable example being Ravel's Piano Concerto for the Left Hand (Davidson, 2010). The work of many researchers, artists, and other professional figures studied how new technologies can, and should, allow persons with restraints or disabilities to successfully participate in music projects (e.g., Abbott, 2007; Irving and Giles, 2011; Cinquin et al., 2019; Förster et al., 2020; Skuse and Knotts, 2020). However, investigating self-treating activities in music performances is still not widely scrutinized and offers a compelling design and research space.

The project presented in this paper focuses on exploring a possible intersection between healthcare and performative arts. By relying on a Internet of Musical Things (IoMusT) (Turchet et al., 2018) system, and by following the principles outlined in the Design Justice vision (Costanza-Chock, 2020), we explore how an artistic project can efficiently implement medical prevention and treatment practices such as heart rate monitoring and meditation, for a singer. The specific goal of the project is the design of an IoT (Berte, 2018) system that can monitor in real time a specific biometric parameter and provide, in case of a (potentially dangerous) stress situation, feedback for the artist as well as the means to actively and safely keep on going with the performance. In particular, the system allows adding sound effects that manipulate the sounds emitted by the body during a self-healing practice. In this way, during the performance, in case of necessity, the singing can be momentarily paused—allowing the singer to rest—and alternated with the affected sounds of the self-healing practice. Additionally, these effects can be used to alternate opera singing techniques (which are very demanding) with other less demanding singing techniques, thus preventing dangerous peaks.

The specific case study presented in this paper illustrates the design of an IoMusT system tailored to the needs of Eleonora Amianto, an opera singer hit by a carotid aneurysm that permanently damaged her caryatid. Following an idiographic design process (Hook et al., 2013), we developed a wearable IoT device that allowed Eleonora to monitor her heartbeat to alternate singing techniques and perform the deep breathing and self-healing sessions she needs to perform within the context of her artistic practice. We named the research project “Below 58 BPM” as 58 BPM ironically refers to an unreal ideal beat rate that Eleonora should maintain for health; a doctor once suggested that as a joke (normal heart rate at rest state: 86). The design process we carried on during the project also aimed to facilitate sharing, replicating and improving the tool among artistic and scientific communities; the research was therefore developed relying on open source tools. Additionally, the final system is complemented with accurate documentation that was uploaded along with the code and the schematics to a public repository and released under GPLv3 license.¹

In order to evaluate the system, we performed an interview with the artist, exploring the impact of the system on her artistic practice and her experience in using the device, which we complemented with field notes collected during the entire process. We also performed a series of measurement sessions of the beat rate related to the artist while performing. The evaluation suggests the tool can be usefully integrated into Eleonora's practice and fruitfully support her pursuit of a career as a singer.

While the research primarily contributes to the socially sustainable and inclusive development of tools in the IoMusT—or more broadly, in the IoT (Arshad et al., 2017), the project also accounts for environmental sustainability by implementing practices that allow for the reduction of waste, contrast obsolescence, and facilitate applicability (thus maximizing the impact of the research). In this sense, the open source approach is particularly relevant as it fosters inclusivity (Hakken et al., 2015; Costanza-Chock, 2020; Bettega et al., 2021), while also supporting sustainability (Blind et al., 2021; Masu et al., 2021; Bettega et al., 2022; Dorigatti and Masu, 2022).

The rest of this article is structured as follows. It starts by exploring definitions of “accessibility” and “inclusion” and provides a description of the state-of-the-art works exploring the intersection between healthcare and performative arts. Next, the article defines the specific goal of the case study. The article continues by describing the idiographic design process that led to the development of the device. In the following section, the evaluation is presented. The paper ends by discussing the project's contribution and delineating possible future research.

2 Background

This background commences by outlining how several aspects of sustainability, including accessibility, inclusivity, and environmental impact, have been explored in the debates focused on technological development. Afterward we funnel to scrutinize sustainability in the domain of performative art.

2.1 Debates on technology sustainability

Various forms of sustainability exist. Social sustainability refers to the impact that technology has on society and how it affects different and specific groups of people (Assefa and Frostell, 2007), such topic it's been discussed through the development of frameworks such the Inclusive Innovation Framework discussed by Nyström and Mustaquim (2018) which is based on the principles defined as simple intuitive use, perceptible information, low physical effort, flexibility in use. In this research, we tackle the issue of social sustainability from the perspective of inclusion and inclusivity, ensuring that the technology we design and develop is accessible to everyone, regardless of their background or abilities. This perspective is particularly relevant from an HCI (Hayes, 2020), as technology can either enhance or limit the experiences of its users. By prioritizing inclusion, we aim to ensure that everyone can benefit from the advancements made in IoT. Social inclusion is also mentioned in the third Sustainable Development Goal developed by the United Nations, which clearly states that ensuring healthy lives and promoting well-being at all ages is an essential aspect of sustainable development (Biermann et al., 2017)

Environmental sustainability refers to technology's impact on the environment and the steps we can take to minimize it. This form of sustainability has been widely discussed in the field of Human-Computer Interaction (HCI) (Mankoff et al., 2007; Knowles et al., 2016; Hansson et al., 2021; Issa and Isaias, 2022) and Internet of Things (IoT) (Laghari et al., 2021). In this research, we primarily address environmental sustainability by focusing on the longevity of the system produced, ensuring that it is designed and built in a way that maximizes its lifespan and reduces the amount of waste generated. This point was highlighted in the design discourse by Chapman years ago (Chapman, 2012), but is particularly relevant in the field of IoT, where a large number of devices are being produced and disposed of each year (Thorp et al., 2022), contributing to the growing issue of electronic waste. This perspective is also aligned with recent literature on music technology development that highlighted how the lifecycle of a new piece of music technology should be carefully considered to minimize the risk of early disposal (Masu et al., 2021; Dorigatti and Masu, 2022).

2.2 Inclusivity and accessibility in the field of music and media arts

Accessibility has become an increasingly important topic for musicians, performers, and organizations. Besides the fact that many music venues now have accessibility features like ramps, elevators, and accessible restrooms to ensure that everyone can participate in music performances, many musicians now use assistive technology in their performances; examples include motion-sensing devices (Camara Machado et al., 2017), adaptive interfaces (Browne, 2016), and alternative controllers (Ward et al., 2019) to be able to perform despite their condition. The roots of this tradition can be traced back to classical music. Indeed, composers have engaged in deep contact with the performers, aiming to help them lower their stress on stage. The example of the aria “In quali eccessi/Mi tradì quell'almaquell'alma ingrata” composed by Wolfgang Amadeus Mozart to value the characteristics of the singer Caterina Cavalieri for the “Don Giovanni” opera performed in Vienna in 1788 (Sartori, 1990) - among the many cases that occurred in the composer's career (Zeiss, 2001)—represents a strong link between the performer's body/voice and the composition practice.

In the field of contemporary arts and music performance, accessibility often refers to the design of performances, events, and spaces that are inclusive and easily functional for persons with a different range of skills and abilities (Förster et al., 2020). The concept of accessibility encompasses different areas, including physical accessibility (Wachs and Kumagai, 1973), cognitive accessibility (Cinquin et al., 2019), and assistive technology (Abbott, 2007). Accessibility is not relevant merely for persons with disabilities but also for anyone who might have difficulties participating in a performance or event. For instance, accessibility is crucial in contemporary dance, especially in those performances that include dancers with disabilities (Irving and Giles, 2011).

The design space defined by the overlapping of Human-Computer Interaction (HCI) and media arts (including music) has recently been characterized by a growing interest in accessibility. For example, the New Interfaces for Musical Expression (NIME) community developed many research projects focused on interfaces targeted for disabled performers (Davanzo and Avanzini, 2020; Skuse and Knotts, 2020, e.g.,). These projects aim to provide solutions that target the compensation of specific persons' characteristics and create opportunities for greater participation and inclusion in music-making.

Apart from the academic and scientific community, various actors and institutions explored inclusivity as a form of social sustainability. A notable example is the Motion Composer company,² which has developed a range of interfaces that allow performers with disabilities to control electronic music through motion-sensing technology. Another example is DRAKE music,³ a UK charity that relies on crafted pieces of music technology to support disabled people to participate in music-making. DRAKE Music developed many projects, including the development of accessible music software and hardware, music therapy, and training for music educators and therapists (Samuels, 2019).

As we have seen, many projects exist; however, the research on inclusivity and accessibility as forms of social sustainability in developing digital technology for artistic practices is far from being saturated. Indeed, this type of research often needs to account for the specific needs of individual persons; this requires continuous updates and the development of new nuances and perspectives. In the paper, we explore a case of a piece of technology designed for an opera singer hit by a carotid aneurysm that permanently damaged her caryatid.

3 Case study and objective

3.1 The case study

The case presented in this paper describes the development of an IoMusT system. The system accounts for IoT as it involves the employ of a wearable device, a client software and a database that stores data. The elements of the system operate with each other through short-range wireless communication (edge computing) (Qiu et al., 2020) and Wifi connectivity (cloud computing) (Sadeeq et al., 2021). The IoT system is tailored around the artistic practice of a specific music performer. In the paper presented, we will focus on describing and analyzing the development of the system's edge computing features. Such parts directly involve the user and system interaction during a musical performance. The system was designed by working in close contact with the Rome-based opera singer Eleonora Amianto who was hit by a carotid aneurysm (de Jong et al., 1989) that permanently damaged her carotid artery. A carotid aneurysm is an abnormal dilation or bulging in one of the carotid arteries, major blood vessels situated on each side of the neck. These arteries are vital as they supply oxygenated blood to the brain, head, and neck regions. The critical concern with carotid aneurysms lies in their potential to rupture. As the aneurysm enlarges, it strains the integrity of the artery walls, increasing the risk of a rupture. If a rupture occurs, it can result in severe bleeding, leading to life-threatening complications. Medical management of carotid aneurysms often involves careful monitoring, especially for smaller aneurysms that may not pose an immediate risk. Treatment approaches can include using medications to control underlying risk factors like high blood pressure or cholesterol.

Eleonora must pay special attention to her lifestyle and avoid stressful situations. To prevent potential crisis, the artist performs deep breathing (Perciavalle et al., 2017) (sessions in which she breaths intensively to decrease her tension) and meditation sessions (Edwards, 2005) to decrease her heartbeat rate. This issue significantly impacted her career as an opera singer, and she found herself in need of adjusting her practice to this new condition.

The system allows Eleonora to integrate a deep breathing session during an artistic performance and, at the same time, upload on a database the data detected by the system to allow the artist to have a database for long-term monitoring of her stress conditions in various performances.

The research team aimed to develop a device allowing Eleonora to monitor her heartbeat while performing. If a stressful situation is detected, the device will provide the means to perform a deep breathing session, which can help decrease the user's heart rate while avoiding disrupting the performance. To achieve this goal, the sounds emitted by the body during the self-healing activity would be digitally processed, thus becoming an integrated artistic component of the performance. In this way, during the performance, in case of necessity the singing can be momentarily paused, allowing Eleonora to rest, thus enabling her to continue her career as a singer while ensuring her health and well-being.

Overall, this research aimed to contribute to the social and environmental sustainability of the IoMusT by promoting the inclusion of self-healing practices in artistic performances. By facilitating the inclusion of self-healing practices in artistic performances, the research aimed to improve the mental and physical well-being of the performer and create a more inclusive and sustainable artistic environment. We want to stress here that this is not a medical project, and the research outcome does not substitute medical treatment. This research aims to support an artist with cardiac issues during a performance.

3.2 First contacts with the artist and agenda definition

The artist Eleonora Amianto was already part of the network of the first author of this research and had previously mentioned her condition during an informal conversation. In one successive moment, she contacted us to find a solution that would allow her to continue her artistic career despite her medical condition. At this moment, she described her condition more formally. Eleonora's condition required her to keep her heartbeat under control while performing on stage. She also clearly stated that she was looking for a solution that would enable her to manage her condition without excessively impacting her ability to perform.

While illustrating her problem and inquiring if a technological solution could help her, Elonora hypothesized the idea of a wearable device that she could use without needing specific knowledge about the technology. However, she was concerned about having a cumbersome element onstage.

The research team asked Eleonora how she usually measured her heartbeat. She explained that she used a commercially available instrument and described some instruments that doctors used when she was hospitalized, which the team identified as an electrode-based ECG measuring tool (Laukkanen and Virtanen, 1998) and a pulse oximeter (Severinghaus, 1993). Based on this description, the research team proposed the development of an IoT device connecting all these devices as a possibility to explore to meet the artist's needs.

During a preliminary brainstorming session, the team watched videos of contemporary vocalists using technology to modulate their voices. This activity led us to explore a solution that could help Eleonora regulate her heartbeat while performing. This decision of the technological setting was tailored around the needs expressed by the artist, together with the need for an agile and graceful element to be worn on stage. In this phase, the idea of adding sound effects to the sound produced during a self-healing session was first advanced and approved by Eleonora.

Throughout the design process, we actively involved Eleonora and carefully avoided a client-customer relationship in favor of a more inclusive workflow. Therefore, the team and Eleonora sat down to create an agenda corresponding to the process that we will outline in the next sections.

3.3 IoT for self-healing practices in a musical performance

A person who has undergone a medical crisis or is affected by chronic diseases such as diabetes often needs to engage in self-healing practices to lead a normal life. However, incorporating these practices into a musical performance without disrupting it can be challenging. In the project, we bridged biometric monitoring and computer music technology to create a system that allows musicians to perform self-healing practices during a musical performance. By combining these two fields, we explore how IoT technology can support Eleonora's artistic practice. We decided to rely on IoT technology, which is robust in data transmission and can embed and combine technology in various forms.

One of the critical advantages of IoT is its ease of management, especially in stage settings where wires and cables can be a hindrance. The Internet of Things (IoT) represents a network interlinking physical devices, home appliances, vehicles, and assorted entities embedded with electronics, software, sensors, and connectivity infrastructure. This interconnected framework facilitates data exchange and communication among these objects. The pervasive integration of IoT technology across diverse domains such as healthcare, transportation, and manufacturing stems from its capacity to seamlessly gather and analyze real-time data, automate operations, and enhance overall efficacy. IoT technologies provide the opportunity to connect a person's body to other devices, such as sound stations, offering unprecedented control and interaction between the performer and the performance, which can be pivotal in self-healing during a music performance. Another key benefit of IoT technology is the possibility of connecting the user's body to an online database. This connection can collect real-time information, providing valuable data for future analysis. The data collected can be used to understand performance patterns, identify improvement areas, and develop new solutions that enhance the performance experience. Finally, IoT technology can provide a musician with an easy-to-use tool.

In the specific case presented in this paper, the ability to monitor biometric data during a musical performance could help Eleonora to monitor her health in real-time, allowing her to make necessary adjustments to avoid negative health consequences. With the help of computer music technology, the system can also provide the means to perform in a less stressful way for the body.

4 Design and development

4.1 Methodology and design phases

The research presented in this paper follows a research-through-design approach (Zimmerman et al., 2007; Gaver, 2012). In particular, we relied on ideographic design, which “involved a close and dialogical engagement with the practices and experiences of an individual live performer” (Hook et al., 2013). Ideographic design is mutated from autobiographical design, which is a form of “design research drawing on extensive, genuine usage by those creating or building a system” (Neustaedter and Sengers, 2012).

Following this methodology, we were able to work closely with Eleonora to design and prototype the device. This approach allowed us to explore various aspects of the project and gather valuable feedback throughout the development process. In a recent podcast, Costanza-Chock, the leading figure of Design Justice (Costanza-Chock, 2020) argued that designing things “for just one person” could be beneficial to tailor specific needs.⁴

The entire design process was characterized by continuous interactions between the research team and the artist and was structured as follows. We commenced by designing suitable IoMusT hardware and software; with the artist, we defined which IoT-capable electronic components were the best to accomplish the tasks demanded. In this preliminary phase, we performed a literature review to identify suitable components. Afterwards, we performed a series of coding sessions focused on programming and optimizing the efficiency of the primary device. We prototyped the breadboards and the code. After testing it with the artist and collecting feedback, we revised our design and proceeded to design and develop the Printed Circuit Board (PCB).

Once the code and the circuits were defined, we proceeded with wearable interface design. To this end, we underwent a process articulated in a series of design sessions where the artist and the research team developed the wearable device together, taking into consideration aspects such as ergonomics, environmental sustainability of the materials used, and usability of the system. This process was articulated in the following steps: designing the initial prototype, CAD sessions, and digital fabrication sessions.

Finally, we focused on a client software that computes the processing of sounds produced by Elonora in a healing session based on data received from the IoT device. To design this software, we engaged in artistic and computer music research to further understand the possibilities and limitations of the device in a musical context. After finalizing the system, we developed a UX design flowchart (Gruen et al., 2002; Hassenzahl, 2008) as a technical instrument to pin down and visualize all the aspects of the interaction with the system and developed detailed documentation to facilitate applicability and maintenance.

4.2 IoMusT hardware and software

4.2.1 Research on IoT hardware

As a first step, we selected the hardware components that better fit the necessity of Eleonora. The research team explored different microcontrollers (Cheour et al., 2020) and decided to use an esp32 microcontroller due to its low price, high performance, and availability combined with IoT capabilities such as Wifi Connection and Bluetooth antenna. To be more specific, the team used a Lolin32 lite board which is a microcontroller board that integrates a battery charger module and other potentially useful implementations (Darmawan et al., 2022); this choice was taken in order to prevent the obsolescence of the system as well as giving a chance person with limited technical knowledge to easily update or hack the device if needed. The artist also participated in choosing input devices and ultimately opted for a photoplethysmogram-based based pulse sensor to measure her heartbeat (Rahman et al., 2021) and basic potentiometers for additional input. As an output device, Eleonora favored the proposal of haptic feedback using a vibration motor because haptic technology allowed her to receive non-intrusive feedback that does not introduce any sensorial element detrimental to the music performance. Additionally, we considered visual feedback using LEDs for communication with the audience, which specifically consisted in setting up an LED that turns on when a pulse is detected by the device and communicating the state of the device (on/off), plus the connection with the the system. This point is further explored in the artistic and computer music research, which represents the final stage of the design process (Figure 1).

Figure 1

Figure 1. Picture of the selected components: (a) Lolin32 lite; (b) LED and resistor; (c) Potentiometers, (d) pulse sensor; (e) vibration motor module; and (f) toggle switch.

4.2.2 Breadboard and code prototyping

Once the components were selected, the team proceeded to develop a prototype (Figure 2). The team discussed the choice of components with the artist and then assembled a prototype using a breadboard. They coded the device's behavior using the Arduino code language, focusing on detecting the performer's heartbeat, sending feedback to the artist, and activating the possibility of sending MIDI CC messages to sound processing software. The pulse sensor sends an analog signal to the ESP32, which is processed by the board. The team coded a specific function to calculate the BPM rate based on the heartbeat detected by the pulse sensor.

Figure 2

Figure 2. Image of the assembled circuit: (a) breadboard prototype; (b) schematic.

The team also worked on the MIDI CC messages sent by the device and the behavior of the vibration motor that was part of the prototype. Initially, the team created an “alarm mode” that would activate the vibration in case of a high heart rate, but the artist rejected this as it triggered anxiety. Therefore, the team changed the device's behavior to associate the motor's vibration with the artist's beats, providing constant tactile feedback. The tactile feedback influences the artist's choices when performing as she gains awareness of her biometric parameters. According to this decision, the team also made the MIDI signal always available for Eleonora to interact with the system whenever she wanted.

We named this interaction modality constant feedback mode: the artist constantly receives tactile feedback of her heartbeat. Thanks to the potentiometers, she can always decide to apply effects to her voice. The visual feedback includes three LED lights that show the single heartbeats, the connection of the BLE device to a computer, and the On state of the whole device. The team also added an On/Off switch to the device so that the battery's lifespan can be optimized, and the artist can understand immediately whether the device is on.

4.2.3 Final PCB design and optimization

After a sufficient level of stability was reached, the team moved on to the optimization phase of the circuit. The goal was to make the circuit as compact as possible while also incorporating sustainability strategies to reduce its environmental impact. The team used the open-source software, KiCAD,⁵ to design the circuit, which is freely available and can prevent obsolescence (on why open-source prevents obsolescence see Blind et al. (2021)). All components were hand soldered, allowing for future manual repair if necessary (thus prolonging the tool's life cycle (Masu et al., 2021)). The main board, the lolin32 board, is attached to the device through a connector, making it removable and reusable for other projects, reducing the need to purchase new boards. This process resulted in a compact and stable Printed Circuit Board (PCB) to be used in the next phase of the prototype's development. The production of the circuit was commissioned to a specialized manufacturer.

4.3 Wearable interface design

4.3.1 Initial interface prototyping

The following stage in the development of the wearable device involved encasing the circuit into a stable and usable form. To ensure the device met the ergonomics, interaction, and artistic expression criteria, the team organized the design sessions in constant contact with the artist to determine the design based on her needs. The artist expressed the desire to perceive the device as an integrated part of herself rather than a separate entity. During a series of design and sketching sessions, Eleonora suggested using the cyberpunk subculture (Csicsery-Ronay, 1988) as an aesthetic reference regarding the relationship between technology and the human body and some media art pieces. The cyberpunk aesthetic is a visual and thematic style deeply rooted in science fiction and futurism. It's characterized by a blend of high-tech and low-life elements, often portraying a dystopian or near-future world where advanced technology coexists with societal decay and urban grit.

In order to test the ergonomics and shape of the device, the team used fast prototyping techniques such as paper prototyping and paper crafting (Figure 3), combining them with the previously developed circuit. At the same time, an inquiry into the aesthetic aspects of the device was developed with a particular focus on how contemporary media artists have explored the relationship between technology and the human body. In particular, we were inspired by Dominic Wilcox's “Switch,” where the implementation of input devices is directly into the human body, and by Anselmo Tumpic's “Beat,” which includes a well-visible heart monitoring device in the performance (Finessi, 2012).

Figure 3

Figure 3. Sketches and paper prototypes of the wearable interface. (a) overview; (b) front pattern; (c) paper prototype of front interface; (d) sketch of the assembly; (e) back pattern sketch; (f) front pattern advanced paper prototype.

The team developed a collar-shaped structure housing IoT components, driven by Eleonora's preference for a device near her vocal area. This design allowed her to manipulate the interface comfortably, giving the sensation of adjusting her body when using the device. The visual feedback, represented by blinking LEDs, reflected Eleonora's body activity, emphasizing the deep connection between the technology and the performer. Alongside the collar, a finger element was designed to secure the pulse sensor, ensuring effective heart rate monitoring.

4.3.2 CAD sessions

We developed all the necessary components after defining the overall design in the prototyping phase. This included two laser-cut textile elements and a set of five plastic shells that would house the primary circuit, The device includes LED lights, a vibration motor, an on/off switch, and four touch-friendly potentiometer knobs for user convenience without visual interaction. The microcontroller's shell was purposefully designed for easy access to the ESP32 board's USB port, simplifying testing and enabling future code updates.

The team utilized the FreeCAD⁶—an open-source 3D modeling software to design the elements. The shells were designed to be sewn onto the textile elements and were equipped with a pattern of holes for this purpose.

The team then designed the collar's textile pattern, resulting in two simple stripe elements with a front and back appendix to hold the shells containing the electronic components. They used Inkscape⁷ software to optimize the textile pattern for laser cutting. The result of this process included the CAD files and STL files for 3D printing and SVG files for laser cutting (Figure 4).

Figure 4

Figure 4. Screenshots of the CAD models developed: (a) PCB design in KiCAD; (b) pattern design in Inkscape; (c) finger element 3D modeled in FreeCAD; (d) vibration motor case 3D modeled in FreeCAD; (e) LED case 3D modeled in FreeCAD; (f) Esp32 board casing 3D modeled in FreeCAD.

4.3.3 Digital fabrication sessions

In the final step of the interface development, we brought the previously designed components to life through digital fabrication. The team relied on Fused Filament Fabrication (FFF) technology to produce the STL files and 3D printed them using PLA as the material (Figure 5). The textile elements were created through laser cutting technology using a felt sheet (Figure 6).

Figure 5

Figure 5. 3D printed elements: (a) overview and PCB; (b) switch case; (c) potentiometers case; (d) knobs; (e) LEDs case; (f) vibration motor case; (g) Esp32 board case; (h) finger element.

Figure 6

Figure 6. Wiring and assembly process: (A) back view; (B) front view.

The team then assembled the device by hand, soldering the electronic components and sewing the plastic components and Velcro patches onto the felt pattern. After a thorough testing process, the device was finally complete and ready to be used.

The wearable element is a collar large enough to be worn on top of a cotton or wool scarf for added comfort and adaptability and to prevent the collar from absorbing the performer's sweat. The collar can be opened and closed using Velcro, making it convenient for the artist to wear. The collar is connected to the pulse sensor casing that the user can easily wear using a Velcro stripe (Figure 7). The wearable interface presents in the front part the knobs, the LEDS, and the vibration motor (Figure 8), while in the back, there is the toggle switch and the main board that can be connected to a USB charger as well as to a computer so that it is possible to load code if needed (Figure 9).

Figure 7

Figure 7. Overview of the wearable interface worn by the artist: (a) front of the device axonometry; (b) back of the device axonometry; (c) finger element axonometry.

Figure 8

Figure 8. Details of the front of the wearable interface and finger element: (a) LED case; (b) finger element; (c) vibration motor; (d) knobs.

Figure 9

Figure 9. Details of the back of the interface: (a) toggle switch; (b) back LEDs position; (c) battery and esp32 location (Microcontroller, BLE, and WiFi) USB connector for charge and code upload.

After building the components, the artist underwent a brief testing session and expressed her satisfaction with the ergonomics and overall appearance of the device.

4.4 Designing the client software: artistic and computer music research

In the context of IoT, “client software” refers to the applications or programs running on devices that interact with or consume services provided by servers or other devices in the IoT network (Sabella et al., 2019). In our case, once the hardware and the interface were developed, the focus shifted to a client software that added effects to the sounds emitted by the body during a self-healing practice, specifically, the sounds of Eleonora's breaths captured by the microphone. As these effects become an integrated sonic element of the performance, the development of this component accounts for artistic and computer music aspects of the project. The core idea was to allow the artist to access a wide range of artistic expression possibilities while reducing the demand for body fatigue. Indeed, even while performing self-healing techniques such as deep breathing, the artist's body still emits sounds that can be processed. By offering this possibility, in case of excessive stress, Eleonora can use the sounds emitted while performing a deep breathing session and singing using techniques that are less demanding in terms of body fatigue. The majority of these sounds are related to breathing and voice; as such, for the sake of readability, for the rest of this section, we will use the lemma voice manipulation as a synecdoche representing all the sounds from the body.

At this stage, the goal was to explore the use of the system in a real performative scenario. These sessions were meant to serve as an exploration of the possibilities of voice distortion. In addressing the sounds emanating from Eleonora during her self-healing practices; initially, methods like time-based loops appeared to be the most viable solution. However, following a series of discussions with the artist, the project shifted its focus toward prioritizing and accentuating the sounds produced by the artist during her breaths. This shift aimed to underscore the significance of the artist's breathing as an active and integral component of the performance. To this end, a series of sessions where the team presented the artist with various examples of voice modulation and distortion projects were organized. We sourced these examples from artistic projects and pop culture artifacts such as sci-fi movies and video games (Fox, 2021). The outcomes of the design process of the wearable element drove this first process. The book “Expanding the Vocalist's Role Through the Use of Live Electronics in Real-Time Improvisation” (Åse, 2014) was particularly influential as it offers many examples. Additionally, we scrutinized the work by Nowitz (2008) and Baumann (2023) as representative of voice manipulation practice paired with tangible interfaces, and analyzed how voice manipulation has been used in contemporary pop music, notably the auto-tuning as an iconic characteristic of Trap music (Provenzano, 2019). In response to the materials provided by the research team, the artist selected a series of features that were more fitting with her artistic practices. The research team used such selection to develop the effects.

With the inspiration from these sessions, the team then began developing a Pure Data (PD)⁸ patch that would couple the wearable device as a client receiving data. The result of this research was the Voice Distortion Patch. This patch allows the artist to control various aspects of voice distortion using the knobs on the wearable device. These controls included a pitch shifter, a single-sideband modulation, a reverb and a comb filter. Inspired by the Dehumanizer VST instrument by Krotos Company, we developed the PD patch by tweaking an existing open-source PD patch designed by Dundee Games.

The Voice Distortion Patch provides the artist with a comprehensive IoT system networking a tangible object (the wearable interface) with a data elaboration client software receiving data from the interface. This software is multi-platform as it can run in any OS and could also be embedded in dedicated chip computers such as a Raspberry (Brock et al., 2013) or a Bela platform (McPherson, 2017), In the presented setup, we used a laptop running Windows 11 as it was the most comfortable solution to develop and tweak the system at that stage of development. Through the system, Eleonora can control the various aspects of voice distortion, allowing her to achieve her artistic vision. The combination of the wearable interface and the Voice Distortion Patch allowed the team to delve into the world of computer music and explore the potential of the TUI as a creative tool that explores the possibility of enriching any sound coming from the performer's mouth, including deep breaths and tongue's clicks.

4.5 Architecture and user experience flow chart

The final outcome of the design process as a collaboration between the team and the artist is a functional Edge Computing (Qiu et al., 2020) IoT musical system (whose schematics and code are available in a repository.⁹ The diagram shown represents the system's IoT architecture (Jabraeil Jamali et al., 2020) in all its parts (Figure 10), listed by tasks performed.

Figure 10

Figure 10. Architecture of the edge computing IoT system.

Finally, we developed a complete UX flowchart (Figure 11) summarizing the possible use condition. Aiming to nail down the flow details, the team carefully went through all the steps of the experience of using the device during a specific session with Eleonora. Once the device is turned on, the user begins to experience augmented tactile feedback of their own heartbeat. The four knobs are constantly connected to the sound effects included in the Pure Data patch. The user starts singing generally without using the effects. When the feedback of the heartbeat indicates signs of stress, the artist can stop using lyrical techniques and perform a deep breathing session. We also use the term “relaxed singing techniques” to describe moments where Eleonora performed a deep breathing session without stopping singing. The artist can then use the knobs to change the sound of their voice.

Figure 11

Figure 11. UX flowchart of the project.

This UX design flowchart showcases the seamless integration of the IoT device networked with the Pure Data patch to create an augmented vocal experience for the artist, in pair with a simplified flowchart that explains the system in Figure 12. The user controls the sound effects and can make adjustments based on their physiological response, making the experience both unique and personalized. The combination of deep breathing and the ability to manipulate the sound of their voice creates an immersive experience that enhances the connection between the body and the voice.

Figure 12

Figure 12. Simplified flow of the system in use.

4.6 Documentation

The research project has resulted in the creation of a complete system that is documented and shared through a public repository.¹⁰ This repository aims to make it possible for anyone interested to reproduce or build upon the system that has been developed. The repository includes all the necessary information and files required for reproduction, including STL and PDF files for the fabrication of the object, including the interface and circuit schematic and GBR files. This documentation facilitates recreating the system and using it in other projects or research.

5 Evaluation

To gather a comprehensive understanding of our project, we conducted both quantitative and qualitative evaluations.

The quantitative evaluation of the system consisted of collecting measures of the heartbeat rate of the performer while using the prototype and comparing it against a regular singing session. The purpose of this session is to evaluate whether the device affected the performer's physical response and to determine if the device impacted the performer's level of engagement with the music.

The quantitative evaluation took place in the Eleonora private house, at a location and time agreed upon with the artist to provide a comfortable and safe context for the measurement. While we acknowledge that this does not entirely recreate the stress of a real performance, we wanted to avoid putting Eleonora under pressure at this stage of the research. The pulse sensor used for the measurements includes ambient light sensors APDS-9008, green super bright LED AM2520ZGC09, and an operational amplifier MCP 6001, and is designed to emit green light to the fingertip or earlobe and sense the reflected green light with the sensor APDS-9008. When blood pumps through tissues, the converted voltage output will change due to green light absorption, and the voltage variation then passes to the equipped filter and amplifier MCP6001. The amplified signal from op-amp MCP6001 can achieve a voltage output between 1.8 V and 6V, with a 16bit resolution. The analog values are then assigned to a value between 0 and 4,095 by the ESP32. To calculate the BPM and manage the data detected by the ESP32, (provided in the documentation similar to what Wang and Jang (2019) developed, although maintaining the analog signal provided by the sensor) we included in our script a function that calculates how many times the value reaches a peak associable to a heartbeat, within a window of 10 s. Using this method, we filtered the data internally and directly exported the BPM value that the ESP32 calculates every 10 s. During the evaluation session, we asked Eleonora to perform a 1-min piece, “Aria di Juni,” composed by Eleonora in collaboration with the composer Jue Wang, as part of the sci-fi opera “Falena.” We asked her to perform the piece ten times, five times with our system–condition A—and five times without it condition B (Figure 13). The two conditions were alternated with at least a three-minute break in between to allow the heart rate value to drop below the normal rest state of circa 86 BMP. A calibration phase preceded the evaluation to set the pulse sensor. The Calibration process consists in uploading a script to the esp 32, in which the system only blinks one of the LEDS when a heartbeat is detected. by plotting the real-time data flow detected by the pulse sensor and observing the actual pulse of the performer's heart through contact or through the use of a stethoscope it is possible. to set the threshold value needed to associate a peak in the data flow to a pulse. In condition B (without the system), Eleonora performed the piece as she would have normally done, using lyrical singing techniques. In condition A (with the system), Eleonora started her performance using her usual technique (as in condition B), but then, she started to use the wearable interface to apply effects to her voice and changed the singing technique, never reaching a critical stress condition. It is also relevant to report that during the sessions, Eleonora never reached a stress peak so high to induce her to perform a proper self-healing exercise. This is probably due to the settings, which only partially recreate the stress of a real performance.

Figure 13

Figure 13. Line charts showing the measurement of each measurement session conducted.

The data was recorded using the serial port of the ESP32 board (described in section 3) and the software CoolTerm.¹¹ The code was optimized to provide a two-column CSV file that shows the number of seconds passed and the beats per minute (BPM) detected by the pulse sensor. The CSV files exported from the serial port of the ESP32 were later processed using Python and the “Pandas” library for analysis and the “Matplotlib” library for visualization and edited with Inkscape for graphic design purposes (Figure 13).

We gathered qualitative data by collecting field notes throughout the entire process and interviewing Eleonora after the final evaluation. This qualitative data provided a deeper understanding of the device's performance from Eleonora's perspective. The structured interview we performed at the end of the evaluation session process aimed at gathering Eleonora's general impressions on the project, the design of the object, the effects developed to interact with her voice, the impact of the device on her artistic practice, and the artist's intentions and opinions regarding possible future developments of the device. The field notes are a compound of information that includes notes taken after informal conversations that happened during the development of the system and relevant observations. This qualitative data was analyzed through thematic analysis (Kiger and Varpio, 2020). Thematic analysis is a standard method in qualitative research to identify, analyze, and interpret patterns or themes within a qualitative dataset. It involves systematically organizing and coding data to uncover recurring ideas, concepts, or topics that help in understanding the underlying meanings or experiences expressed by participants. This method provides a structured approach to analyzing qualitative data and extracting meaningful themes or patterns within the collected information. Following an inductive thematic analysis, we initially coded our notes and the interview using a bottom-up approach. Afterward, we recursively clustered the codes into themes and subthemes; this process was double-checked by two authors.

5.1 Quantitative results

In the quantitative analysis, we compared the means of the signals and the peaks (as the point of maximum effort) between the two conditions—following a method standard in repealed measure studies. As the distributions of peaks and the means did not meet the Assumption of Normality, we analyzed the peaks as nonparametric data, and we evaluated the strength of the difference of the two datasets against the null hypothesis using a Mann'Whitney U test.

The median of themeans entire signals without the system is 159 bpm and the mean is 155.92 bpm, while the median of the means with the system is 150 bpm and the mean is 130 bpm (reduction of 8.5%); the distributions in the two groups of peaks did not differ significantly (p-value = 0.35197) (the test did not reject the null hypothesis).

The median of the heart-rate peaks is 142 bpm, and the mean is 142.2 bpm, while the median of the heart-rate peak with the system is 130 bpm and the mean is 131.4 bpm; the distributions in the two groups of peaks differed significantly (p-value = 0.00604) (the test rejected the null hypothesis).

Our results might suggest that while the system does not have a positive impact on the overall heart-rate level, it has a positive impact in reducing the heart-rate peaks. However, given the small sample size, these results should not be taken as scientific definitive proof of the validity of the system, but rather as a complementary support of the qualitative analysis that we outline below.

5.2 Qualitative results

Overall Eleonora reported that the system made her feel safe and more comfortable while performing. The thematic analysis produced five themes with several sub-themes highlighting specific elements of the project. We suggest that these themes can be used to design similar systems.

5.2.1 Theme 1: monitor state for prevention

The first theme emerged, highlighting the importance of preventing critical conditions rather than managing them.

Subtheme: prevention of critical condition: Eleonora highlighted the importance of preventing critical conditions, with her goal being to prevent a critical situation from happening rather than fixing it while performing. She reported that having continuous haptic feedback helped prevent critical situations as the artist avoided excessive efforts thanks to the tactile feedback provided by the device.

Subtheme: constant awareness through augmentation of biometric data: Eleonora was pleased with the feeling of constant awareness provided by the haptic feedback. She explicitly expressed that she preferred this idea to an alarm system that triggers in case of a critical situation. This feature also helped Eleonora to relax, as it was perceived as a sort of "massage."

5.2.2 Theme 2: important of embodiment

The second theme is related to the importance of a good embodied relation with the instrument, which has to be fruitfully integrated into artistic practice. This theme has two subthemes.

Subtheme: general ergonomics and comfort in wearing the element: Eleonora emphasized the importance of general ergonomics and comfort in wearing the device. The comfort of wearing the device helped her to familiarize herself with it and smoothly integrate it into her artistic practice.

Subtheme: perceived embodiment of the device: Eleonora placed great emphasis on the embodied feeling with the IoT device, stating that it is crucial to perceive it as an interaction with her body rather than with an external entity. The term “embodiment” here accounts for the definition of designed artifacts as extensions of human cognition - as scaffolding onto which we delegate parts of our cognitive processes (Magnusson, 2009)

5.2.3 Theme 3: aesthetics considerations

Three main sub-themes are related to the aesthetic components of the system, both audio and visual.

Subtheme: total control of the sonic behavior of the device: Eleonora stressed the importance of total control over the device's behavior. She felt the need to control any sound effect that the system provided, except for the monitoring system, which operated automatically. This is motivated by the fact that the device needs to be integrated into her artistic practice.

Subtheme: exposing the design of the object: Eleonora expressed a strong interest in the design of the object, insisting on keeping the LEDs even though she could not see them while performing. She refused the idea of an “invisible object” and instead preferred to have a clearly visible object assuming a clear role in the performance.

Subtheme: low importance fidelity to the previous artistic practice: Eleonora admitted that the device does not give her back a chance to be a lyrical singer as she was before the accident. However, her overall perception of the project was extremely positive, and she expressed a desire to continue the development of the system. Fidelity in reconstructing her previous artistic practice was less important than the self-healing system's positive impact on her well-being.

5.2.4 Theme 4: sustainability of the system

The fourth theme is related to the importance of having a reliable system in future collaborations and projects. This theme is composed of two subthemes.

Subtheme: open and flexible system: Eleonora expressed concern about the system's compatibility with various software and the ease of starting collaborations with producers from her network.

Subtheme: affordability: Although the team was always clear regarding the fact that the artist would not be charged for any of the economic costs that the project required, Eleonora often expressed interest and concern regarding the cost of the system and its singular parts, envisioning the future need to replace one or more parts that could break.

6 Discussion

The system described in this paper is an IoT-based system aimed at enabling an artist with a medical condition to pursue her musical career. In this section, we discuss this IoMusT system in relation to the existing debates on inclusively, social sustainability, and environmental sustainability. On the one hand, we hope this reflection can support the development of similar music-oriented projects. On the other hand, we hope to contribute to environmentally and socially sustainable discourse more broadly in the IoT field. We also want to emphasize the significance of the ideographic design process within our methodology. This approach enabled us to capture and analyze data encompassing various aspects of the system. It would have likely been challenging to uncover these insights solely by evaluating the final outcomes without documenting the process that influenced the numerous decisions made along the way.

6.1 Inclusivity and embodiement

The system we presented is tailored to the idiosyncratic need of one singer, rejects a one solution fit all approach, and embraces a vision that accounts for individual needs and differences. Such a perspective aligns with the Design Justice vision (Costanza-Chock, 2020). Design Justice epitomizes a multifaceted movement and conceptual framework nested within the design milieu, meticulously directed at redressing societal inequalities while propounding ethically sound and inclusive design methodologies. Its foundational intent is to interrogate prevailing power constructs, advocate for marginalized cohorts, and accord paramount significance to their perspectives and requisites within the design continuum, which suggests a working model that targets the specific needs of persons, groups, or small communities.

The research joins the debate on the accessibility of instruments and interfaces for musical production (Frid, 2020), extending the focus to the specific case of a singer hit by a carotid aneurysm. The article also joins the broader topic of accessibility in the musical industry (Jarvis Holland et al., 2020). Eisentraut (2012). In particular, (although tailored around a specific case study) this work extends the literature discussing music professionals' stress (Iñesta et al., 2008), and the impact of music on heart rate variability (Vickhoff et al., 2013) by presenting a system that helps release this stress and control the heart rate dustíring a musical performance.

The design of a system for constant feedback during a performance and the positive comments about this characteristic highlights the importance of considering the body in its entirety during the design of such types of technologies. Indeed, to fully engage specific needs and conditions derived from a physical disability, we argue that it is essential to consider bodies and differences among bodies. For this reason, we opted for a wearable device, which allows for an embodied and tangible relationship between the singer and the system. The perceived embodiment of the device, together with the ergonomics and design, are parameters that connect the project with the debate on prosthetics and how they are perceived by the user (Fox, 2021).

This perspective is also aligned with the soma design approach. The SOMA design approach prioritizes social and human aspects in the design process, creating spaces catering to cultural, social, and psychological needs. It fosters community interaction, well-being, and a sense of belonging within the built environment. SOMA Design practices include “designing with and through kinesthetic experience” aiming at bridging the body-mind dichotomy “by attending to our inner universe and dissolving or traversing dichotomies between inside and outside […] body and technology” (Höök et al., 2021). In the case study we developed, we could observe an entangled relationship (Frauenberger, 2019) between Eleonora and the technology. Indeed, the coupling of Eleonora with the system creates a complex performance ecology (Gurevich and Treviño, 2007; Masu et al., 2019), where an embodied relationship allows for a new artistic “meaning” of the healing practices to emerge. On the one hand, the system facilitates a higher level of awareness in relation to the heart rate; on the other hand, it supports the inclusion of healing practices in the aesthetic of a performance. Such an aesthetic reflection is derived from a cyber-punk and the sci-fiction characters that share biological and computational components: the cyborgs. In her “Cyborg Manifesto,” Haraway rejects the rigid boundaries that clearly discriminate humans from machines, embracing a more holistic post-human vision (Haraway, 2013). Referring to the cyborg aesthetic in the project thus supported us in creating the situated and fluid configurations that allowed Eleonora to engage with the system, both in terms of embodied relationship and in terms of aesthetic integration. The collar design embeds the metaphor of a singer changing her voice and breath by tweaking knobs embedded in her neck that, together with the LEDs and the switch, do embrace the human-machine aesthetic often present in cyberpunk narratives. Such configuration is further strengthened by the IoT nature of the project, which establishes a connection between Eleonora's body and the internet.

6.2 Social sustainability of a IoT system

Design for a specific health problem supporting inclusively and facilitating self-healing practices is in itself a formsocial sustainability. However, we also accounted for the future uses and the long-term sustainability of the system. A central aspect of inclusive design (Costanza-Chock, 2020), post-human and cyborg discourse (Haraway, 2013), and of the Entanglement HCI (Frauenberger, 2019) is the struggle to avoid considering the interactions between the limited boundaries of a coupled human-machine relationship. On the contrary, acknowledging and addressing them within broader societal and ethical frameworks emerge as imperative facets in navigating the complexities inherent in human-computer interaction, fostering more comprehensive, and responsible technological development. Therefore, we accounted for the uses of the system in Eleonora's real-life scenarios, not only within the situated and temporally defined performance experience but considering future uses, repairs, and eventual related costs accounting for the instance artist's concern about the cost of the system. This perspective fostered us to reflect upon accessibility and inclusively from the perspective of economic affordability, aiming to find a solution for Eleonora to engage in a prolonged relationship with the system or allow other possible artists to use and adapt it to their cases while minimizing additional costs. This reasoning lies behind the decision to develop the entire system using open-source tools (Merendino and Rodà, 2021), and release the project under a GPLv3 license combined with abundant and extensive documentation. Furthermore, by making all the parts of the system available, some portions of the project can benefit other projects that imply the use of haptic feedback.

6.3 Social sustainability calls for environmental sustainability

The artist expressed strong concern about the topic of durability and the possibility of further developing the device in collaboration with professionals outside the research team. In the previous paragraph, we discussed this point as a social and economic issue (minimize re-paring costs); however, it also impacts one of the core aspects of environmental sustainability, as avoiding obsolescence and promoting reusing and updating is one of the best strategies to reduce waste (Chapman, 2012; Blind et al., 2021).

The device has been designed in line with well-known guidelines to reduce material waste, avoid sharp and fragile edges, and reduce the risk of easy disposal (?). The project was fabricated using widely available digital fabrication techniques (Scheeren and Sperling, 2020; Soomro et al., 2022), using recycled PLA and felt two environmentally friendly materials.

Starting from the artist's demand for a flexible and accessible system, particular attention was given to open-source strategies that make the project less subject to obsolescence and promote longevity [which is a well-investigated problem in the music technology debate (Marquez-Borbon and Martinez-Avila, 2018)]. This aspect is being explored by the scientific community as well as the European Commission, which reported how an open-source approach contributes to the sustainability of a project (European Commission. Directorate General for Communications Networks, Content and Technology). Other communities of researchers have been working on the potential of FLOSS software and how it can contribute to the advancement of this debate (Blind et al., 2021).

To summarize, the solutions applied in the IoMusT project are not novel in themselves, but the study shows a case study where these strategies have been applied and tested with positive feedback from the user. Communities such as NIME, UBIMUS (Schiavoni et al., 2019), and Audiomostly, which have recently shown awareness of environmental sustainability, can find interesting insights on how an environmentally friendly approach can be implemented in their practice when developing IoT systems.

7 Conclusion

In this paper, we described an ideographic design process that led to the development of a custom IoT-based DMI that provides feedback to a singer to implement self-healing practices during a music performance.

We evaluated the resulting instrument with the singer, collecting qualitative and quantitative data. The data analysis suggested this instrument's potential to reduce cardiac stress successfully. Additionally, we extrapolated a set of themes about the system's design that might be of use for future similar projects.

This paper's main contribution to the field of inclusive music technology design is a system - composed of a wearable device coupled with client software - supporting singers with cardiac issues. The documentation of the device and the client software are freely available from a repository. The software is open source, and the documentation allows to recreate the device with a minimum cost since we accounted for this other aspect of inclusivity.

As a secondary contribution, this paper also provides a clear example of how environmentally sustainable practices can be integrated into the development of an IoMusT system (Gabrielli and Turchet, 2022). We hope that this paper can serve as a practical demonstration of the commitment to sustainability and provide inspiration and guidance to other practitioners or researchers looking to follow in the same direction. As this environmental sustainability angle is coupled with a replicable and budget-friendly design, this system serves as a blueprint for future developments and enables artists and researchers to build upon and improve upon the existing work.

Overall, the project accounts for both social and environmental sustainability instances, aiming to support a convergent direction for these two types of sustainability in relation to IoT applied to musical performance. By doing so, the research helps to drive the transition toward a more socially sustainable future for the IoT.

However, there is still ample room for improvement in the system. For instance, the system was tested in a controlled environment, which differs from a real performance, and additional research is needed to prove its effectiveness in the actual field. In future works, we plan to create a database collecting data over a more extended period of use to analyze its effectiveness in various conditions and settings. Other improvements can be made to the design of the system itself. For example, the efficiency of the code can be improved by implementing features such as a deep sleep mode to save energy.

Data availability statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found in the article/supplementary material.

Ethics statement

Ethical approval was not required for the studies involving humans because the study does not require approval from ethics committee. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author contributions

NM: development of the system and relationship with the artist, data collection, and analysis. AR: support in the development of the system and for the measurements sessions. RM: contextualized the study within the debates on embodiment and sustainability, and contributed to the thematic analysis. All authors contributed to the article and approved the submitted version.

Funding

This work was partially supported by the project Creative Recommendations to Avoid Unfair Bottlenecks of the Dept of Information Engineering of the University of Padova. Support was also provided by a PON scholarship (grant number DOT1487343-5) from the Italian Ministry of Research.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Author disclaimer

To enhance the quality of our manuscript, we employed artificial intelligence algorithms for minor tasks, such as grammar and spelling checks. These technological aids were utilized strictly for the purpose of improving the readability of the paper and did not influence the research findings or conclusions.