Designing accessible musical instruments by addressing musician-instrument relationships

1. Introduction

Designers of Digital Musical Interfaces (DMI) and Accessible Musical Instruments (AMI)¹ have provided several frameworks to assist with the design of these categories of instruments (Jordà, 2004; Johnston et al., 2008; Overholt, 2009; Morreale et al., 2014). These frameworks have proved helpful in defining the musician's goals, physical abilities, and environments and matching them to existing technology. However, these frameworks lack propositions on how to inquire into and design for the intimacy of the musician-instrument relationship. As we argue in this paper, we hold that this enquiry is essential in designing bespoke DMIs, especially with disabled musicians.

Research on musician-instrument relationships has been studied in relation to non-disabled musicians connecting with conventional instruments (Nijs et al., 2009; Simoens and Tervaniemi, 2013; Waters, 2021), but specific investigations with disabled musicians and DMIs/AMIs are currently missing. Also, current DMI/AMI frameworks have been primarily constructed by non-disabled designers. Although they often forge constructive relationships with disabled musicians, they lack the lived experience of disability. We propose that this deficiency not only prevents designers from having an intimate understanding of a disabled musician's needs and desires for a successful musician-instrument relationship but also—and probably mostly—what “successful” actually means.

In this paper, through the lived experiences of the first author, Andrew McMillan, a musician with acquired disability, we propose that questions need to be asked about the relationship with an instrument when defining guidelines for AMIs. Following an autoethnographic methodology, we provide insights into the evolution of Andrew's musician-instrument relationships. Specifically, we account for the evolution of his practice and his relationship; first, as a non-disabled musician performing traditional instruments, then as a disabled musician performing DMIs. We also discuss “success” in this context by outlining Andrew's objectives in pursuing an equivalent disabled musician-instrument relationship. As a disabled musician and DMI designer and researcher, Andrew has the unique vantage point of being able to intimately investigate the gap between the critical aspects that drive the successful musician-instrument relationship for a person using AMI and place the information gathered from his insights into design discourse.

The goal of this article is twofold. First, it identifies the necessity to account for the relationship between a disabled musician and their instrument in the design process of AMIs. Second, it offers methodological suggestions on how to elicit discussions around a disabled musician's phenomenological experiences and characteristics that contribute toward a successful musician-instrument relationship. We identify that cultural probes and conceptual metaphors are excellent tools for this purpose.

The rest of this article is structured as follows. In the next section, we cover the related work in this article's various areas of interest. The successive section is the core part of this article, in which we present an autoethnographic account of the first author's experience and how his relationship with musical instruments evolved after a life-changing accident. We then discuss the implications of this work and offer suggestions on how to elicit meaningful musician-instrument relationships at the design stage.

2. Background

This section reports the state of the art in the areas relevant to this paper: DMIs and AMI design and the relationship between musicians and their instruments. We also describe cultural probes and conceptual metaphors as methodologies for research investigations.

2.1. DMI and AMI design frameworks

The HCI (Human Computer-Interaction) legacy of the idea of musicians being “users” of DMIs has been recently challenged by Rodger et al. (2020), who points out that musicians should not be considered users, but rather agents in musical ecologies. This concept resonates with Brown (2016), who proposed that creative practices can be framed as an agency network, which includes human and non-human agents. When dealing with AMIs, the human agents participating in the musical ecology have further complexities that need to be addressed in the design. An example of addressing these complexities in Special Education Needs settings is discussed by Blatherwick et al. (2017), who suggested a number of aspects to be considered when designing for and with mixed-ability students. Different musicians have different needs of their instruments; thus, it is inherently complex to know what is demanded or expected from the instrument. Waters (2021) points out that an HCI expert and a musician have different views on the demands of the design process or framework. He states that designers approach a problem by breaking it down into component functions that they can solve whereas musicians are focused on deciding or making choices in the moment of doing (music). The way designers create and intend for instruments to be played is often altered or extended by musicians, a phenomenon that has been referred by HCI and NIME scholars as appropriation (Dix, 2007; Masu et al., 2016; Zappi and McPherson, 2018). Therefore, assessing the success of an instrument on the basis of its functions or behaviors as the designer knows them to attain predetermined goals is restrictive (Rodger et al., 2020). The variable nature of a musician's activities challenges the idea of a prototypical user that is often the implicit subject of design frameworks. Rodger et al. (2020) proposed to use a specification-type framework that tailors various methods to meet the needs of individual agents. The authors suggest that musical instruments do not have a prototypical user; the musician develops effective skills within their environment through a multiplicity of processes afforded by the design and specificities.²

This article is focused explicitly on a sub-category of DMIs—Accessible Musical Instruments (AMI). AMIs have evolved over the years from adapting conventional instruments to bespoke instruments and interfaces. Two recent studies, a survey of inclusive instruments (Frid and Ilsar, 2021) and a review of inclusive musical interfaces (Frid, 2019), added important knowledge to this research area. Categories of technologies that apply to AMIs are listed by Larsen et al. (2016) as they discuss the prospect of musical instruments for people with disabilities.³ Instruments mentioned by the authors include Soundbeam⁴, EyeMusic (Hornof and Sato, 2004), EyeGuitar (Vickers et al., 2010), Skoog⁵, and TouchTone (Bhat, 2010). In their practice, some AMIs designers work closely with disabled musicians on bespoke instruments. This is the case with Drake Music.⁶ Some examples of their creations include The Kellycaster⁷, a bespoke type of guitar co-designed by John Kelly, a disabled musician who has a self-described “punk at heart” approach to music (Harrison et al., 2019), and Charles Matthews. Another example of an interface that has been used for AMIs is that of the MI.Mu Gloves⁸, an interface designed to play a sophisticated version of an air guitar, air synth and air drums. There are also current developments that lie somewhere in between bespoke and generic instruments, such as the Touch Chord⁹ and Jamboxx.¹⁰ Although these instruments can meet the needs of a large range of users with disabilities, they do require some physical abilities to use each instrument. For instance, the Touch Chord requires the ability to activate sounds by touching a sensor placed on a board and the Jamboxx requires the ability for the disabled musician to be able to produce enough breath to activate it.

With the many varieties of components and interfaces available and being utilized in DMI and AMI design, challenges exist in evaluating bespoke design in instruments. Instruments specifically designed to be inclusive have been highlighted by Lucas et al. (2019). Their methodological approach consisted in using quantitative data and qualitative observations whilst also taking in observations and viewpoints from the participant. This holistic approach covers the many variables around designing with individuals with complex needs around accessibility and function.

2.2. Conceptual metaphors and cultural probes

Conceptual metaphors help explain or describe concepts that contribute toward assessing what elements and functionalities of an interface a user might intuitively relate to. This understanding can therefore assist a designer in forming design concepts. Metaphors have been used to understand embodied interactions, such as how abstract sound concepts like pitch, volume, and tempo might be associated with body movements through a subject's actions (Antle et al., 2008; Bakker et al., 2012). Kim and Maher (2020) discuss how conceptual metaphors inform design decisions when mapping interface elements to function for consumer devices. When considering conceptual metaphors as an approach to participatory design, Wilkie et al. (2013) introduce the connections between an image schema and embodied cognition theories with interaction. In the above examples, participants considered their responses from sets of concepts and items (e.g., words, images) developed by the researchers. Waters (2021) adopted conceptual metaphors in his investigations by referring to the connection between the musician understanding the analog world whilst imagining or conceptualizing that into the digital realm. Waters describes a continuum of inseparable Deleuze-Guattarian's assemblages (Deleuze and Guattari, 1988) as a player-instrument-social expectation.

Another methodological tool discussed in our work is the cultural probes (Gaver et al., 1999). Cultural probes are design artifacts designed to stimulate reflections on the relationship between humans and technology and, in the DMI space, between musicians and their instruments. Tahıroğlu et al. (2020) used this tool to investigate how technology changed the mode-of-being of musical instruments. Instrumentalists familiarize themselves with sensing the unique characteristics of an instrument (De Souza, 2017), and we can consider this relationship as a probe into musical possibilities (Tahıroğlu et al., 2020). The idea that traditional instruments can serve as probes also applies to DMIs. Cultural probes can be employed to deepen the understanding of musicians' experiences to eventually support the design of the DMI to better assist musicians in achieving their goals. As a result, DMIs themselves become probes that we can use to understand our and other musicians' experiences relating to the human condition, the instrument, and the music we wish to make (Tahıroğlu et al., 2020). Interfaces and technical components have also been used as probes to explore specific research questions (Jack et al., 2020; Guidi and McPherson, 2022). Waters (2021) provides a good example of what a probe could help discover from the account of a drummer describing his experience of playing their instrument:

“Bodily exploration of movement on top of, across, and within, the interchangeable pathways of the drum kit. Physical restrictions considered, I tend to think of this style of playing as waves of circular phrasing moving above, around, and passing through the kit, the presence of which is felt both in the feet and the arms/hands, as well as the knees, chest, and stomach.”

2.3. Musician-instrument relationship

A musician-instrument relationship is developed via the intimacy that exist between a musician and their instrument as well as the perceptual transparency between the two entities (Nijs et al., 2009). This perceptual transparency, which is acquired with practice, refers to the musician's perception of their instrument as an extension of their body (Rabardel, 1995; Leman, 2007; Morreale et al., 2018). These elements contribute to a sense of flow in their performances as they help provide a sense of control, motivation, and wellbeing contributing to longevity in a musician's application to the instrument (Csikszentmihalyi and Csikszentmihalyi, 1992; Simoens and Tervaniemi, 2013). Simoens and Tervaniemi (2013) quote the violinist Stephen Bryant in an interview “I don't like the thought of anyone else playing it [his violin]. It's such a close relationship”. This comment exemplifies the intimate relationship a musician can have with their instrument or their tools. The musician-instrument relationship can thus be similar to that one might have with another person they care deeply for. When playing or performing, this close relationship creates a unity between the instrument and the musician contributing to creative expressions in the interplay between the two (Nijs et al., 2009; Waters, 2021). This unity stems from the studies in embodied music cognition, which provide a framework for studies of the musician-instrument relationship. According to Leman (2007), the human body is considered the natural mediator between the musician's mind and the physical environment that contains musical energy. Considering how a musician might communicate ideas into this physical environment creating and containing musical energy, the musician must first establish a musician-instrument relationship as an entity that players can establish dynamic relations with Jordà (2004).

A musician's subjective experience when performing an instrument thus establishes and strengthens their relationship with it. The experience can be considered from a perspective “in which the interaction between musician and musical environment, the nature of human activities, and the quality of subjective experience are addressed” (Nijs et al., 2009). A successful interaction in which the subjective experience is addressed contributes to the feeling that the musical instrument has become part of the body. In other words, a musician experiences an embodied relationship with an instrument. The embodied experience creates entanglements between humans, machines, objects, and environments, including social structures (Mice and McPherson, 2022). The resulting entanglements are part of embodiment theory in how we interact with objects and environments and are intrinsically a part of them as they are a part of us. Entanglement theories specific to HCI are discussed by Frauenberger (2019). The authors suggest that humans are inseparable from the technologies we engage with; a relation that is described by a philosophical concept knows as relational ontology (Rosenberger and Verbeek, 2015). A specific area of embodiment theory related to sensor-motor skill is discussed by Guidi and McPherson (2022) in their investigation of skilled musicians using an unfamiliar interface (an augmented guitar pick). The musician's motor skills can be impacted by changes to the interface's physical or sonic characteristics (Morreale et al., 2019) affecting interactions and the embodied experience.

The musician regulates the goal-directed activity structure of music performance, which is inspired and influenced through the musician's subjective experience of the musical ecology during the performance.

Forms of entanglements also apply to interactions between the musician and the musical environment created throughout the performance (Nijs et al., 2009). Partially conflicting with the view of Rodger et al. (2020) on goal-directed activities, Nijs et al. (2009) suggest that the musician regulates the goal-directed activity structure of music performance, which is inspired and influenced by the musician's subjective experience of the musical ecology during the performance. Both elements (goal direction and subjective experience) influence each other and are optimized through an iterative process. These viewpoints are elaborated within the framework of embodied music-cognition (Nijs et al., 2009). This framework is based on Ecological Philosophy, Activity Theory, and Flow and Presence Research. Ecological philosophy describes a musician-instrument relationship in which the musician and their instrument merge and boundaries are no longer experienced. This functional transparency occurs when a musician directly responds to the musical environment and is lost when the instrument is altered (Morreale et al., 2018). Using the activity theory framework, a musician acts as the mediator that establishes an intimate relationship with their instrument (Nijs et al., 2009). When the subject, object, and environment combine, a state of flow is produced, which results in enjoyment, engagement, and increased motivation (Csikszentmihalyi and Csikszentmihalyi, 1992). Achieving this flow state enables a musician to establish a long-term constructive relationship with their instrument. An embodied experience of the instrument becoming an extension of the musician contributes to withdrawing of the instrument from consciousness. The state of flow is achieved through this immersion of the subject and object through any musical activities (Nijs et al., 2009). Each successful experience of flow and transparency contributes to strengthening long-term relationship bonds between a musician and their instrument (Csikszentmihalyi and Csikszentmihalyi, 1992). A positive, satisfying, and constructive relationship between a musician and the instrument relies on not being hindered or obstructed by technical difficulties, performance anxiety, or accessibility in playing an instrument. These obstacles will ultimately affect not only the positive experience of creative practice but interrupt the flow state, which is the ultimate loss of awareness of oneself when performing a task (Nakamura et al., 2002). The hindrance of any obstacle must be reduced to establish and maintain engagement and ultimate embodiment with an instrument. Waters (2021) and Simoens and Tervaniemi (2013) provide examples of how interfaces might interrupt or obstruct a musician or performer from engaging in the process of making music.

Through the processes explained above that lead to embodiment, an experience is established where the musician and the instrument are operating in what Nijs et al. (2009) consider an instrumental genesis. The instrumental genesis is a two-fold movement: the instrument constraints the creative possibilities of a musician, who then generates automatic responses within the musical environment. A relationship of reciprocal affordances is then established, which integrates the instrument with a musician through the activities, contributing to the feeling of the instrument being a natural extension of the musician within a musical environment.

Music, musical instruments, technology, and social and cultural structures do not develop in isolation. Each plays a role in contributing to the thinking and designing of the day. For example, the valve that was adopted into brass instruments relates to metal tubes and plumbing used at the time (Waters, 2021). Also, the transgression or migration (of musical instruments) from region to region or even rural to urban/city environments affects change in construction and even the playing style of some instruments (Bates, 2012). Bates (2012) suggests that humans are connected with instruments through the multiple functions of instruments (e.g., dance, celebration, experimentation, rebellion) and our relationship with them, from instrument making to performance. These connections impact how our expressions, playing techniques, and instruments develop. As Bates (2012) points out, our embodiment of musical instruments and their sound materialize in how parts of instruments are named and can refer to the human anatomy, such as neck, body, and head. With respect to the technology surrounding a musician's connection and physical relationship with their instrument, there are significant differences between conventional acoustic or electric instruments and DMIs. With acoustic or electric instruments, it can be argued that there is a more transparent connection between the musician and the instrument. The musician faces directly transmitting energy through their body to excite the resonance of an instrument and produce the sound. Jordà (2004) describes this connection as fuzzy and unclear. DMIs, on the other hand, listen for gestures performed on an interface and, via mapping, changes sound parameters. Tahıroğlu et al. (2020) suggests that our relationships with DMIs can be “decoupled from the established relationships we have with more traditional musical instruments.” This suggestion resonates with the notion that coupling is considered important for establishing a successful musician-instrument relationship (Nijs et al., 2009). Our relationships with DMIs and musical norms are shaped and transformed through opportunities (Tahıroğlu et al., 2020). This, in turn, impacts compositions, performances, and musical experiences, and our intentions forge relationships with machine/interface technologies. It is through the creative intent and gestures that a musician uses to connect with the technologies available that establish and determine the strength of the musician's relationship with the instrument. This has been described as a “dynamic re-formation of gestural and expressive intent” (Van Nort, 2011). How physical parameters respond to our intentions plays a part in establishing expressive, strong, and meaningful relationships that help create shared authorship, agency, and intentionality between a musician and an instrument. Experiencing or feeling the effects of input devices is proposed as important by Hunt et al. (2000) because interactions with interfaces can determine a musician's experience.

3. Autoethnographic account

In this section, we offer Andrew's autoethnographic account of the evolution of his relationship with his musical instruments. This account provides insights into the evolution of his musician-instrument relationships before and following an injury resulting in a life-changing disability. Autoethnography combines autobiography and ethnography to systematically describe and analyse personal experiences and observations in retrospect (Ellis et al., 2011). Autoethnography offered Andrew the methodological tools to thoroughly observe, challenge, and interrogate his thoughts, beliefs, and assumptions on the evolving relationships with his musical instruments. These experiences, reflections, and analyses produce insights for further discussions and development as they are cross-referenced with existing research and collaborators. Following typical autoethnographic procedures, Andrew will use the first-person singular in the rest of this section.

3.1. Relations with the instrument before the accident

Before the accident, I was able to pursue my creative practice on various instruments. I was primarily a saxophone player but also played flute, clarinet, and piano. I was also an electronic musician, composer, and dedicated improviser. All of these creative activities or practices were undertaken with relative physical ease. I was performing many gigs on saxophone and sometimes piano and keyboards, composing and performing for theater and dance, and running recording sessions in my studio. I ran workshops in free-improvised performances with other musicians and artists from disciplines such as dance, spoken word, moving image, and graphic art. I played a key part in organizing events and took up many opportunities to attend and perform at various festivals around Aotearoa (New Zealand). Alongside these creative activities across different communities, I was completing an Honors degree in Composition. Unfortunately, the accident occurred before I completed my degree. When considering my relationship with my instruments before the accident, what mostly emerges is the deep connection with each instrument I played. This connection is described in the following journal entry, in which I recalled the ritualistic aspect that involved setting up my saxophone:

“When considering the relationship between an artist and their instrument, I would like to start by considering my relationship with my saxophone before my accident. I fondly remember, when preparing to play either for practice, rehearsal, or performance, the ritual of taking the saxophone from the case and assembling it. This formal process allowed me to connect with the instrument in a way that would not only prepare the instrument to be played but prepare me to play the instrument. As each part of the saxophone was placed together, I would feel as if I was connecting with the instrument. This would be especially apparent when testing the reed on the mouthpiece and strongly felt as I placed my fingers on the keys. As I felt the pressure of the springs of the keys against the pads of my fingers, this sensation of touching the instrument before producing the first sound gave me an extremely strong connection. This is where my connection or response begins in how the relationship between the artist and their instrument can be considered. Similarly, when sitting to play the piano, a shorter ritual would take place. I would feel that the act of approaching and sitting down at the piano, opening the lid if necessary, and then pausing, considering the musical possibilities before placing my hands on the keys was important. Once placing my hands on the keys, there would be a further pause to engage with the sensation of the keys beneath his fingers. Through making this connection with these instruments and forging a strong relationship, it now felt as if we were not separate entities, I was not a user, and the instrument was not merely a tool. Still, we were bound together, ready to produce a creative output. That creative output was music.”

The ritual described in the note above shows how the relationship is not best described in terms of musician-tools-music but rather between the (musician-instrument)-music, where musician-instrument constitutes a unique element. The tools needed to create music are intimately bound to the musician. I do not perceive them as separate entities but rather as one entity. Although these rituals may sound romantic, they are extremely fond memories for me and offer insight into what I consider necessary conditions to engage positively with a new DMI. In outlining how a bespoke DMI would be, I need to account for the sensations that take place even before playing it, as well as how those sensations contribute to creating an intimate relationship with the instrument and performing with satisfying action and response.

3.2. Experiments with instruments as a disabled musician

An accident left me paralyzed with tetraplegia from the chest down, dramatically affecting my creative practice. This accident put on hold the completion of my degree. Whilst in my first year of rehabilitation, I had the opportunity to create music for theater and dance. I created this music using my laptop, which had accessible aids¹¹ I continued composing and producing music for theater and dance, along with getting back to organizing performances and events in Tamaki Makaurau (Auckland). After a few years, I started looking into ways to perform, compose, and produce music. These early engagements with live creative practice mainly involved using my voice, a small percussion instrument that could fit on my lap, and a slide whistle. Although rewarding in some ways, there was always an unpleasant feeling of novelty to performing in this manner as most of these creative activities were undertaken in freely improvised collaborative settings. I eventually completed my Honors degree and joined a Master's degree in Composition and interactive technologies. During my Masters degree, I discovered how to use the laptop as an instrument. These early experiments involved using webcam tracking, voice processing, and mapping of the joystick, mouse, and keyboard. I also experimented with some sensor technology, but due to the complexities around setup, calibration, and reliability, it eventually failed to become a permanent part of my performance setup. In 2019, I started playing the guitar, placing it on a custom-made case that converts to a stand supporting the guitar horizontally across my lap. I processed the guitar signal through a distortion pedal and a volume pedal. With this setup, I often performed in an ensemble with a bass player and a drummer, playing rock-influenced free improvisations. To play the guitar, I use two splints attached to my hands, one for picking and strumming, the other for replacing a slide on the strings. The range of sounds I am able to produce in this way includes slide guitar tones, strum “bar” chords across an open tuning, sustained feedback, and short and sharp percussive tones.

3.2.1. Issues that hinder musician-instrument relationship

Instruments from my early experimentation felt “novelty-like.” Although they provided me with the opportunity to creatively reconnect with fellow artists, they failed to create a feeling that was similar to performing on instruments before my accident. Using the laptop as an instrument opened up opportunities that moved away from this feeling of being included as a novelty for participation. However, the lack of haptic/tactile feedback results in feeling disconnected from the creative process and disrupts the intimacy of my relationship with the instrument. To date, the guitar has been the most rewarding instrument in terms of feeling connected with an instrument in a physical or haptic/tactile way. However, the size and the cumbersome setup of the instrument have significant limitations when it comes to musical control and expressivity. Playing single notes is possible, but not with accuracy and speed. As a result of what I can play, collaboration with other artists is possible only in experimental music settings. For instance, I can play chords only as bar chords across the strings and single notes cannot be timed accurately. In brief, I am prevented from performing in more conventional music settings. The lack of connection with the laptop or other computer interfaces tested so far results in a limited relationship with the instruments. By contrast, having easy access to the guitar, and being able to play it with some ease (i.e., a low entry level and then the opportunity to progress with a high ceiling), has alluded to the possibility of feeling the same sense of purpose, accomplishment, and satisfaction I had before my accident. As much as the desire of discovering new ways of playing and considering music is understandable, I still desire to be able to construct conventional musical ideas and expressions. This is part of the relationship that I wish to build with an instrument and will help establish relationships with musicians performing on conventional instruments.

I extensively used designed guidelines from previous design frameworks in identifying my specific aims with the design of my DMI and in creating prototypes (Overholt, 2009; Morreale et al., 2014). What I have not found available are suggestions centered on the elicitation of specific musician-instrument relationships. In other words, these frameworks offered rich insight into the specificities of the instruments but failed to offer insights into how to design for specific musician-instrument relationships. This is the gap I evidence in current frameworks, and I urge more research in DMI and AMI design: the musician-interface relationship should have priorities over specific functionalities.

3.2.2. Consideration on AMI design

When creating bespoke instruments, my experiences before and after the accident kept informing successive design choices. Each of the approaches I have worked with has indeed offered useful takeaways. The accurate and deterministic response of a computer keyboard, the interpretation of gestures through reliable, consistent and intuitive sensors, the ease of control of the mouse joystick, and the physicality of the guitar are all important features I came to appreciate and that I will consider for my future AMIs.

In the list below, I indicate a series of instrument characteristics that I found crucial in establishing a relationship with an instrument. The intention is not to offer a comprehensive list of design guidelines. This quest would be pointless as these features worked for the specificities of my condition. However, I hope these sorts of suggestions could pave the way for future directions in the design of AMIs.

• Enable a physical connection with the instrument. Interfaces can be analog (i.e., keys, strings) or digital (i.e., breath controllers, pressure sensors).

• Create a linear or predictable connection between energy, the energy injected in the instrument, and the energy generated by the instrument.

• Ensure an intuitive, natural, and predictable response from the instrument that can generate a visceral connection and symbiotic relationship.

• Consider ease of setup, possibly in a way that any person can assist in setting up the instrument.

• Ensure the accessibility to the instrument: Positioning oneself and the instrument to play or move away or out from the instrument when finished playing should require little to no assistance once the instrument is set up.

• Design for a low entry level so that a musician can start playing the instrument with relative ease while offering the opportunity to progress and develop those skills to a high ceiling for more complex performances.

• Enable accuracy and the repetition of musical ideas so that more intricate musical lines can be created beyond simply improvising sonic textures or effects.

Several of these points have been illustrated in other DMI frameworks. However, addressing them is more complex when designing AMIs. The added complexities come from physical (e.g., limited mobility and reduced tactility) and cognitive impairments (e.g., ability to concentrate, understand tasks, issues with speech and language, behavioral difficulties; Blatherwick et al., 2017).

4. Discussions

Existing DMI and AMI design frameworks primarily focus on technical and goal-based solutions around the environments they investigate. However, the complexities surrounding the subjective aspects of the musician-instrument relationship have been mostly overlooked. Understanding these complexities involves exploring musicians' emotional and physical connections with their instruments. These aspects are undoubtedly more challenging to measure and not readily available compared to technical specifications commonly addressed in design frameworks. This observation is aligned with what Born (2020) called analytical ontology. With this term, Born referred to the assumptions often implicit in Music Information Retrieval research and practice about what music is. Many music subtleties (mostly of non-Western music), which are difficult to extract and analyse, are simply ignored. We see a similar issue in the design and design frameworks of DMI and AMI.

Scholarly investigations on connectivity and embodiment of musician-instrument relationships, how they contribute to creative practice, and the phenomenology of the experience that goes with them have primarily focused on conventional instruments and non-disabled musicians. In successful musician-instrument relationships where functional transparency takes place (Nijs et al., 2009), key concepts such as flow (Csikszentmihalyi and Csikszentmihalyi, 1992) contribute to the embodiment and entanglements of instruments with the musician. Musician-instrument relationships of disabled musicians seem underrepresented in current research. An opportunity exists for designers and musicians to explore ways to investigate the complexities surrounding the feelings and values within a musician's relationship with an instrument, how they are accessed, and lastly, the technological solutions to be considered.

Andrew's comment on the relation (musician−instrument)−music resonates with the embodied relationship described by Ihde (1990). The American philosopher talks about embodiment needing to be constituted, or learnt, for a technology to become transparent. To do so, Ihde explains, the technology “must be technically capable of being seen through.” We identify a designer's paramount role in the musician-instrument relationship to be constituted in the case of disabled musicians. Thus, the relationship might be best represented as [(musician−designer)−instrument]−music. For this relationship to exist, the designer must develop an understanding of the physical limitation and possible technological options and an intimate appreciation of the musician's desires, goals, and expected output. It is not only the physical limitations that significantly differ among disabled musicians but also their intended relationship with the instrument. Creating instruments with generic inputs and outputs that can easily adapt to a wide range of users is problematic for AMI designers due to individual nature of disabilities and impairments.

The question remains, how can these technical solutions best satisfy a musician-instrument relationship? How do designers find the best way to match these solutions to the specific demands of the needs of an individual's musician-instrument relationship? A combination of processes and ecologies, termed specificities by Rodger et al. (2020), offers insights into constructing a relationship between the musician and the instrument beyond goals, environments, and solution approaches. The work of Rodger et al. (2020) is focused on an already developed system or prototype to be evaluated rather than investigating a musician-instrument relationship before the design. We argue that considerations about a successful musician-instrument relationship should be given at the conceptual design stage.

One possibility is to integrate conceptual metaphors with cultural probes, technology probes, or research products (Jack et al., 2020). Probes can play a part in discovering more details around the complexities of emotions that establish the musician-instrument relationship. Understanding these complexities might help describe what feelings produce engagement and embodiment of the instrument to fulfill the desire for expressions through a successful design. One example is the conceptual metaphor of asking what acoustic instrument they would wish to play if they could (Waters, 2021). And if so, why that instrument? What are the elements around that instrument that excites them?

A designer could create a probe that cultivates an answer from the musician they are designing with, who could describe how they would like to engage with an instrument. An example could be for a designer to ask the musician to describe, draw, or demonstrate movements that communicate musical intentionalities or sensations and how to physically embody them in the design of the instrument and the gesture-sound mapping. This probe would help inform the musician-instrument relationship and could assist in how the designer approaches further developments. The probe could consider pathways, physical restrictions, phrasing or movements, connections to and with an instrument that would likely be felt or resonate, and what instrument/sonic responses would feel the most natural or interconnected. Conversely, asking a musician to freely associate and describe the complexities around the subjective feelings they are looking for when playing a musical instrument could create a set of probes for any designer to work with. However, do the answers for these types of probes establish an understanding for a designer of the intimate relationship a musician may wish to experience? This is what we are seeking to understand through future research.

One option would be to start with questions that explore the phenomenology surrounding the musicians' desires when wishing to engage with an instrument and experience its response. We propose the following questions to inform the design of the probes:

1. How do they wish for the instrument to “feel,” and what psychological or subjective connections and responses would they consider rewarding?

2. How do they intuitively and instinctively understand or control the instruments' responses? What are the physical limitations and opportunities?

3. Which movements (and how do they) feel the most natural, intuitively connected to music/sound production parameters and ultimately rewarding

4. How does creating combinations of these movements intuitively engage with the imagination around making music/sound?

5. What feedback responses are the most effective or appropriate to feel connected to the instruments' responses?

6. How can a link between the psychological and the physical connection(s) and responses lead to “functional transparency”?

The musician can respond to these questions in any way they wish to communicate. Through words, pictures, demonstrations of movement, or communication in any form of media that best suits their interpretation of a set of questions. This could be prescribed from past experiences, imagination, and ideals. For instance, the following quotes describe Andrew's answers to the first question points:

• “The ability to be intuitively accurate through my responses”

• “To feel the music around me, or be moved within a musical environment and be able to place sound from my mind into that environment with as little conscious cognitive thought and physical impedance as possible”

• “To create the idea of a sound or note and play accurately without needing to think about the physical tasks required to produce it”

• “To have responses from the instrument that feel innately connected to the energy put into it; this means having the opportunity to put energy into the instrument in a haptic/tactile way, and have that represented as a sonic realization or representation directly connected to that input of energy”

From here, the design can refer to the principles of activity theory¹² as framing to analyse the responses. Starting from these responses, designers would create further probes for investigation and development and move toward constructing a prototype.

5. Conclusion

Existing design frameworks for DMI largely focus on technological solutions and goal-based approaches while overlooking the subjective aspects of the musician-instrument relationship. Additionally, research on the embodiment and connectivity of musician-instrument relationships in creative practice has primarily focused on conventional instruments and non-disabled musicians, leaving the experiences of disabled musicians underrepresented. In this article, we proposed that DMIs and AMIs design strategies should extend their scope of investigation to also account for the connections that musicians might have with their instruments, which are difficult to measure and not typically addressed in design frameworks. Notably, the intention of this article was not to provide a comprehensive framework. This article is intended to surface fundamental points that are needed to be tackled when designing for and with disabled musicians. These points can be integrated as complementary features in existing DMI and AMI design frameworks. Specifically, we proposed that musician-instrument relationships should be considered in the design of AMIs at an early stage before considering goals and technological solutions. We also indicated methods to surface the intended musician-instrument relationship. We discussed the possibility of borrowing conceptual metaphors and cultural probes as tools to be used within a DMI and AMI framework to account for the subjective experience and the ecology and specificity that determine the disabled musician's desires for a musician-instrument relationship.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author/s.

Ethics statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent from the participants was not required to participate in this study in accordance with the national legislation and the institutional requirements.

Author contributions

AM ideated the project and presented the autoethnographic account. FM offered theoretical support on DMI/AMI and guidance throughout the project. Both authors wrote the article. Both authors contributed to the article and approved the submitted version.

Funding

This work was funded by University of Auckland CAI RDF grant (9483-3728468) and University of Auckland PREeSS grant (312820-9441-AMCM028).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^Whether I in both acronyms refers to Interfaces or Instrumentsis still the subject of a scholarly debate (see, for instance, Jensenius and Lyons, 2017) that we don't intend to contribute to. Some AMIs are not digital; thus AMI are not fully included within the DMI set.

2. ^Rodger et al. (2020) use the term specificities to refer to configurations of things or objects created for particular needs of users, agents, and ecologies. In music, these configurations can be involve instruments, musicians-as-agents, and ecological contexts.

3. ^There is an ongoing debate around the expressions people with disabilities or disabled people across various public and social spaces. Andrew is confortable in using the expression people with disabilities as it well represents his feeling around his disability not being an external factor to him.

4. ^www.soundbeam.co.uk

5. ^www.skoogmusic.com

6. ^www.drakemusic.org

7. ^www.drakemusic.org/technology/instruments-projects/the-kellycaster

8. ^www.mimugloves.com

9. ^www.humaninstruments.co.uk/instruments

10. ^www.jamboxx.com

11. ^Accessible aids include a special joystick mouse, a small USB keyboard so that both the joystick and keyboard could fit on a tray on my lap. These worked in conjunction with bespoke designed typing splints. I also had “Dragon Dictate” dictation software and a headphone microphone to communicate speech to text for emails, word/text documents, or navigating the internet.

12. ^Activity Theory principles are outlined as: (1) the unity of consciousness and activity, (2) object-orientedness, (3) hierarchical structure of an activity, (4) iternalization and externalization, (5) mediation, and (6) continuous development.

References

Antle, A. N., Droumeva, M., and Corness, G. (2008). “Playing with the sound maker: do embodied metaphors help children learn?,” in Proceedings of the 7th International Conference on Interaction Design and Children (Chigago, IL), 178–185. doi: 10.1145/1463689.1463754

CrossRef Full Text | Google Scholar

Bakker, S., Antle, A. N., and Van Den Hoven, E. (2012). Embodied metaphors in tangible interaction design. Pers. Ubiquit. Comput. 16, 433–449. doi: 10.1007/s00779-011-0410-4

CrossRef Full Text | Google Scholar

Bates, E. (2012). The social life of musical instruments. Ethnomusicology 56, 363–395. doi: 10.5406/ethnomusicology.56.3.0363

CrossRef Full Text | Google Scholar

Bhat, S. (2010). “Touchtone: an electronic musical instrument for children with hemiplegic cerebral palsy,” in Proceedings of the Fourth International Conference on Tangible, Embedded, and Embodied Interaction (Cambridge, MA), 305–306. doi: 10.1145/1709886.1709955

CrossRef Full Text | Google Scholar

Blatherwick, A., Woodbury, L., and Davis, T. (2017). “Design considerations for instruments for users with complex needs in SEN settings,” in Proceedings of the International Conference on New Interfaces for Musical Expression (Copenhagen: Aalborg University Copenhagen), 216–221.

Google Scholar

Born, G. (2020). Diversifying MIR: knowledge and real-world challenges, and new interdisciplinary futures. Trans. Int. Soc. Mus. Inform. Retriev. 3, 193–204. doi: 10.5334/tismir.58

CrossRef Full Text | Google Scholar

Brown, A. R. (2016). “Understanding musical practices as agency networks,” in Proceedings of the Seventh International Conference on Computational Creativity.

PubMed Abstract | Google Scholar

Csikszentmihalyi, M., and Csikszentmihalyi, I. S. (1992). Optimal Experience: Psychological Studies of Flow in Consciousness. Cambridge: Cambridge University Press.

PubMed Abstract | Google Scholar

De Souza, J. (2017). Music At Hand: Instruments, Bodies, and Cognition. Oxford: Oxford University Press. doi: 10.1093/acprof:oso/9780190271114.001.0001

PubMed Abstract | CrossRef Full Text | Google Scholar

Deleuze, G., and Guattari, F. (1988). A Thousand Plateaus: Capitalism and Schizophrenia. London: Bloomsbury Publishing.

Google Scholar

Dix, A. (2007). “Designing for appropriation,” in Proceedings of HCI 2007 The 21st British HCI Group Annual Conference University of Lancaster (Lancaster), 1–4. doi: 10.14236/ewic/HCI2007.53

CrossRef Full Text | Google Scholar

Ellis, C., Adams, T. E., and Bochner, A. P. (2011). Autoethnography: an overview. Histor. Soc. Res. 273–290. doi: 10.17169/fqs-12.1.1589

CrossRef Full Text | Google Scholar

Frauenberger, C. (2019). Entanglement hci the next wave? ACM Trans. Comput. Hum. Interact. 27, 1–27. doi: 10.1145/3364998

CrossRef Full Text | Google Scholar

Frid, E. (2019). Accessible digital musical instruments—a review of musical interfaces in inclusive music practice. Multimodal Technol. Interact. 3:57. doi: 10.3390/mti3030057

CrossRef Full Text | Google Scholar

Frid, E., and Ilsar, A. (2021). “Reimagining (accessible) digital musical instruments: a survey on electronic music-making tools,” in Proceedings of the International Conference on New Interfaces for Musical Expression (Shanghai). doi: 10.21428/92fbeb44.c37a2370

CrossRef Full Text | Google Scholar

Gaver, B., Dunne, T., and Pacenti, E. (1999). Design: cultural probes. Interactions 6, 21–29. doi: 10.1145/291224.291235

CrossRef Full Text | Google Scholar

Guidi, A., and McPherson, A. (2022). “Quantitative evaluation of aspects of embodiment in new digital musical instruments,” in Proceedings of the International Conference on New Interfaces for Musical Expression (Auckland: The University of Auckland). doi: 10.21428/92fbeb44.79d0b38f

CrossRef Full Text | Google Scholar

Harrison, J., Chamberlain, A., and McPherson, A. P. (2019). “Accessible instruments in the wild: engaging with a community of learning-disabled musicians,” in Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow), 1–6. doi: 10.1145/3290607.3313037

CrossRef Full Text | Google Scholar

Hornof, A., and Sato, L. (2004). “Eyemusic: making music with the eyes,” in Proceedings of the 2004 Conference on New Interfaces for Musical Expression (Hamamatsu), 185–188.

Google Scholar

Hunt, A., Wanderley, M. M., and Kirk, R. (2000). “Towards a model for instrumental mapping in expert musical interaction,” in ICMC (Berlin).

Google Scholar

Ihde, D. (1990). Technology and the Lifeworld: From Garden to Earth. Bloomington, IN: Indiana University Press.

Google Scholar

Jack, R., Harrison, J., and McPherson, A. (2020). “Digital musical instruments as research products,” in Proceedings of the International Conference on New Interfaces for Musical Expression, eds R. Michon and F. Schroeder (Birmingham: Birmingham City University), 446–451.

Google Scholar

Jensenius, A. R., and Lyons, M. J., (eds.) (2017). A NIME Reader: Fifteen Years of New Interfaces. Berlin: Springer. doi: 10.1007/978-3-319-47214-0

CrossRef Full Text | Google Scholar

Johnston, A., Candy, L., and Edmonds, E. (2008). Designing and evaluating virtual musical instruments: facilitating conversational user interaction. Design Stud. 29, 556–571. doi: 10.1016/j.destud.2008.07.005

CrossRef Full Text | Google Scholar

Jordà, S. (2004). Instruments and players: Some thoughts on digital lutherie. J. New Mus. Res. 33, 321–341. doi: 10.1080/0929821042000317886

CrossRef Full Text | Google Scholar

Kim, J., and Maher, M. L. (2020). Conceptual metaphors for designing smart environments: device, robot, and friend. Front. Psychol. 11:198. doi: 10.3389/fpsyg.2020.00198

PubMed Abstract | CrossRef Full Text | Google Scholar

Larsen, J. V., Overholt, D., and Moeslund, T. B. (2016). “The prospects of musical instruments for people with physical disabilities,” in NIME, Vol. 16 (Brisbane, QLD), 327–331.

Google Scholar

Leman, M. (2007). Embodied Music Cognition and Mediation Technology. Cambridge, MA: MIT Press. doi: 10.7551/mitpress/7476.001.0001

PubMed Abstract | CrossRef Full Text | Google Scholar

Lucas, A. M., Ortiz, M., and Schroeder, D. F. (2019). “Bespoke design for inclusive music: the challenges of evaluation,” in Proceedings of the International Conference on New Interfaces for Musical Expression, eds M. Queiroz and A. X. Sedó (Porto Alegre: UFRGS), 105–109.

Google Scholar

Masu, R., Conci, A., Menestrina, Z., Morreale, F., and De Angeli, A. (2016). “Beatfield: an open-meaning audiovisual exploration,” in COOP 2016: Proceedings of the 12th International Conference on the Design of Cooperative Systems (Trento: Springer), 309–314. doi: 10.1007/978-3-319-33464-6_20

CrossRef Full Text | Google Scholar

Mice, L., and McPherson, A. P. (2022). “Super size me: Interface size, identity and embodiment in digital musical instrument design,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA), 1–15. doi: 10.1145/3491102.3517626

CrossRef Full Text | Google Scholar

Morreale, F., Armitage, J., and McPherson, A. (2018). Effect of instrument structure alterations on violin performance. Front. Psychol. 9:2436. doi: 10.3389/fpsyg.2018.02436

PubMed Abstract | CrossRef Full Text | Google Scholar

Morreale, F., De Angeli, A., and O'Modhrain, S. (2014). “Musical interface design: an experience-oriented framework,” in NIME (London), 467–472.

Google Scholar

Morreale, F., Guidi, A., and McPherson, A. P. (2019). “Magpick: an augmented guitar pick for nuanced control,” in Proceedings of the International Conference on New Interfaces for Musical Expression, eds M. Queiroz and A. X. Sedó (Porto Alegre: UFRGS), 65–70.

Google Scholar

Nakamura, J.Csikszentmihalyi, M., et al. (2002). The concept of flow. Handb. Posit. Psychol. 89:105.

Google Scholar

Nijs, L., Lesaffre, M., and Leman, M. (2009). “The musical instrument as a natural extension of the musician,” in The 5th Conference of Interdisciplinary Musicology (Paris: LAM-Institut jean Le Rond d'Alembert), 132–133.

Google Scholar

Overholt, D. (2009). The musical interface technology design space. Organised Sound 14, 217–226. doi: 10.1017/S1355771809000326

CrossRef Full Text | Google Scholar

Rabardel, P. (1995). Les Hommes et les Technologies; Approche Cognitive des Instruments Contemporains. Paris: Armand colin.

Google Scholar

Rodger, M., Stapleton, P., Van Walstijn, M., Ortiz, M., and Pardue, L. (2020). “What makes a good musical instrument? A matter of processes, ecologies and specificities,” in Proceedings of the International Conference on New Interfaces for Musical Expression (Birmingham: Birmingham City University), 405–410.

Google Scholar

Rosenberger, R., and Verbeek, P.-P. (2015). “A field guide to postphenomenology,” in Postphenomenological Investigations: Essays on Human-Technology Relations (Lanham, MD), 9–41.

Google Scholar

Simoens, V. L., and Tervaniemi, M. (2013). Musician-instrument relationship as a candidate index for professional well-being in musicians. Psychol. Aesthet. Creativ. Arts 7:171. doi: 10.1037/a0030164

CrossRef Full Text | Google Scholar

Tahıroğlu, K., Magnusson, T., Parkinson, A., Garrelfs, I., and Tanaka, A. (2020). Digital musical instruments as probes: how computation changes the mode-of-being of musical instruments. Organised Sound 25, 64–74. doi: 10.1017/S1355771819000475

CrossRef Full Text | Google Scholar

Van Nort, D. (2011). Human: machine: human: gesture, sound and embodiment. Kybernetes 40, 1179–1188. doi: 10.1108/03684921111160403

CrossRef Full Text | Google Scholar

Vickers, S., Istance, H., and Smalley, M. (2010). “Eyeguitar: making rhythm based music video games accessible using only eye movements,” in Proceedings of the 7th International Conference on Advances in Computer Entertainment Technology (Taipei), 36–39. doi: 10.1145/1971630.1971641

CrossRef Full Text | Google Scholar

Waters, S. (2021). The entanglements which make instruments musical: rediscovering sociality. J. New Music Res. 50, 133–146. doi: 10.1080/09298215.2021.1899247

CrossRef Full Text | Google Scholar

Wilkie, K., Holland, S., and Mulholland, P. (2013). Towards a participatory approach for interaction design based on conceptual metaphor theory: a case study from music interaction. Music Hum. Comput. Interact. 259–270. doi: 10.1007/978-1-4471-2990-5_15

CrossRef Full Text | Google Scholar

Zappi, V., and McPherson, A. (2018). Hackable instruments: supporting appropriation and modification in digital musical interaction. Front. ICT 5:26. doi: 10.3389/fict.2018.00026

CrossRef Full Text | Google Scholar