Feedback in augmented and virtual reality piano tutoring systems: a mini review

1 Introduction

This survey reviews virtual reality (VR) and augmented reality (AR) systems for helping beginning students to practice piano. Beginning students typically receive short, infrequent lessons, which they practice with little to no feedback between lessons. Authors such as Percival et al. (2007) view this lack of feedback as a key impediment to learning piano, due to the time that students spend in solo practice and these students’ inability to identify, let alone correct, improper technique.

The survey reviews systems that were identified through the use of Internet searches between Fall 2021 and Spring 2023, using keywords like augmented reality, virtual reality, mixed reality, music, piano, tutoring, guidance, and feedback. It focuses on systems that support solo practice and provide visual feedback using AR/VR headsets. Most of these systems help users to identify, in real time, a sequence of keys that a piece requires them to play. The most common approach for guiding performances, referred to here as “falling notes,” uses moving shapes that approach, then touch, the keys to play. These applications detect key presses using MIDI, visual, audio, or tactile input; provide immediate feedback on the correctness of key presses; and, in some cases, summarize the quality of a user’s performance. Some applications attempt to accomplish additional goals, including encouraging practice through gamification and the teaching of notation literacy (Chow et al., 2013): the association of scored notes with a keyboard’s keys.

The survey’s emphasis on applications for the Meta Quest 2 headset is largely due to Meta’s (formerly Oculus’s) dominance in the AR/VR headset marketplace. As of July 2022, Quest had more than 90% of the world’s market share and was the exclusive supplier of U.S. and Canada VR headsets (Sevilla, 2022).

While accuracy of pitch is an essential element of music, these applications fail to offer feedback on other, essential aspects of performance, such as consistency of tempo, dynamics, and articulation (viz. Sarrazin, 2016). Authors commonly attribute their applications’ limitations to current AR/VR headsets, which use underpowered chipsets to limit the discomfort that the waste heat from these headsets would produce. The one exception to our focus on AR/VR headsets, Rogers et al.’s projector-based P.I.A.N.O., is included as an alternative that could support the additional computational power that contemporary headsets lack.

2 Piano tutoring systems

In what follows, piano tutoring systems are divided into research efforts, which were primarily the subject of publications, and open source and commercial systems, which were intended for common distribution. Examples of the former include work by Chow et al. (2013), Hackl and Anthes (2017), Molloy et al. (2019), Wijaya et al. (2020), and Rigby et al. (2020). Examples of the latter include Magic Keys, VRtuos, and PianoVision.

2.1 Research systems

Chow et al. (2013) sought to develop a system that encourages practice and promotes notation literacy. The authors’ system uses a Trivisio ARvision-3D HMD1 AR headset, a keyboard, and a computer. It uses fixed markers on keyboards to track a headset’s orientation. Issues with a stuttering AR overlay were resolved by taking a moving average of the translation positions.

Chow et al.’s system used a MIDI interface to track performances. The system’s display shows what keys to play, using line indicators that approach the keys and line lengths that correspond to each note’s duration: a “falling notes” approach (Figure 1). The display shows the musical score above the notes to aid in notation literacy. The system tracks key presses and releases, supplying feedback about mistakes in real time and using colors to show when notes were missed or released too soon. The system enforces a piece’s tempo by default but supports a “pausing notes” mode in which it pauses for its user to play each note before continuing. Finally, it displays a summary after each performance to help users identify mistakes.

FIGURE 1

FIGURE 1. An example of the “falling notes” display, where the notes to be played approach the keys of a keyboard. When the note hits the key on the keyboard, the note should be played. The length of the notes corresponds to their duration.

Chow et al. tested their system’s effectiveness with the help of seven students of varying skill. Overall, the students enjoyed the system, including its feedback and game-like structure, which helped with motivation. The system, however, failed to improve the students’ notation literacy; most students ignored notation to focus on the line indicators.

Chow et al. identified several opportunities for future work. Adding support for monitoring dynamics and articulation could be helpful for experts. More comprehensive feedback could help students to better identify where to improve. The system could provide more informative difficulty rankings and requirements. Finally, the authors recommended additional research on teaching notation literacy.

Hackl and Anthes’ HoloKeys (2017) adds virtual highlights to a physical keyboard’s keys to indicate the next note its user should play. It interfaces to a physical piano with a Microsoft HoloLens headset and an on-piano marker for tracking the piano’s position. The authors used Unity as their game engine, integrating it with Vuforia, an AR tracking software, and C# Synth Project, a MIDI processing library. HoloKeys allows its user to select a song to play and the song’s tempo. The interface then switches into playback mode, which plays the song while highlighting the keys being played as its notes are sounded. The user can skip ahead, rewind, pause, or play the song. The interface’s highlights can be resized to match the physical keyboard’s size.

The authors tested two approaches for highlighting keys in playback mode. One highlights a key at the moment its user should press it and removes the highlight when the key should be released. This approach proved difficult for learners, who needed to anticipate a song’s notes to play along. The other uses the aforesaid “falling notes” approach.

Hackl and Anthes encountered difficulties with the HoloLens’s technical limitations, such as the device’s limited field of view (FoV): the area visible through optical device’s lens, such as a headset (Awati, 2022). The authors noted that different augmentation methods and advances in AR technology could address these issues.

Molloy et al. (2019) developed a VR system to assess whether gamifying piano playing in MR could motivate beginners to practice and improve their skills. Their system uses a MIDI keyboard, an HTC Vive VR headset, and a camera. It uses a falling notes approach to indicate notes to play, colorizing keys to indicate note correctness and using pop-up text to indicate timing accuracy: e.g., “good” or “great”. A color-coded piano roll shows a user’s note correctness throughout the piece, while a final summary screen rates their performance. The system also features a game mode, which allows the user to fight an enemy spacecraft by playing accurate notes.

The system suffered from inaccuracies of the overlaid display and inconsistencies in tracking stability. Other limitations included the system’s failure to provide more comprehensive feedback and more innovative strategies for motivating students.

Wijaya et al. (2020)’s piano tutoring system records and analyzes piano fingering data. It uses a physical, transparent board to simulate a physical piano keyboard and to give tactile feedback on key presses. It uses the HTC Vive VR headset to display a virtual keyboard. It uses fingertip-mounted pressure sensors to identify when keys are pressed and two Leap Motion sensors to detect which keys are pressed.

Wijaya et al. (2020) used twelve test subjects without previous piano experience to assess their system’s effectiveness. Using an initial exercise, the authors first determined that VR distortion due to key presses at the edges of the Leap Motion sensors had little to no impact on user performance. The authors then conducted five-to-ten-minute learning sessions, some with the VR headset and others with a 2D screen. Subjects indicated the VR system was more intuitive, helpful for learning, and enjoyable than the non-VR system, but less comfortable, due to the headset’s weight. For future work, the authors planned to improve their hand tracking algorithm, to continue collecting data on piano fingering, and to improve their visualizations of fingering data.

Rigby et al. (2020)’s piARno, an AR-based piano tutoring system, was intended to enhance traditional teaching techniques; to improve a student’s notation literacy; and to encourage students to practice. Additionally, piARno provided quantitative analyses of student performance—a feature that was missing from piARno’s predecessors.

The authors selected the HTC Vive Pro VR headset for its larger FoV, lower cost, ample frame rate, and support for six degrees of freedom. To represent notes in a computer-readable format, the authors selected MusicXML (W3C Music Notation Community Group, 2021). For processing MusicXML documents, they selected music21 (Cuthbert and Ariza, 2010). To generate AR-based images of sheet music, they used MuseScore, an open-source music notation application with a built-in renderer. They included a MIDI interface to track key presses and releases. piARno displays a virtual image of sheet music in a user’s headset and overlays keys on a user’s physical keyboard with virtual note name labels. The HTC Vive Pro’s controllers are situated on the keyboard to help piaARno position overlays. piARno colorizes the virtual sheet music’s notes as a song is played. Notes that are accurate, accurate up to an incorrect accidental, and otherwise inaccurate are shown in green, yellow, and red, respectively.

Rigby et al. (2020) evaluated piARno with the help of 22 computing students, some with experience in AR/VR and most without music experience. Rigby et al. (2020) first evaluated participants’ ability to recognize notes and play them on a keyboard. Participants were then introduced to piARno and given 10 minutes to practice with it, followed by a questionnaire about their experience with piARno and its usability. Finally, participants were given the initial note recognition test a second time.

Rigby et al. (2020) identified improvements in the average number of notes named and played correctly in the pre- and post-tests. While this result indicates that piARno improves users’ notation literacy and ability to sight-read music, the authors noted that recalling notes’ names may be easier than playing them. Participants gave piARno an above average usability score. They found the system to be enjoyable and felt motivated to increase practice frequency. They generally agreed that piARno helped with reading and playing sheet music and enjoyed keyboard note overlays and the system’s visual note color feedback.

Since participants were more familiar with AR headsets than typical piano students, this may have skewed the study’s results. Participants also found the headset to be bulky and uncomfortable at times, and often found the limited FoV and tracking inaccuracies frustrating. These issues, however, could be resolved with advances in AR/VR technology. Rigby et al. noted that their study was too short to assess how well participants retained what they learned. They recommended repeating the study with different pools of participants and comparing piARno to other forms of instruction.

2.2 Public domain and commercial systems

Hackl’s Magic Keys¹, HoloKeys’ successor, uses Meta’s Passthrough API to display virtual hints and feedback on a performance in combination with a user’s physical piano. Magic Keys released a beta version in August 2022. The Magic Keys Beta supports the use of a virtual piano, a regular piano, or a MIDI keyboard. For non-MIDI instruments, Magic Keys uses a microphone and hand tracking to detect played notes; it struggled with detecting inputs at lower piano volume ranges. Magic Keys’ MIDI mode, which uses MIDI input to detect played notes, proved more reliable.

Magic Keys’ “Play” mode allows a player to select a song. It provides options to change the song’s tempo, an optional metronome, and a “pausing notes” mode. While playing the song, Magic Keys uses the “falling notes” approach to preview key presses, while presenting the song’s recommended fingerings. It includes a mode for practicing chords and scales, along with a lessons mode that was not implemented at the time of review. Following a performance, Magic Keys displays a note correctness percentage, but offers no further feedback to its users. The application, in its current state, seems of limited use for students who need more detailed feedback to improve.

Marceluch’s VRtuos² uses the “falling notes” approach to tutor piano. The application assumes the use of a connected MIDI keyboard. It provides options for loading a song, changing the song’s speed, pausing/playing the song, and scrubbing to any point in the song.

VRtuos features three modes. One plays a song and allows its user to play along without feedback. A second provides a “pausing notes” mode. It also displays red “X” indicators for incorrect notes, green “+” for correct and on time, yellow “/” for correct but late, and blue “-” for correct but early. The third gives feedback like second mode, but without pausing. It also awards a final score based on a user’s performance.

PianoVision³, like Magic Keys, uses Meta’s Passthrough API. It takes input from a virtual piano or a MIDI keyboard, which is connected to a PC. The MIDI keyboard is synced to the headset via the PianoVision companion application or connected directly to the headset. PianoVision uses a MIDI keyboard’s messages to determine when keys are pressed and released. PianoVison’s virtual piano overlay labels keys with their associated note names and provides calibration options.

PianoVision provides options to preview or play a piece, pause, rewind, skip ahead in a song, and customize its operation: e.g., change a piece’s tempo, feature a metronome, enable a “pausing notes” mode, and to play the right-hand part, left-hand part, or both. While performing the song, PianoVision provides the “falling notes” notation and sheet music. The falling notes use color-coding to indicate the hand to use and numbers to indicate fingerings. Text pops up next to the falling notes to indicate timing correctness. The sheet music shows note correctness using green for correct and black for incorrect. Following a performance, PianoVision provides a rating and note correctness percentage. It also provides sample exercises for practicing scales and chords, functioning just like the regular pieces of music.

PianoVision, like other systems in this category, provides only minimal feedback. Also, PianoVision’s mode for one-handed practice, which is supposed to play one hand’s part while its user plays the other, plays at a pace that lags the music, rendering it unusable.

3 Non-AR/VR based systems: P.I.A.N.O.

Rogers et al. (2014)’s P.I.A.N.O. is similar to applications reviewed in the previous section. In lieu of a headset, P.I.A.N.O. uses a projector to project a simulated keyboard onto a wooden board. It provides a “falling notes” display that includes indicators for legato, staccato, trills, grace notes, and correct fingering.

P.I.A.N.O. focuses on teaching users to play piano without the need to read sheet music. It tracks a user’s performance using a MIDI keyboard connected to a desktop computer. It provides feedback by comparing a user’s performance to a MusicXML characterization of the piece.

P.I.A.N.O. provides three learning modes. These correspond to social learning theory’s four steps of learning: attention for listen mode, retention for practice mode, and reproduction and motivation for play mode. Listen mode allows users to listen to a piece of music and watch a visualization. Practice mode provides a “pausing notes” mode. Play mode analyzes the user’s performance and gives real-time feedback on pitch and duration of notes.

Rogers et al. evaluated P.I.A.N.O. using two studies. The first compared a test group’s experiences with P.I.A.N.O.; Synthesia, an educational piano game; and Finale, a traditional sheet music notation software. The study’s 56 participants, none of whom had prior piano experience, worked in 15-min intervals to learn one song of the same difficulty using each of these systems. This study showed that P.I.A.N.O. helped improve note correctness in terms of pitch and duration. Users pressed more incorrect notes than with other systems, but also missed fewer notes, meaning they attempted to play more notes while using P.I.A.N.O.

The authors’ second study compared the performance of 18 users who worked with the three systems over a week’s time. Each participant practiced daily with one of the systems. At the end of the week, participants completed a brief questionnaire, listened to a song once, practiced it for 15 min, and underwent an assessment using their system’s play mode. This study confirmed that P.I.A.N.O. was the most effective system for improving the percentage of correctly played notes, decreasing the percentage of incorrectly played notes, and minimizing missed notes. As part of this study, six experts analyzed P.I.A.N.O.’s impact on performance quality by rating users’ recordings, based on pitch accuracy, duration accuracy, tempo, rhythm, continuity, dynamics, articulation, interpretation, and overall impression. P.I.A.N.O. recordings received the highest ratings for all these attributes.

4 Opportunities for further research

The degree to which current AR/VR headsets can provide feedback on other aspects of performance was explored by the survey’s first author. This work involved the use of a simulation to test the Meta Quest 2 headset’s ability to display images while running a curve-fitting application: a process that could be used to track a performer’s consistency of tempo and dynamics over time relative to a baseline performance. The Meta Quest 2 headset repeatedly froze for several seconds at a time while doing these calculations, an observation that supports authors’ observations about the current limitations of AR/VR technology.

While most of the survey’s authors assert the need for real-time feedback, Percival et al. (2017) (ibid.) argue for post-performance feedback, to avoid distracting an application’s users from performing and impeding their ability to critique themselves during performances. Even so, Percival et al. note that real-time feedback could be useful for inherently mechanical exercises such as scales. From what we have seen, no one has focused on which of these perspectives on automated feedback seems most appropriate.

A related field for potential research involves the use of AR to help people visualize, annotate, and follow scores without the need for physical music and stands. One study from Liu et al. (2019) uses a Microsoft HoloLens AR headset to overlay annotations on a user’s FoV to manipulate a virtual score using voice commands, and a “bouncing ball” to track a performance in real time. A second, Kohen et al. (2020)’s MiXR (Music in Mixed Reality) system uses a Microsoft HoloLens headset and a tablet to present scores and enable their annotation in real-time.

Author contributions

KW was the primary researcher and author of this document. PP reviewed and edited the document. All authors contributed to the article and approved the submitted version.

Funding

East Tennessee State University’s Department of Computing provided funding for publishing this document.

Acknowledgments

The authors thank members in East Tennessee State University’s Department of Computing for their support, notably Dr. Jeffrey Roach, as well as the paper’s reviewers for their contributions to the material’s final form.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

¹https://dominikhackl.com/magickeys/

²https://vrtuos.eu/

³https://www.pianovision.app/

References

Awati, R. (2022). Field of view (FOV). https://www.techtarget.com/whatis/definition/field-of-view-FOV (Accessed March 13, 2023).

Google Scholar

Chow, J., Feng, H., Amor, R., and Wünsche, B. C. (2013). Music education using augmented reality with a head mounted display. AUIC '13 Proc. Fourteenth Australas. User Interface Conf. 139, 73–79. doi:10.5555/2525493.2525501

CrossRef Full Text | Google Scholar

Cuthbert, M., and Ariza, C. (2010). “music21: A toolkit for computer-aided musicology and symbolic music data,” in Proceedings of the 11th International Society for Music Information Retrieval Conference, 637–642. https://www.academia.edu/243058/music21_A_Toolkit_for_Computer-Aided_Musicology_and_Symbolic_Music_Data.

Google Scholar

Hackl, D., and Anthes, C. (2017). “HoloKeys – an augmented reality application for learning the piano,” in Proceedings of the 10th Forum Media Technology and third All Around Audio Symposium (FMT 2017), St. Pölten, Austria, November 2017. https://ceur-ws.org/Vol-2009/fmt-proceedings-2017-paper19.pdf.

Google Scholar

Kohen, S., Elvezio, C., and Feiner, S. (2020). MiXR: A hybrid AR sheet music interface for live performance. 2020 IEEE Int. Symposium Mix. Augmented Real. Adjun. (ISMAR-Adjunct) 1, 76–77. doi:10.1109/ISMAR-Adjunct51615.2020.00035

CrossRef Full Text | Google Scholar

Liu, Z., Adcock, M., and Gardner, H. (2019). “An evaluation of augmented reality music notation,” in Proceedings of the The 17th International Conference on Virtual-Reality Continuum and its Applications in Industry, Brisbane, Australia, November 2019, 1–2. doi:10.1145/3359997.3365748

CrossRef Full Text | Google Scholar

Molloy, W., Huang, E., and Wünsche, B. C. (2019). “Mixed reality piano tutor: A gamified piano practice environment,” in Proceedings of the 2019 International Conference on Electronics, Information, and Communication (ICEIC), Auckland, New Zealand, January 2019, 1–7. https://ieeexplore.ieee.org/abstract/document/8706474.

CrossRef Full Text | Google Scholar

Percival, G., Wang, Y., and Tzanetakis, G. (2007). “Effective use of multimedia for computer-assisted musical instrument tutoring,” in Proceedings of the Emme '07: Proceedings of the international workshop on educational multimedia and multimedia education, Bavaria, Germany, September 2007, 67–76. Available at: doi:10.1145/1290144.1290156

CrossRef Full Text | Google Scholar

Rigby, L., Wünsche, B. C., and Shaw, A. (2020). “piARno - an augmented reality piano tutor,” in Proceedings of the 32nd Australian Conference on Human-Computer Interaction, Sydney, Australia, December 2020. doi:10.1145/3441000.3441039

CrossRef Full Text | Google Scholar

Rogers, K., Röhlig, A., Weing, M., Gugenheimer, J., Könings, B., and Klepsch, M., 2014. “P.I.A.N.O. Faster piano learning with interactive projection: Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces. Dresden, Germany, June 2014, doi:10.1145/2669485.2669514

CrossRef Full Text | Google Scholar

Sarrazin, N. (2016). “Music: Fundamentals and educational roots in the U.S,” in Music and the child (Geneseo, NY, USA: Geneseo: Open SUNY Textbooks). https://open.umn.edu/opentextbooks/textbooks/283.

Google Scholar

Sevilla, G. (2022). Meta captures 90% of VR headset market share. https://www.insiderintelligence.com/content/meta-captures-90-of-vr-headset-market-share (Accessed May 19, 2023).

Google Scholar

W3C Music Notation Community Group, MusicXML version 4.0 final community group report 1 june 2021. https://www.w3.org/2021/06/musicxml40/ (Accessed June 9, 2023).

Google Scholar

Wijaya, F., Tseng, Y. C., Tsai, W. L., Pan, T. Y., and Hu, M. C. (2020). “VR piano learning platform with Leap motion and pressure sensors,” in Proceedings of the 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Atlanta, GA, USA, March 2020, 584–585. doi:10.1109/VRW50115.2020.00143

CrossRef Full Text | Google Scholar