ORIGINAL RESEARCH article

Front. Comput. Sci.

Sec. Human-Media Interaction

Volume 7 - 2025 | doi: 10.3389/fcomp.2025.1570249

This article is part of the Research TopicEmbodied Perspectives on Sound and Music AIView all 7 articles

BrAIn Jam: Neural Signal-Informed Adaptive System for Drumming Collaboration with an AI-Driven Virtual Musician

Provisionally accepted
Torin  HopkinsTorin Hopkins1,2*Ruojia  SunRuojia Sun1Suibi Che Chuan  WengSuibi Che Chuan Weng1Shih-Yu  MaShih-Yu Ma1James  CrumJames Crum1Leanne  HirshfieldLeanne Hirshfield1Ellen Yi-Luen  DoEllen Yi-Luen Do1
  • 1University of Colorado Boulder, Boulder, United States
  • 2National University of Singapore, Singapore, Singapore

The final, formatted version of the article will be published soon.

Collaboration between improvising musicians requires a dynamic exchange of subtleties in human musical communication. Many musicians can intuit this information, however, translating this knowledge to embodied computer-driven musicianship systems-be they robotic or virtual musicians-remains an ongoing challenge. Methods of communicating musical information to computer-driven musicianship systems have traditionally been accomplished using an array of sensing techniques such as MIDI, audio, and video. However, utilizing musical information from the human brain has only been explored in limited social and musical contexts. This paper presents "BrAIn Jam," utilizing functional near-infrared spectroscopy to monitor human drummers' brain states during musical collaboration with an AI-driven virtual musician. Our system includes a real-time algorithm for preprocessing and classifying brain data, enabling dynamic AI rhythm adjustments based on neural signal processing. Our formative study is conducted in two phases: 1) training individualized machine learning models using data collected during a controlled experiment, and 2) using these models to inform an embodied AI-driven virtual musician in a real-time improvised drumming collaboration. In this paper, we discuss our experimental approach to isolating a network of brain areas involved in music improvisation with embodied AI-driven musicians, a comparative analysis of several machine learning models, and post hoc analysis of brain activation to corroborate our findings. We then synthesize findings from interviews with our participants and report on the challenges and opportunities for designing music systems with functional near-infrared spectroscopy, as well as the applicability of other physiological sensing techniques for human and AI-driven musician communication.

Keywords: FNIRS (functional Near-InfraRed Spectroscopy), Brain-Computer Interfaces, embodied AI, Music, Neuroscience, machine learning, music improvisation, human-computer interaction

Received: 03 Feb 2025; Accepted: 04 Jul 2025.

Copyright: © 2025 Hopkins, Sun, Weng, Ma, Crum, Hirshfield and Do. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Torin Hopkins, University of Colorado Boulder, Boulder, United States

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.