AUTHOR=Caetano Arthur , Aponte Alejandro , Sra Misha 

TITLE=A design toolkit for task support with mixed reality and artificial intelligence

JOURNAL=Frontiers in Virtual Reality

VOLUME=Volume 6 - 2025

YEAR=2025

URL=https://www.frontiersin.org/journals/virtual-reality/articles/10.3389/frvir.2025.1536393

DOI=10.3389/frvir.2025.1536393

ISSN=2673-4192

ABSTRACT=Efficient performance and acquisition of physical skills, from sports techniques to surgical procedures, require instruction and feedback. In the absence of a human expert, Mixed Reality Intelligent Task Support (MixITS) can offer a promising alternative. These systems integrate Artificial Intelligence (AI) and Mixed Reality (MR) to provide realtime feedback and instruction as users practice and learn skills using physical tools and objects. However, designing MixITS systems presents challenges beyond engineering complexities. The complex interactions between users, AI, MR interfaces, and the physical environment create unique design obstacles. To address these challenges, we present MixITS-Kit—an interaction design toolkit derived from our analysis of MixITS prototypes developed by eight student teams during a 10-week-long graduate course. Our toolkit comprises design considerations, design patterns, and an interaction canvas. Our evaluation suggests that the toolkit can serve as a valuable resource for novice practitioners designing MixITS systems and researchers developing new tools for human-AI interaction design.