Improving IMU based Human Activty Recognition Using Simulated Multimodal Representations and a MoE Classifier

Lala Shakti Swarup Ray^1,2*Qingxin Xia^3*Vitor Fortes Rey^1,2Kaishun Wu³Paul Lukowicz^1,2

• ¹German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany
• ²Technical University of Kaiserslautern, Kaiserslautern, Rhineland-Palatinate, Germany
• ³Hong Kong University of Science and Technology, Guangzhou, China

The lack of labeled sensor data for Human Activity Recognition (HAR) has driven researchers to synthesize Inertial Measurement Unit (IMU) data from video, utilizing the rich activity annotations available in video datasets. However, current synthetic IMU data often struggles to capture subtle, fine-grained motions, limiting its effectiveness in real-world HAR applications. To address these limitations, we introduce Multi 3 Net+, an advanced framework leveraging cross-modal, multitask representations of text, pose, and IMU data. Building on its predecessor, Multi 3 Net, it uses improved pre-training strategies and a mixture of experts classifier to effectively learn robust joint representations. By leveraging refined contrastive learning across modalities, Multi 3 Net+ bridges the gap between video and wearable sensor data, enhancing HAR performance for complex, fine-grained activities. Our experiments validate the superiority of Multi 3 Net+, showing significant improvements in generating high-quality synthetic IMU data and achieving state-of-theart performance in wearable HAR tasks. These results demonstrate the efficacy of our approach in advancing real-world HAR by combining cross-modal learning with multi-task optimization.