Multimodal Intelligent Biosensing for Life Assistance Monitoring and Intension Recognition

Danyal Khan¹Haifa Alhasson²Asaad Algarni³Nouf Abdullah Almujally⁴Ahmad Jalal^1*Hui Liu^5*

• ¹Air University, Islamabad, Pakistan
• ²Qassim University, Ar Rass, Saudi Arabia
• ³Northern Border University, Arar, Northern Borders, Saudi Arabia
• ⁴Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia
• ⁵University of Bremen, Bremen, Bremen, Germany

Human activity recognition (HAR) systems are a rapidly evolving field, particularly in applications ranging from healthcare to surveillance, which require both accuracy and interpretability. This study presents a multimodal HAR framework that incorporates both RGB and inertial sensor data to address the limitations of single-modality approaches. Using the LARA and UTD Multimodal Human Action Datasets, our proposed model employs a sequence of advanced pre-processing steps, which include extracting frames and skeletons from RGB data and minimizing noise in signal and segmentation for inertial data, to capture detailed information regarding position, structure, and shape over time. The architecture combines these modalities through a feature fusion layer, then integrates statistical features from inertial sensors with spatial features from RGB-derived skeleton data, resulting in a diverse, highdimensional space. A Gated Recurrent Unit (GRU) network processes this fused feature vector, specifically best suited for sequential data analysis. The model was rigorously evaluated using Stratified Fold cross-validation, achieving 96% accuracy on the UTD multimodal human action (UTD-MHAD) dataset and a similarly high 93% accuracy on the LARA dataset, demonstrating high performance across different activity types. Furthermore, to enhance model transparency and trust, the SHAP interpretability (Explainable AI) method was also applied, which provides an understanding of feature contributions and ensures interpretability in real-world applications. Our proposed model leveraged the benefits of multimodal fusion and handcrafted feature engineering in HAR to contribute to the development of high-precision, interpretable HAR models for complex activity classification tasks.