Convolutional Spatio-Temporal Sequential Inference Model for Human Interaction Behavior Recognition

Lizhong Jin^*Rulong FanXiaoling HanXueying Cui

• Taiyuan University of Science and Technology, Taiyuan, China

Research on human action recognition holds significant theoretical and practical importance. It encompasses multiple disciplines, including computer vision, pattern recognition, and machine learning, making it a complex and dynamic challenge to accurately model and recognize human actions. However, research in human interaction behavior recognition faces several challenges, such as correctly modeling human behaviors and distinguishing the sequence of interactions between parties. Current techniques mainly include skeleton sequence-based and RGB videobased recognition models. While recent methods like ActCLR and AutoGCN achieve high accuracy, they often require extensive computational resources. This paper focuses on improving RGB video-based models to enable effective fusion with skeleton point-based modalities. In the fusion of RGB video-based recognition models with skeleton point-based recognition models, we address the challenges of multi-modal data fusion and the excessive parameter size of temporal sequence models. We propose a Convolutional Spatiotemporal Sequence Inference Model (CSSIModel) for recognizing human interaction behaviors. The CSSIModel leverages DINet for extracting features from skeleton data and ResNet-18 for RGB frames. Subsequently, it fuses these features and uses a multi-scale two-dimensional convolutional peak-valley inference module to classify the human behavior categories from the fused features. Experimental results demonstrate that CSSIModel achieves 87.4% accuracy on NTU RGB+D 60 (XSub), 94.1% on NTU RGB+D 60 (XView), 80.5% on NTU RGB+D 120 (XSub), and 84.9% on NTU RGB+D 120 (XSet). These results are competitive with or superior to state-of-the-art models, highlighting the model's strong generalization performance, especially on large-scale datasets. Our method balances accuracy and efficiency, achieving competitive performance while being significantly more lightweight than existing approaches, making it suitable for real-time applications. This method is lightweight, effective, and provides a valuable reference for future research in human behavior recognition.