AUTHOR=Wang Zhuang , Sui Jie 

TITLE=Multilevel Attention Residual Neural Network for Multimodal Online Social Network Rumor Detection

JOURNAL=Frontiers in Physics

VOLUME=Volume 9 - 2021

YEAR=2021

URL=https://www.frontiersin.org/journals/physics/articles/10.3389/fphy.2021.711221

DOI=10.3389/fphy.2021.711221

ISSN=2296-424X

ABSTRACT=In recent years, with the rapid rise of social networks such as Weibo and Twitter, multimodal social network rumors have also spread. Unlike traditional unimodal rumor detection, the main difficulty of multimodal rumor detection is how to avoid the generation of noise information while using the complementarity of different modal features. In this paper, we propose a multimodal online social network rumor detection model based on the multi-level attention residuals neural network(MARN). Firstly, the features of text and image were extracted by Bert and Resnet18 respectively, and the cross-attention residual mechanism was used to enhance the representation of images with text vector. Secondly, the enhanced image vector and text vector are concatenated and fused by the self-attention residual mechanism. Finally, the fused image-text vectors are classified into two categories. Among them, the attention mechanism can effectively enhance the image representation and further improve the fusion effect between the image and the text, while the residual mechanism retains the unique attributes of each original modal feature while using different modal features. To assess the performance of MARN model, we conduct experiments on Weibo dataset, and the results show that MARN model outperforms the state-of-the-art models in terms of accuracy and F1 value.