AUTHOR=Liu Shuqi , Wang Gang , Song Yong , Huang Jinxiang , Huang Yiqian , Zhou Ya , Wang Shiqiang 

TITLE=SiamEFT: adaptive-time feature extraction hybrid network for RGBE multi-domain object tracking

JOURNAL=Frontiers in Neuroscience

VOLUME=Volume 18 - 2024

YEAR=2024

URL=https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2024.1453419

DOI=10.3389/fnins.2024.1453419

ISSN=1662-453X

ABSTRACT=Integrating RGB and Event (RGBE) multi-domain information obtained by high-dynamic-range and temporal-resolution event cameras has been considered an effective scheme for robust object tracking. However, existing RGBE tracking methods have overlooked the unique spatio-temporal features over different domains, leading to object tracking failure and inefficiency, especally for objects against complex backgrounds. To address this problem, we propose a novel tracker based on adaptive-time feature extraction hybrid networks, namely Siamese Event Frame Tracker (SiamEFT), which focuses on the effective representation and utilization of the diverse spatiotemporal features of RGBE. We first design an adaptive-time attention module to aggregate event data into frames based on adaptive-time weights to enhance information representation.Subsequently, the SiamEF module and cross-network fusion module combining artificial neural networks and spiking neural networks hybrid network are designed to effectively extract and fuse the spatio-temporal features of RGBE. Extensive experiments on two RGBE datasets (VisEvent and COESOT) show that the SiamEFT achieves a success rate of 0.456 and 0.574, outperforming the state-of-the-art competing methods and exhibiting a 2.3-fold enhancement in efficiency. These results validate the superior accuracy and efficiency of SiamEFT in diverse and challenging scenes.