AniDriveQA: A VQA Dataset for Driving Scenes with Animal Presence

Rui Wang¹Ruiqi Wang²Hao Hu^1,3*Huai Yu⁴

• ¹The Institute of Computing Technologies, China Academy of Railway Sciences Corporation Limited, Beijing, China
• ²School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an, China
• ³The Center of National Railway Intelligent Transportation System Engineering and Technology, Beijing, China
• ⁴Signal & Communication Research Institute, China Academy of Railway Sciences Corporation Limited, Beijing, China

Animal-involved scenarios present significant challenges for autonomous driving systems due to their rarity, unpredictability, and safety-critical nature. However, existing vision-language datasets for autonomous driving largely neglect these long-tail situations. This paper introduces AniDriveQA, a novel visual question answering dataset specifically designed to evaluate vision-language models in driving scenarios involving animals. It aims to advance the reasoning, perception, and decision-making capabilities of VLMs in rare yet safety-critical autonomous driving scenarios. The dataset is built through a scalable pipeline that collected diverse animal-related traffic scenes from internet videos, filtered and annotated the data using object detection and scene classification models, and generated multi-task VQA labels with a large vision-language model. AniDriveQA encompassed three core task types: scene description, animal description, and driving suggestion. For evaluation, this paper adopted a hybrid scheme that combined classification accuracy for structured tasks with LLM-based scoring for open-ended responses. Extensive experiments on open-source VLMs revealed substantial performance disparities across models and tasks, highlighting the difficulty and diagnostic value of the dataset.