AUTHOR=Xu Ke , Zhu Yan , Cao Weixing , Jiang Xiaoping , Jiang Zhijian , Li Shuailong , Ni Jun 

TITLE=Multi-Modal Deep Learning for Weeds Detection in Wheat Field Based on RGB-D Images

JOURNAL=Frontiers in Plant Science

VOLUME=Volume 12 - 2021

YEAR=2021

URL=https://www.frontiersin.org/journals/plant-science/articles/10.3389/fpls.2021.732968

DOI=10.3389/fpls.2021.732968

ISSN=1664-462X

ABSTRACT=Single-modal images carry limited information for object representation, and RGB
images fail to detect gramineous weeds in wheat fields because of their similarity to
wheat in shape. We propose a framework based on multi-modal information fusion for
accurate detection of weeds in wheat fields in a natural environment, overcoming the
limitation of single modality in weeds species recognition. A three-channel network
combining modality independence and complementarity is designed based on the
concept of multi-scale object detection, and ensemble learning is performed by weight
assignment at the decision level. An experiment shows a mean average precision (mAP)
of 36.1% for gramineous weeds and 42.9% for broadleaf weeds, and the overall
detection precision, as indicated by intersection over ground truth ( IoG), is 89.3%,
with weights of RGB and depth images at  =0.4 and =0.3. The results suggest that
this model can accurately detect the dominant species of weeds in wheat fields, and that
multi-modal fusion can effectively improve model detection performance.