Introduction
Weeds in wheat fields compete with wheat for light, water, fertilizer, and growth space, and therefore are one of the main biohazards that limit the yield and quality formation of wheat (Shiferaw et al., 2013; Singh et al., 2013). Obtaining weed species and location information quickly and accurately is the first step in precise weed control. Existing methods for detecting weeds in wheat fields based on machine learning are highly dependent on the scale and quality of datasets. In particular, deep learning methods usually require massive training samples. However, Multi-modal and Multi-View Image of weed datasets in natural wheat field are very rare currently.
Existing public weed datasets in crop field are shown in Table 1. Compared with other crops such as cotton and sugar beet, the plant space for wheat crop is smaller. Therefore, the background in wheat field is more complex, and the labeling process is more difficult. In addition, the existing weed datasets still have the following drawbacks. First, the number of samples is small, which is difficult to meet the requirements of deep learning. The mode of information is very single and limited. The existing datasets are constructed based on RGB images, which cannot provide a more complete feature space for weed detection. Especially detecting weeds in wheat field, the features extracted from RGB images are easy to detect broadleaf weeds, but it is difficult to detect weeds with similar appearance to wheat. Studies have shown that because of growth competition, height information is an important parameter to discriminate wheat from weeds (Fahad et al., 2015; Xu et al., 2020), and the fusion of RGB image features and height features has become a new method used to improve the efficiency of weed detection. Finally, the existing datasets are mostly based on vertical views for image acquisition (Wu et al., 2021), which is difficult to be applied to weed detection under a complex field background. Leaf overlap and occlusion will have a great impact on detection.
Table 1
| Dataset | Purpose | Plants | Description | References |
|---|---|---|---|---|
| Dataset of annotated food crops and weed images | Weeds detection and control | Common beet, carrot, zucchini, pumpkin, radish, radish and 8 weed species | 1,118 images with 7,853 XML manually annotated annotations | Sudars et al., 2020 |
| A crop/weed field image dataset | Instance segmentation for weeds and plants | Carrot and common weeds in North Germany | 60 images with annotations | Haug and Ostermann, 2015 |
| 2016 sugar beets dataset | Classification of weeds and plants | Sugar Beet and common weeds in Germany | 4-channel multi-spectral images | Chebrolu et al., 2017 |
| Early-crop-weed | Classification of weeds and plants | tomato, cotton, velvetleaf and black nightshade | 766 field images of crops in early stage | Espejo-Garcia et al., 2020 |
| Deep weeds | Classification of multiple weeds species | Eight nationally significant weed species | 17,509 images with annotations | Olsen et al., 2019 |
| Plant seedlings dataset | Classification of weeds and crops | Maize, wheat, sugar beet and nine weed species | 5,539 images with annotations | Giselsson et al., 2017 |
| CNU weeds dataset | Classification of multiple weeds species | 21 weeds species in the Republic of Korea | 208,477 images with annotations | Vo Hoang et al., 2020 |
| Carrot-weeds | Weeds detection | Carrots and unspecified weeds | 39 images with annotations | Lameski et al., 2017 |
| Lincoln Beet | Weeds detection | Sugar beet and unspecified weeds | 4,402 images with annotations | Salazar-Gomez et al., 2021 |
| Cobbity Wheat | Weeds detection | Wheat and two weed species | 101 images with annotations | Coleman, 2021 |
| Radish Wheat Dataset | Weeds detection | Four growth stages wheat and four weed species | 552 images with annotations | Rayner, 2022 |
| Crop and weed | Instance segmentation for weeds and plants | Maize, the common bean and a variety of weeds | 2,489 images with annotations | Champ et al., 2020 |
Public weed datasets in crop field.
Therefore, we proposed Multi-modal and Multi-view Image Dataset for Weeds Detection in Wheat Field (MMIDDWF) that can be used for deep learning. The dataset contains wheat, broad-leaf weed, and grass weed images of two modes and nine views, and aims to provide a public weed dataset to promote the development of weed detection methods in wheat field.
Value of the data
A multimodal image dataset will be provided for weed detection in open wheat field, including an RGB image and a depth image of the same scene. Compared with a single RGB image, a depth image can provide three-dimensional structure features for weed detection in wheat field, which is helpful to solve the problem of detecting grass weeds.
The dataset also contains multi-view images. Images from nine views can provide a more complete feature space for weed detection in open wheat field, thus helping to solve weed detection problems under a complex background such as leaf occlusion and overlapping.
Materials and methods
Experiment design and image acquisition
Experiments on wheat and weeds were carried out from December 2017 to April 2021 at the demonstration base of the National Engineering and Technology Center for Information Agriculture in Rugao county, Nantong City, Jiangsu province, China. Weeds were not controlled during field management, and seeds of six weed species commonly associated with wheat were randomly sown to simulate weed growth in the open field. Alopecurus aequalis, Poa annua, Bromus japonicus, and E. crusgalli are grass weeds; Amaranthus retroflexus and C. bursa-pastoris are broad-leaf weeds; species composition was similar to that of actual weed species in wheat fields.
The data were collected at the peak of weed occurrence in wheat fields, i.e., in the tillering and jointing stages. The acquisition equipment is shown in Figure 1. The RGB and depth images were acquired using Intel® RealSense™ Depth Camera D415 (Integrated Electronics Corporation, Santa Clara, CA, United States), an RGB-D camera that adopts active infrared stereo vision technology. Infrared stereo cameras generate depth images, and the color sensor generates RGB images, both with a resolution of 1,280 × 720.
Figure 1
(A) Image acquisition equipment, (B) Intel® RealSense™ Depth Camera D415, and (C) TL-IPC44AN-4camera.
RGB and depth field images under natural conditions were obtained during the wheat tillering and jointing stages. Multi-view images were collected with TL-IPC44AN-4(TP-Link Corporation, Shenzhen, China) cameras in four positions at angles of 15 and 30° horizontally. Image collection was conducted from 9 a.m. to 4 p.m. under clear and windless weather conditions. The camera was 70 cm above the crop canopy. Images were transmitted to a computer in real time via USB 3.0.
Image annotation and dataset production
The original depth information is less representative. In particular, feature extraction from depth images with a convolutional neural network generates feature maps of distance rather than geometric structures with physical significance. Therefore, single-channel depth images were transformed to three-channel images by re-encoding the original images to make them more representative and structurally similar to RGB images (Gupta et al., 2014). The three channels of re-encoded images are phase, height above ground, and angle with gravity, and re-encoded images are referred to as PHA images. For the image re-encoding method, refer to Xu et al. (2021). Therefore, the multimodal image dataset in MMIDDWF includes three parts: RGB images, single-channel depth images corresponding to RGB images, and PHA images obtained by recoding depth images. In MMIDDWF, each type includes 1,288 images measuring 500 × 500 pixels. The multi-view image dataset contains 692 images, including 79 RGB images from vertical view, 79 vertical depth images corresponding to vertical view RGB images, and 534 images from eight other views. The details of the dataset are shown in Table 2. LabelImg is employed to annotate broadleaf and grass weeds in images as shown in Figure 2A, and the annotation information of RGB images corresponds to that of depth and PHA images. Figure 2B shows the detection results of weeds in wheat fields based on the multimodal dataset (Xu et al., 2021), which achieved precise detection results in an open wheat field by proposing a dual-channel convolutional neural network and fusing multimodal information.
Table 2
| Wheat varieties information | Planting row spacing | Weeds species information | |
| Experiment design | |||
| Shengxuan No.6, Sumai No.8, Yangmai No.16 and Yangmai No.23 | 20, 35,and 50 cm | Four grass weeds, two broadleaf weeds and other native weeds in wheat fields | |
| Multi-modal image dataset | |||
| Camera | Angle | Type and number of images | Image size |
| Intel® RealSense™ Depth Camera D415 | vertical horizontal plane 90° | 1,288 RGB images and 1,288 PHA images | 500 × 500 |
| Multi-view image dataset | |||
| Camera | Angle | Type and number of images | Image size |
| TL-IPC44AN-4 camera | with the horizontal plane is 15° and 30° | 534 RGB images | 2,560 × 1,440 |
| Intel® RealSense™ Depth Camera D415 | vertical horizontal plane 90° | 79 RGB images and 79 depth images | 720 × 1280 |
Details about MMIDDWF.
Figure 2
(A) Labeling of grass and broadleaf weeds in wheat fields using LabelImg and (B) weed detection result in wheat field.
Funding
This study was supported by the National Natural Science Foundation of China (Grant No: 31871524), Modern Agricultural Machinery Equipment and Technology Demonstration and Promotion of Jiangsu Province (Grant No: NJ2021-58), Primary Research and Development Plan of Jiangsu Province of China (Grant No: BE2021304), and Six Talent Peaks Project in Jiangsu Province (Grant No: XYDXX-049).
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Statements
Data availability statement
The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found below: https://github.com/cocococoxu/MMDWWF.
Author contributions
JN, YZ, KX, and WC designed the research. KX, ZJ, QL, and QX conducted the experiment. KX and ZJ analyzed the data and wrote the manuscript. All authors have read and approved the final version of the manuscript.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
1
ChampJ.Mora-FallasA.GoëauH.Mata-MonteroE.BonnetP.JolyA. (2020). Instance segmentation for the fine detection of crop and weed plants by precision agricultural robots. Appl. Plant Sci.8, e11373. 10.1002/aps3.11373
2
ChebroluN.LottesP.SchaeferWinterhalter, Burgard Stachniss. (2017). Agricultural robot dataset for plant classification, localization and mapping on sugar beet fields. Int. J. Robot Res. 36, 1045–1052. 10.1177/0278364917720510
3
ColemanG. (2021). 20201014 - Cobbity Wheat BFLY. Weed-AI. Available online at: https://weed-ai.sydney.edu.au/datasets/73468c19-b098-406a-86fa-df172caaec16.
4
Espejo-GarciaB.MylonasN.AthanasakosL.FountasS.VasilakoglouI. (2020). Towards weeds identification assistance through transfer learning. Comput. Electron. Agric.171, 105306. 10.1016/j.compag.2020.105306
5
FahadS.HussainS.ChauhanB. S.SaudS.WuC.HassanS.et al. (2015). Weed growth and crop yield loss in wheat as influenced by row spacing and weed emergence times. Crop Protect.71, 101–108. 10.1016/j.cropro.2015.02.005
6
GiselssonT. M.JørgensenR. N.JensenP. K.DyrmannM.MidtibyH. S. (2017). A public image database for benchmark of plant seedling classification algorithms. arXiv. 1711.05458.
7
GuptaS.GirshickR.ArbelaezP.MalikJ. (2014). “Learning rich features from RGB-D images for object detection and segmentation,”13th European Conference on Computer Vision (ECCV). Zurich, Switzerland (Cham: Springer). p. 345–360.
8
HaugS.OstermannJ. (2015). “A crop/weed field image dataset for the evaluation of computer vision based precision agriculture tasks,”13th European Conference on Computer Vision (ECCV). Zurich, Switzerland (Cham: Springer). p.105–116.
9
LameskiP.ZdravevskiE.TrajkovikV.KulakovA. (2017). “Weed detection dataset with RG, images taken under variable light conditions,”International Conference on IC, Innovations. (Cham: Springer). p.112–119.
10
OlsenA.KonovalovD. A.PhilippaB.RiddP.WoodJ. C.JohnsJ.et al. (2019). deepweeds: a multiclass weed species image dataset for deep learning. Sci. Rep.9. 10.1038/s41598-018-38343-3
11
RaynerG. (2022). RadishWheatDataset. Weed-AI. Available online at: https://weed-ai.sydney.edu.au/datasets/8b8f134f-ede4-4792-b1f7-d38fc05d8127.
12
Salazar-GomezA.DarbyshireM.GaoJ.SklarE. I.ParsonsS. (2021). Towards practical object detection for weed spraying in precision agriculture. arXiv. 2109.11048. 10.48550/arXiv.2109.11048
13
ShiferawB.SmaleM.BraunH. J.DuveillerE.MurichoG. (2013). Crops that feed the world 10. Past successes and future challenges to the role played by wheat in global food security. Food Sec.5. 10.1007/s12571-013-0263-y
14
SinghV.SinghaH.RaghubanshiA. S. (2013). Competitive interactions of wheat with Phalaris minor or Rumex dentatus: a replacement series study. Pans Pest Articles News Summaries.59, 245–258. 10.1080/09670874.2013.845320
15
SudarsK.JaskoJ.NamatevsI.OzolaL.BadaukisN. (2020). Dataset of annotated food crops and weed images for robotic computer vision control. Data in Brief.31, 105833. 10.1016/j.dib.2020.105833
16
Vo HoangT.Gwang-HyunY.Dang ThanhV.Jin-YoungK. (2020). Late fusion of multimodal deep neural networks for weeds classification. Comput. Electron. Agric.175. 10.1016/j.compag.2020.105506
17
WuZ.ChenY.ZhaoB.KangX.DingY. (2021). Review of weed detection methods based on computer vision. Sensors.21, 3647. 10.3390/s21113647
18
XuK.LiH.CaoW.ZhuY.NiJ. (2020). Recognition of Weeds in Wheat Fields Based on the Fusion of RGB Images and Depth Images. IEEE Access.1–1. 10.1109/ACCESS.2020.3001999
19
XuK.ZhuY.CaoW.JiangX.JiangZ.LiSet al. (2021). Multi-modal deep learning for weeds detection in wheat field based on RGB-D images. Front. Plant Sci.12. 10.3389/fpls.2021.732968
Summary
Keywords
multi-modal image, multi-view image, grass weeds detection, wheat field, machine learning, deep learning
Citation
Xu K, Jiang Z, Liu Q, Xie Q, Zhu Y, Cao W and Ni J (2022) Multi-modal and multi-view image dataset for weeds detection in wheat field. Front. Plant Sci. 13:936748. doi: 10.3389/fpls.2022.936748
Received
05 May 2022
Accepted
20 July 2022
Published
22 August 2022
Volume
13 - 2022
Edited by
Yuzhen Lu, Mississippi State University, United States
Reviewed by
Borja Espejo-Garca, Agricultural University of Athens, Greece; Fengying Dang, Michigan State University, United States; Chetan Badgujar, Kansas State University, United States
Updates
Copyright
© 2022 Xu, Jiang, Liu, Xie, Zhu, Cao and Ni.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Jun Ni nijun@njau.edu.cn
†These authors have contributed equally to this work and share first authorship
This article was submitted to Sustainable and Intelligent Phytoprotection, a section of the journal Frontiers in Plant Science
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.