Leveraging Image Analysis for High-Throughput Plant Phenotyping

The complex interaction between a genotype and its environment controls the biophysical properties of a plant, manifested in observable traits, i.e., plant's phenome, which influences resources acquisition, performance, and yield. High-throughput automated image-based plant phenotyping refers to the sensing and quantifying plant traits non-destructively by analyzing images captured at regular intervals and with precision. While phenomic research has drawn significant attention in the last decade, extracting meaningful and reliable numerical phenotypes from plant images especially by considering its individual components, e.g., leaves, stem, fruit, and flower, remains a critical bottleneck to the translation of advances of phenotyping technology into genetic insights due to various challenges including lighting variations, plant rotations, and self-occlusions. The paper provides (1) a framework for plant phenotyping in a multimodal, multi-view, time-lapsed, high-throughput imaging system; (2) a taxonomy of phenotypes that may be derived by image analysis for better understanding of morphological structure and functional processes in plants; (3) a brief discussion on publicly available datasets to encourage algorithm development and uniform comparison with the state-of-the-art methods; (4) an overview of the state-of-the-art image-based high-throughput plant phenotyping methods; and (5) open problems for the advancement of this research field.


INTRODUCTION
The temporal variation in the plant's morphological and functional traits regulated by genotype and the environment plays a crucial role for the development of crops that impact both yield and quality . High-throughput image-based plant phenotyping facilitates the computation of phenotypes by analyzing a large number of plants in short time interval with precision, nullifying the need for time-consuming physical human labor (Das Choudhury et al., 2018). The process is generally non-destructive, allowing the same traits to be quantified repeatedly at multiple times and scales during a plant's life cycle. It is an interdisciplinary research field involving computer science, biology, remote sensing, statistics, and genomics in the effort to link intricate plant phenotypes to genetic expression in order to meet current and emerging issues in agriculture relating to future food security under dwindling natural resources and projected climate variability and change.
Plants are not static but living organisms with constantly increasing complexity in shape, architecture, and appearance. Many plants alter leaf positioning (i.e., phyllotaxy) in response to light signals perceived through the photochrome in order to optimize light interception (Maddonni et al., 2002). In addition to variation in phyllotaxy, growth of individual leaves leads to self-occlusions and leaf crossovers over time posing challenges to image-based plant phenotyping analysis. Other challenges include variation in illumination, cluttered background, and change in zoom levels in the cameras often used to adjust for plant growth.
The state-of-the-art computer vision based approaches mainly focus on the structural and morphological aspects of a plant for computing 2D and 3D phenotypes. However, quantification of physiological processes in plant components, i.e., leaves, stem segments, flowers, fruits, etc., may show differential behavior as a function of environmental stress, and result in different leaf-level photosynthetic activity or carbohydrate content in a plant segment. Similarly, dynamic event-based phenotypes, i.e., identifying important events in plant life cycle, e.g., emergence timing of a new leaf, automated germination detection, timing of emergence detection of fruits and flowers, may provide important insights into the acclimation and adaptation strategies of plants. Thus, we present a novel taxonomy, beyond the well-studied class of structural phenotypes, to illustrate the vast potential of image analysis based methods to assist in understanding of insightful physiological and temporal phenotypes. To achieve this, full range of available imaging modalities and mechanisms must be used. Therefore, we propose a general computational framework for image-based plant phenotyping. We also summarize state-of-the-art image-based plant phenotyping methods with discussion on potential future developments in this field. To systematically evaluate and uniformly compare the methods, benchmark datasets are indispensable. Thus, the paper also provides a comprehensive summary of the publicly available datasets. Figure 1 shows a schematic diagram for image-based highthroughput plant phenotyping platform. It analyses temporal image sequences of a group of plants (belonging to different genotypes) captured by multimodal cameras, i.e., visible light, fluorescent, near infrared, infrared and hyperspectral, from different viewing angles. The plants are imaged at regular intervals under various environmental conditions, e.g., drought, salinity, and thermal (Das Choudhury et al., 2016). Motivated by the high-throughput plant phenotyping platform presented in Fahlgren et al. (2015), we design a more advanced representation to explore computer vision based plant phenotyping algorithms in multiple dimensions, i.e., multimodal, multiview and temporal, regulated by genotypes under various environmental conditions. The image sequences captured by visible light camera are often used to compute structural or morphological phenotypes. Fluorescent, near infrared and infrared images are respectively used to analyze the chlorophyll, water, and temperature content of the plants. Thermal infrared imaging is used as proxy for a plant's temperature to detect differences in stomatal conductance as a measure of the plant response to the water status and transpiration rate for abiotic stress adaptation (Lei et al., 2014b). Hyperspectral imaging is uniquely suited to provide insights into the functional properties of plants, e.g., leaf tissue structure, leaf pigments, and water content (Mahlein et al., 2011) and stress resistance mechanisms (Wahabzada et al., 2016). The high-throughput plant phenotyping platforms presented by Rahaman et al. (2015) and Araus and Cairns (2014) provide emphasis on bridging gap between phenotype-genotype relationship from the molecular point of view.

A TAXONOMY FOR PLANT PHENOTYPES
We present a plant phenotypic taxonomy (Figure 2) which reflects plant phenotypes introduced in recent literature as well as the new challenges that require future research attention. The figure shows that the aboveground plant phenotypes are broadly classified into three categories, namely, structural, physiological, and temporal. The structural phenotypes refer to the morphological attributes of the plants, whereas the physiological phenotypes are related to traits that affect plant processes regulating growth and metabolism. Structural and physiological phenotypes are further divided into two groups: holistic and component. The holistic phenotypes consider the whole plant as a single object and compute its basic geometrical properties, e.g., height of the bounding rectangle to account for plant height, area of the convex-hull to account for plant size (Das Choudhury et al., 2017. Component phenotypes are computed by considering individual components of the plants, i.e., leaves, stem, flower, and fruit. Examples of component phenotypes include leaf length, chlorophyll content of each leaf, stem angle, flower size, and fruit volume (Gage et al., 2017;He et al., 2017;Das Choudhury et al., 2018;Yin et al., 2018;Zhou et al., 2019).
The different parts of plants grow non-uniformly over space and time. This non-uniformity in growth is also regulated by the genotypes. The temporal phenotypes are computed by analyzing a sequence of images. We propose two different types of temporal phenotypes: trajectory-based and eventbased. The structural and physiological phenotypes can be computed from a single image or a sequence of images to take into account of temporal phenotypic characteristics for better genetic variability, e.g., plant growth rate, leaf elongation rate, and trajectories of stem angle (Das Choudhury et al., 2016Sun et al., 2018). It includes change in phenotypic traits as a function of time, time-rate of change in stem width and relative growth rates of different leaves. Eventbased phenotypes are based on specific events in a plant's life cycle, e.g., germination, emergence of a new leaf, flowering, etc. The timing of the events with reference to an origin (say planting date) is important in understanding a plant's behavior (Agarwal, 2017). While the primary phenotypes refer to one specific characteristic of a plant, derived phenotypes are composed of two or more primary phenotypes, i.e., biangular convex-hull area ratio to account for phyllotaxy and plant aspect ratio to provide information on canopy architecture (Das Choudhury et al., 2016).

PUBLIC DATASETS
Development and public dissemination of datasets are vital for research in image based plant phenotyping as it provides the broader computer vision research community access to datasets that they typically do not have the ability to generate. Also, standard datasets provide a common basis to compare the performance of plant phenotyping algorithms. We briefly summarize the datasets below that are publicly disseminated.
• Leaf segmentation challenge (LSC) dataset: LSC dataset is released to advance the state-of-the-art in leaf segmentation, counting, and tracking of rosette plants. The dataset consists of images of two plant species, i.e., Arabidopsis (Arabidopsis thaliana) and tobacco (Nicotiana tabacum) organized into three subsets. Subsets A1 (Ara2012) and A2 (Ara2013)

RECENT ADVANCEMENTS IN IMAGE-BASED PLANT PHENOTYPING
This section provides a brief description of the state-of-the-art methods to compute phenotypes from images organized using our taxonomy, i.e., structural, physiological, and temporal.

2D Phenotypes
The method in Dellen et al. (2015) uses a leaf-shape model to detect each leaf of a tobacco plant. A graph-based tracking algorithm is used to track the detected leaves in a sequence to measure growth rate as a trajectory-based structural phenotype.  It has been suggested that future progress in image-based plant phenotyping will require a combined effort in the domains of image processing for feature extraction and machine learning for data analysis (Tsaftaris et al., 2016). In recent times, machine learning techniques have gained popularity in high-throughput 2D phenotyping, e.g., in detection of branch shaking locations for robotic cherry harvesting (Amatya et al., 2017) and plant growth analysis (Navarro et al., 2016). Deep learning is an emerging field that promises unparalleled results on many data analysis problems. Building on artificial neural networks, deep approaches have many more hidden layers in the network, and hence have greater discriminative and predictive power. A deconvolutional network is used for segmenting the rosette leaves from the background, and then a CNN is used for leaf counting in the method by Aich and Stavness (2017). The method by Atanbori et al. (2018) re-architects four existing deep neural networks to create "Lite" CNN models in an attempt to reduce their parameters while avoiding overfitting for cost-effective solutions in plant phenotyping. The method also introduces a global hyper-parameter for efficient trade-off between parameter size and accuracy of "Lite" CNN models. The method by Pound et al. (2017) uses CNN to identify quantitative trait loci by classifying biologically relevant features such as root tips, leaf and ear tips, and leaf bases to determine root and shoot architecture.
An open source software tool based on CNN called Deep Plant Phenomics is proposed in Ubbens and Stavness (2017) to compute complex phenotypes from plant image sequences. ResNet50, a deep residual neural network, is used in Dobrescu et al. (2017) as a leaf prediction model to count the number of leaves of the rosette plants. In this method, leaf counting is modeled as a direct regression problem. A comprehensive summary of deep learning algorithms for identification, classification, quantification, and prediction of plant stress phenotypes is presented by Singh et al. (2018). Deep learning has applications in a variety of plant phenotyping tasks, e.g., plant stalk count and stalk width (Baweja et al., 2017), leaf counting in rosette plants (Giuffrida et al., 2018;Ubbens et al., 2018), maize tassel counting (Lu et al., 2017), cotton bloom detection , wheat spikes detection (Hasan et al., 2018), and rice panicle segmentation (Xiong et al., 2017).

3D Phenotypes
The plants exhibit increasing architectural complexity with time due to self-occlusions and leaf crossovers, which pose challenges to accurate estimation of 2D component phenotypes, e.g., leaf length, stem angle, and leaf curvature. To overcome these challenges, attempts have been made to reconstruct a 3D model of a plant for accurate estimation of phenotypes. A shape-from-silhouette method is used in Golbach et al. (2016) to reconstruct the 3D model of a tomato seedling from multi-view images. An algorithm for depth imagingbased detection of muskmelon plant for phenotyping in the greenhouse is proposed in Lei et al. (2014a). A detailed review on 3D reconstruction techniques for shoot topology for applications in phenotyping is provided in Gibbs et al. (2016). The method in Liu et al. (2017) uses the structure from motion method to reconstruct the 3D model of a plant using images from multiple side views. The method also deducted the optimal number of images needed for reconstructing a high-quality model. Most cereal crops, e.g., rice, wheat, and maize, have thin ribbon-like architectures with lack of textural surfaces, which limits the success of 3D reconstruction to early growth stages where the distance between the camera and the plant is relatively small facilitating accurate camera calibration.
3D reconstruction of soybean canopies using multisource imaging is presented in Guan et al. (2018). A densitybased spatial clustering of applications with noise removal is used to extract canopy information from the raw 3D point cloud. The method in Scharr et al. (2017) uses a voxelbased volume carving for 3D reconstruction of maize shoot to compute leaf-level traits. Depth cameras capture 3D information from plants to segment leaves and reconstruct plant models (Chéné et al., 2012). A semi-automated software pipeline is developed to reconstruct a 3D plant model from the images captured by a depth camera in McCormick et al. (2016). Standard shoot phenotypes such as shoot height, leaf angle, leaf length, and shoot compactness are measured from 3D plant reconstructions to characterize the shoot architecture. The Microsoft Kinect, originally designed for computer gaming environments, has found applications in 3D plant phenotyping in recent times (Chéné et al., 2012;Paulus et al., 2014;Polder and Hofstee, 2014.) The method in Polder and Hofstee (2014) fuses RGB and depth image captured by Microsoft Kinect sensor to segment muskmelon plants in the greenhouse with cluttered background. The method in Srivastava et al. (2017) provides an algorithm for 3D model reconstruction of wheat plants with occluded leaves, and uses deep learning for drought stress characterization. The remote sensing technology using light detection and ranging (LiDAR) is hypothesized to dominate the future generation of plant phenotyping analysis in outdoor environments Lin (2015). The methods in Sun et al. (2017Sun et al. ( , 2018 compute morphological traits of the cotton plants, e.g., canopy height, projected canopy area, and plant volume, based on 3D model reconstruction of the plants using top view images captured by LiDAR and real-time kinematic global positioning system (RTK-GPS).

Physiological Phenotypes
A comprehensive review on close proximal assessment of functional dynamics of plants using hyperspectral image analysis is provided in Mishra et al. (2017). Water scarcity causes serious crop losses in agriculture. Global climate change and growing population require research advancement in the understanding of plant resistance mechanisms to drought stress for improved crop yield with minimum resource utilization. The method in Römer et al. (2012) uses a simplex volume maximization technique for early drought stress detection using hyperspectral image sequences of barley plants. Recent developments in hyperspectral imaging for assessment of food quality and safety are discussed in Huang et al. (2014), Lu et al. (2017), and Rungpichayapichet et al. (2017). Hyperspectral image analysis has been used to investigate physiological processes, e.g., determination of salt stress in wheat based on vector-wise similarity measurement (Moghimi et al., 2018), early detection of abiotic stresses (Mohd et al., 2018), disease detection to prevent yield losses (Wahabzada et al., 2016), and early yield prediction (González-Sanchez et al., 2014).
A Matlab based software called SK-UTALCA is introduced in Lobos and Poblete-Echeverría (2017) for applications in plant breeding, precision agriculture, crop protection, ecophysiology plant nutrition, and soil fertility by analyzing high-resolution spectral reflectances. A comprehensive overview of the machine learning techniques in the identification, classification, quantification, and prediction of various biotic and abiotic stress phenotypes are provided in Singh et al. (2016). The method by Raza et al. (2014) combines information extracted from thermal and visible light images, and uses support vector machine (SVM) and Gaussian processes (GP) to identify regions of spinach canopy showing a response to soil water deficit. A review of machine learning approaches including back-propagation neural networks, GP, SVM, rotation forest, CNN and LSTM, for crop yield prediction and nitrogen status estimation in precision agriculture is presented in Chlingaryan et al. (2018).

Temporal Phenotypes
The method in Das Choudhury et al. (2018) uses line graphs to represent the trajectories of component phenotypes, i.e., leaf length, integral leaf-skeleton area, mid-leaf curvature, apex curvature and stem angle, as a function of time in order to demonstrate the genetic influence on the temporal variations of these phenotypes. A novel method for plant emergence detection using adaptive hierarchical segmentation and optical flow based tracking is introduced in Agarwal (2017). The efficacy of the method is demonstrated based on experimental analysis on a dataset consisting of images captured at every 2-5 min intervals starting immediately after planting the seeds.

OPEN PROBLEMS
Most of the research in image-based plant phenotyping is focused on structural and mainly holistic physiological phenotypes.
However, timing determination of important events in the life of a plant, e.g., germination, emergence and senescence of leaves, emergence of flowers and fruits, can provide crucial information about plant's growth and response to biotic and abiotic stresses. Timing detection of such events using computer vision techniques remain an important open problem.
Most of the phenotypes have focused on either aboveground or belowground phenotypes. We propose a new category of phenotype, called integrated phenotype, that establishes relationship between above-and belowground phenotypes affected by abiotic and biotic stresses. Primary root growth is inhibited during P-limitation, and its length determines plant's capability to access stored water in the deeper layers of the soil substratum (Prasad et al., 2012;Paez-Garcia et al., 2015). A new integrated phenotype, e.g., the ratio of stem height to primary root length as a function of time, may be investigated for increased yield of stress tolerant crops by enhancing the capacity of the plant for soil exploration and, thus, water and nutrient acquisition. Algorithms to compute them from plant imagery need to be developed.
Efforts have been made in early detection of stress, e.g., drought and mineral (Kim et al., 2011;Chen and Wang, 2014;van Maarschalkerweerd and Husted, 2015). However, future work is required in the investigation of phenotypes that characterize the prorogation of stress as a function time and also categorize stress into different stages, e.g., slight, moderate, extreme, and exceptional. The speed of recovery from the stresses regulated by genotypes is also an open challenge. The method by Uga et al. (2013) demonstrates that controlling root growth angle contributes to drought tolerance. Hence, we propose a new integrated phenotype to quantify speed of drought recovery from different stress levels in relation to controlled root angle.
Extending the manual leaf tracking method in Das Choudhury et al. (2018), we formulate a new problem called visual growth tracking (i.e., tracking of different parts of an object that grows at different rates over time) using plant image sequences for automated growth monitoring of different components of plants, i.e., leaves, flowers, and fruits.
Computing phenotypes from 2D images (e.g., stem angle) are inherently error-prone as they are dependent on accurate camera views. They are deficient too because the plant is a 3D structure and any projection onto 2D plane results in loss of information. Thus, new and innovative 3D phenotypes, both holistic and component, based on accurate reconstruction of the model of a plant must be developed to accurately characterize its properties. 3D plant model reconstruction has been successful for early growth stages with less architectural complexity (Golbach et al., 2016). Advanced 3D plant model reconstruction algorithms are yet to be developed for entire life cycle of plants covering vegetative and reproductive stages for computation of derived phenotypes, e.g., stem cross-section area as a function of plant height at its different stages, 3D leaf area to leaf length ratio, 3D leaf curvature to leaf length ratio, carbohydrate content of stem with respect to stem volume and plant temperature to convex-hull volume ratio at different stages of stressed plants.

AUTHOR CONTRIBUTIONS
SD contributed as the first author in reviewing the literature, compiling the information, preparing the review, and writing the manuscript. AS and TA contributed as co-authors to outline the sections. They critically reviewed the manuscript and provided constructive feedback throughout the process.