# OPTICAL APPROACHES TO CAPTURE PLANT DYNAMICS IN TIME, SPACE, AND ACROSS SCALES

EDITED BY : Eetu Puttonen, Norbert Pfeifer, Alexander Bucksch and András Zlinszky PUBLISHED IN : Frontiers in Plant Science

#### Frontiers Copyright Statement

© Copyright 2007-2018 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.

The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.

Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.

Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.

As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.

All copyright, and all rights therein, are protected by national and international copyright laws.

The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use. ISSN 1664-8714 ISBN 978-2-88945-543-0 DOI 10.3389/978-2-88945-543-0

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# OPTICAL APPROACHES TO CAPTURE PLANT DYNAMICS IN TIME, SPACE, AND ACROSS SCALES

Topic Editors:

Eetu Puttonen, National Land Survey of Finland, Finland Norbert Pfeifer, Technische Universität Wien, Austria Alexander Bucksch, University of Georgia, United States András Zlinszky, Centre for Ecological Research, Hungarian Academy of Sciences, Hungary

Side view of the 3D model of an olive tree produced by terrestrial laser scanning for the purpose of following overnight movement. The colors were produced by RGB blending, with the number of points in a search neighbourhood scaled to the red channel, point amplitude negatively scaled to the green channel, and height above ground negatively scaled to the blue channel. The circle on the bottom right is an artificial target sphere used for co-registration of the individual scans. Image: Dr. Zlinszky.

Quantifying temporal changes in plant geometry as a result of genetic, developmental, or environmental causes is essential to improve our understanding of the structure and function relationships in plants. Over the last decades, optical imaging and remote sensing developed fundamental working tools to monitor and quantify our environment and plants in particular. Increased efficiency of methods lowered the barrier to compare, integrate, and interpret the optically obtained plant data across larger spatial scales and across scales of biological organization. In particular, acquisition speed at high resolutions reached levels that allow capturing the temporal dynamics in plants in three dimensions along with multi-spectral information beyond human visual senses. These advanced imaging capabilities have proven to be essential to detect and focus on analyzing temporal dynamics of plant geometries.

The focus of this Research Topic is on optical techniques developed to study geometrical changes at the plant level detected within the wavelength spectrum between near-UV to near infrared. Such techniques typically involve photogrammetric, LiDAR, or imaging spectroscopy approaches but are not exclusively restricted to these. Instruments operating within this range of wavelengths allow capturing a wide range of temporal scales ranging from sub-second to seasonal changes that result from plant development, environmental effects like wind and heat, or genetically controlled adaption to environmental conditions. The Research Topic covered a plethora of methodological approaches as suggestions for best practices in the light of a particular research question and to a wider view to different research disciplines and how they utilize their state-of-the-art techniques in demonstrating potential use cases across different scales.

Citation: Puttonen, E., Pfeifer, N., Bucksch, A., Zlinszky , A., eds. (2018). Optical Approaches to Capture Plant Dynamics in Time, Space, and Across Scales. Lausanne: Frontiers Media. doi: 10.3389/978-2-88945-543-0

# Table of Contents

*05 Editorial: Optical Approaches to Capture Plant Dynamics in Time, Space, and Across Scales*

Eetu Puttonen, Alexander Bucksch, András Zlinszky and Norbert Pfeifer


Rui Xu, Changying Li, Andrew H. Paterson, Yu Jiang, Shangpeng Sun and Jon S. Robertson


Haiyan Cen, Haiyong Weng, Jieni Yao, Mubin He, Jingwen Lv, Shijia Hua, Hongye Li and Yong He


Shangpeng Sun, Changying Li, Andrew H. Paterson, Yu Jiang, Rui Xu, Jon S. Robertson, John L. Snider and Peng W. Chee

*159 Considerations for Achieving Cross-Platform Point Cloud Data Fusion Across Different Dryland Ecosystem Structural States*

Tyson L. Swetnam, Jeffrey K. Gillan, Temuulen T. Sankey, Mitchel P. McClaran, Mary H. Nichols, Philip Heilman and Jason McVay

Jon Robertson

# Editorial: Optical Approaches to Capture Plant Dynamics in Time, Space, and Across Scales

Eetu Puttonen1,2 \*, Alexander Bucksch3,4,5, András Zlinszky <sup>6</sup> and Norbert Pfeifer <sup>7</sup>

<sup>1</sup> Remote Sensing and Photogrammetry, Finnish Geospatial Research Institute, Kirkkonummi, Finland, <sup>2</sup> Center of Excellence in Laser Scanning Research, Masala, Finland, <sup>3</sup> Warnell School of Forestry and Natural Resources, University of Georgia, Athens, GA, United States, <sup>4</sup> Institute of Bioinformatics, University of Georgia, Athens, GA, United States, <sup>5</sup> Department of Plant Biology, University of Georgia, Athens, GA, United States, <sup>6</sup> Balaton Limnological Institute, Centre for Ecological Research, Hungarian Academy of Sciences, Tihany, Hungary, <sup>7</sup> Department of Geodesy and Geoinformation, Technische Universität Wien, Vienna, Austria

Keywords: plant growth, plant development, geometric changes in plants, seasonal changes in vegetation structure, change detection

**Editorial on the Research Topic**

#### **Optical Approaches to Capture Plant Dynamics in Time, Space, and Across Scales**

#### Edited and reviewed by:

Yann Guédon, Centre de Coopération Internationale en Recherche Agronomique pour le Développement (CIRAD), France

> \*Correspondence: Eetu Puttonen eetu.puttonen@nls.fi

#### Specialty section:

This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science

> Received: 28 March 2018 Accepted: 24 May 2018 Published: 11 June 2018

#### Citation:

Puttonen E, Bucksch A, Zlinszky A and Pfeifer N (2018) Editorial: Optical Approaches to Capture Plant Dynamics in Time, Space, and Across Scales. Front. Plant Sci. 9:791. doi: 10.3389/fpls.2018.00791 The quest to decipher the phenotype to genotype relationships involves the quantification of plant adaption to the environment at various scales to solve some of the world's most pressing problems (Bucksch et al., 2017). The role of phenotype to genotype relationships within initiatives to increase crop yields for food, fiber, and fuel and to improve prediction of future environmental conditions (IPCC, 2014) is central to the function and well-being of societies world-wide. Quantifying plant morphology over time captures the dynamic structure and function relationships of how plants interact with and respond to environmental stimuli (Balduzzi et al., 2017). A deeper understanding of plant adaption can be achieved if technologies to monitor plants across spatial and biological scales are further developed. Spanning biological scales from the community over the organismal down to the molecular scale is inherently coupled to spatial scales. A wide variety of technological concepts are facilitating the revival of the science of plant morphology and anatomy (Ledford, 2018).

Over the last decades, optical imaging and remote sensing developed the fundamental working tools to monitor and quantify our environment and plants in particular. Satellite imaging increased its spatial, temporal, and spectral resolution to levels where individual trees in forests can be partly identified. However, it remains a challenge to develop pipelines that quantify the traits of plants and their processes from the level of plant populations down to the detail obtained with microscopy. The reason for the pipeline challenge is the heterogeneity of the obtained data ranging from unorganized point clouds over surface models to rasterized imaging and hyperspectral data. For example, airborne methods increased their importance in ecology by measuring aggregate tree traits such as crown width or stem diameter to study community composition (Wieser et al., 2017). However, combining airborne and highresolution terrestrial data is non-trivial despite improved automatization and higher resolutions of measured plant traits. Unmanned aerial vehicles (UAV) equipped with cameras and multispectral sensors allow the fusion of temporal, spatial, and spectral data to record plant dynamics on ecosystem level at resolutions that also allow the quantification of individual plant morphology.

In particular, the advances in algorithms to handle heterogenous data of various sensors revealed details of plant adaptation to discover the genes controlling adaptation mechanisms. The wealth of options presents a new challenge in testing and selecting an appropriate approach that scales well with the research questions. Making modern technology applicable to the hypothesis driven science process requires more than learning a few techniques. In the light of a particular hypothesis it requires also resources to acquire, build, and refactor equipment and software. Here, a wider view to different research disciplines and how they utilize their state-of-the-art techniques can be most beneficial in demonstrating potential use cases. Ideally, this will lead to joint use of best practices in technology innovation and discovery in the plant sciences.

The Topic received a diverse set of 11 original research manuscripts that cover a wide array of applications across different scales. On the spatial scale, Scharr et al. present a fast imaging approach to perform volumetric 3D carving of plant roots on submillimeter level. Sun et al. apply LiDAR for high-throughput phenotyping on cotton plants while Lin et al. use LiDAR mapping to investigate how Hallé's models of tree architecture reflect in real data. Xu et al. apply Structurefrom-Motion to images collected with unmanned aerial vehicle in cotton fields to detect the exact flowering time and the precise locations of cotton balls using convolutional neural networks. Swetnam et al. link individual plant morphology to plant populations by fusing point clouds collected with groundbased, UAV mounted, and airborne systems to study different structural states of a dryland ecosystem.

Temporal dynamics were quantified by Zlinszky et al. who measured variation in nocturnal branch movements between different species with short interval LiDAR. Similarly, Herrero-Huerta et al. studied the movement of leaves in Calathea roseopicta under different lighting conditions with a LiDAR system. The outcome of these investigations challenges our view of plants as passive, static organisms, and shows that they are capable of short-term changes in shape at the whole plant level.

While not completely non-invasive, Doan et al. published a first low-cost imaging system to detect formation of free amino acid groups on maize roots to elucidate exudate responses. In doing so, a special, ninhydrin-injected paper was monitored over a 3-week growth period. Jiang et al. reported an application using low-cost consumer RGB-D cameras to monitor and quantify the growth of cotton canopies over a 3-month period.

In addition, two manuscripts harnessed spectral information to study dynamic responses in plant leaves outside the visible region. Cen et al. detected photosynthetic fingerprints of citrus Huanglongbing (so-called "greening") disease with fluorescence

#### REFERENCES

Bucksch, A., Atta-Boateng, A., Azihou, A. F., Battogtokh, D., Baumgartner, A., Binder, B. M., et al. (2017). Morphological plant modeling: unleashing geometric and topologic potential within the plant sciences. Front. Plant Sci. 8:900. doi: 10.3389/fpls.2017.00900

spectroscopy by combining imaging of specific fluorescence properties with advanced image analysis. Junttila et al. estimated leaf equivalent water thickness using combinations of different LiDAR wavelengths. Their results showed significant correlation with the standard laboratory reference measurements. These findings demonstrate the significant potential in expanding multi-wavelength laser scanning measurements to whole forests to accurately monitor ecophysiological parameters.

Our Research Topic proves that optical technologies have reached a level of precision, reliability and detail that is sufficient to study minute physiological processes in plants. In addition to documenting plant status at various levels, imaging, and scanning methods allow following processes of adaptation, movement or disease in real time as they happen. In addition to serving as early detection systems (Cen et al.; Junttila et al.) for management purposes, these methods allow new insights into plant function. One particular example for this is the observed short-term periodicity in plant movement and trunk diameter, which suggests a new approach to water transport in trees (Zlinszky and Barford, 2018).

The Research Topic covered a plethora of methodological approaches as suggestions for best practices in the light of a particular research question. As such, the Research Topic collected papers that demonstrate how technology development and scientific discovery in the plant sciences can accelerate each other toward deciphering the adaption of plants as part of the quest to reveal the phenotype to genotype map. The future challenge will be to extend the multi-scale measurement of static traits to multi-scale measurement of chemical processes within and between plants. For example, how do diseases or ripening processes affect the community structure or how does microclimate fluctuation of single plants affect overall edible yield? How do shape changes in canopies relate to the hydraulic properties at tissue level? We believe that the optical methods in this Research Topic will become openly available such that all researchers across the plant sciences will be capable to perform research at this exciting frontier.

#### AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

## FUNDING

AZ received funding under the grant No. OTKA PD 115833.

Balduzzi, M., Binder, B. M., Bucksch, A., Chang, C., Hong, L., Iyer-Pascuzzi, A. S. et al. (2017). Reshaping plant biology: qualitative and quantitative descriptors for plant morphology. Front. Plant Sci. 8:117. doi: 10.3389/fpls.2017.00117

IPCC (2014). "Climate Change 2014: synthesis report," in Contribution of Working Groups I, II and III to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change [Core Writing Team], eds R. K. Pachauri and L. A. Meyer (Geneva: IPCC), 151. Available online at: https://www.ipcc.ch/index.htm


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Puttonen, Bucksch, Zlinszky and Pfeifer. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Recruiting Conventional Tree Architecture Models into State-of-the-Art LiDAR Mapping for Investigating Tree Growth Habits in Structure

#### Yi Lin<sup>1</sup> \*, Miao Jiang<sup>2</sup> , Petri Pellikka<sup>3</sup> and Janne Heiskanen<sup>3</sup>

<sup>1</sup> Beijing Key Lab of Spatial Information Integration and Its Applications, School of Earth and Space Sciences, Institute of Remote Sensing and GIS, Peking University, Beijing, China, <sup>2</sup> Institute of Mineral Resources Research, China Metallurgical Geology Bureau, Beijing, China, <sup>3</sup> Department of Geosciences and Geography, University of Helsinki, Helsinki, Finland

#### Edited by:

András Zlinszky, Institute of Ecology Research Center, Hungarian Academy of Sciences, Hungary

#### Reviewed by:

Christophe Pradal, Agricultural Research Center for Development, France Evelyne Costes, INRA UMR Amélioration génétique et adaptation des plantes méditerranéennes et tropicales, France

> \*Correspondence: Yi Lin yi.lin@pku.edu.cn

#### Specialty section:

This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science

> Received: 30 September 2017 Accepted: 05 February 2018 Published: 20 February 2018

#### Citation:

Lin Y, Jiang M, Pellikka P and Heiskanen J (2018) Recruiting Conventional Tree Architecture Models into State-of-the-Art LiDAR Mapping for Investigating Tree Growth Habits in Structure. Front. Plant Sci. 9:220. doi: 10.3389/fpls.2018.00220 Mensuration of tree growth habits is of considerable importance for understanding forest ecosystem processes and forest biophysical responses to climate changes. However, the complexity of tree crown morphology that is typically formed after many years of growth tends to render it a non-trivial task, even for the state-of-the-art 3D forest mapping technology—light detection and ranging (LiDAR). Fortunately, botanists have deduced the large structural diversity of tree forms into only a limited number of tree architecture models, which can present a-priori knowledge about tree structure, growth, and other attributes for different species. This study attempted to recruit Hallé architecture models (HAMs) into LiDAR mapping to investigate tree growth habits in structure. First, following the HAM-characterized tree structure organization rules, we run the kernel procedure of tree species classification based on the LiDAR-collected point clouds using a support vector machine classifier in the leave-one-out-for-cross-validation mode. Then, the HAM corresponding to each of the classified tree species was identified based on expert knowledge, assisted by the comparison of the LiDAR-derived feature parameters. Next, the tree growth habits in structure for each of the tree species were derived from the determined HAM. In the case of four tree species growing in the boreal environment, the tests indicated that the classification accuracy reached 85.0%, and their growth habits could be derived by qualitative and quantitative means. Overall, the strategy of recruiting conventional HAMs into LiDAR mapping for investigating tree growth habits in structure was validated, thereby paving a new way for efficiently reflecting tree growth habits and projecting forest structure dynamics.

Keywords: tree growth habits, Hallé architecture model (HAM), light detection and ranging (LiDAR), tree species classification, morphological features

# INTRODUCTION

Trees play a fundamental role in maintaining forest ecosystems by adapting their biophysical, biochemical or physiological characteristics to the integrity of biotic and abiotic environmental influences (Saxe et al., 2001). The adaptation effects apparently show in their structures and growth habits (Creber and Chaloner, 1984). For example, the contemporary structures of trees mirror

**8**

the inherent rules of their various organic components being organized; by following these rules, people estimated the function capacities of trees such as crown photosynthesis efficacy (Lovelock et al., 2006). As regards to the dynamical growth habits of trees such as their continuous or rhythmic growth modes, tree habit modeling facilitates decisions-making in advance, which is favorable for sustainable forest managements (Pretzsch et al., 2006). Hence, learning tree growth habits in structure has considerable implications for understanding from how single trees cope with diversified environmental stresses to how integral forests function in multiplex terrestrial processes (Pinard et al., 1999). This task is especially interesting for the trees growing in the boreal environment, since characterizing tree growth habits in structure in such a condition that is enduring quicker warming than global warming (Shepherd, 2016) facilitates projecting how trees will respond to the climatic scenarios in the future.

However, mensuration of tree growth habits in structure across large areas is a non-trivial task. This is due to the difficulty of measuring and modeling tree crown structures that comprise leaves and branches of varying shape forms, growth modes, and organization layouts at the fine scales. In addition, the conventional approaches, e.g., field investigation and estimation of tree growths by comparing multi-temporal canopy surface models (Yu et al., 2006), are far from enough for accurately mapping crown structure and dynamics. Therefore, new techniques with higher potentials of characterizing tree growth habits in structure are widely and urgently demanded.

Remote sensing (RS) supplies a more efficient solution plan to accomplish this task (Heinzel and Koch, 2012). The stateof-the-art RS technology—static terrestrial laser scanning (TLS), with the merits of 3D high-density sampling of objects, seems to be the best choice to date. Based on TLS data, an LiDAR data to tree Architecture (L-Architect) architecture model has been designed to provide a feasible framework for quantifying the spatial distributions of tree components and explicitly describing 3D tree architectures (Côté et al., 2011). From TLS-recorded 3D points, quantitative retrievals of crown structure and foliage assemblage situations have also been tried (Yang et al., 2013). In a broader sense, TLS has also been enthusiastically attempted for deriving various structural parameters of trees, including leaf area density (Hosoi and Omasa, 2007), tree height and crown diameter (Moorthy et al., 2011; Lin et al., 2012), and tree biomass (Calders et al., 2015). However, although TLS has proved to be able to characterize tree structures well, it is difficult to derive tree growth habits directly from TLS point clouds due to the complex structures and organizations of leaves and branches. Eitel et al. (2013) placed a TLS system at a fixed location for monitoring forest plot dynamics, but the time span was too short to infer the growth habits in structure. Worse still, TLS is principally restricted by its limited coverage; consequently, TLS is more assumed to supply the reference samples for calibrating large-area forest canopy structure measurements (Côté et al., 2012).

As another technical branch of light detection and ranging (LiDAR), airborne laser scanning (ALS) that is applicable for large-scale forest mapping can solve this shortage to some extent. In retrospect, the usages of ALS for forest inventory had a longer history than TLS (Yang et al., 2013), while this history showed that applying ALS for forest inventories still needs to consider tree architectures when mapping accuracies are concerned. This is evidenced by the fact that numerous ALS-based forest researches were conducted based on presumptions of diversified tree crown archetypes, as reviewed by Calders et al. (2013). For example, Koetz et al. (2007) used a hemi-ellipsoidal tree crown archetype, and Ferraz et al. (2012) assumed an ellipsoidal one. The results indicated that geometric archetypes of tree crowns did help in forest investigations, yet the previous selections of archetypes were mostly based on the external shape outlines of tree crowns represented by 3D ALS points (Koetz et al., 2007; Ferraz et al., 2012; Calders et al., 2013). That is, almost neither geometries nor topologies of crown-internal structures such as the layouts of leaves and branches were examined, and most of the previous studies have not considered tree architectures in a real sense. This is caused by that the relatively low sampling density of airborne LiDAR is the most kernel factor restricting its performance on detailed representations of crown architectures. This adverse condition also restricts the solutions of fine-scale plant architecture modeling (Davidson et al., 2008) from playing their full roles. Although the latest ALS systems are continuously improved with their sampling densities increased, they are far from enough for characterizing crown-internal structural details (Ko et al., 2013).

Fortunately, botanists have generalized tree forms of a gigantic diversity into a limited number of tree architecture models, which facilitate sketchily giving a-priori knowledge of tree structure and development. Hallé et al. (1978) proposed a scheme to deduce the branch and stem architectures of tropical trees into 23 tree architecture models (hereafter called Hallé architecture models, HAMs). These HAMs have also been applied onto multiple temperate tree species (Fisher, 1984; Costes et al., 2006). As a representative set of tree architecture models, their proposal was rooted in that there are inherent modes of organizing stems and branches for different tree species, and the associated geneticallydetermined rules don't vary along with trees growing (Negrón et al., 2013). Tree architecture models have also been used to determine the branching forms considered in plant constructions (Barthélémy and Caraglio, 2007), e.g., based on the hidden semi-Markov models (Guédon et al., 2001) or the stochastic models (Costes and Guédon, 2002; Renton et al., 2006). HAMs have also shown their usefulness in improving the understanding of tree, canopy, and even forest properties. For example, Mutke et al. (2005) applied a Rauh HAM for characterizing the grafted stone pines (Pinus pinea L.) to figure out their shoot growths and bud differentiations. Feng et al. (2012) synthesized a Rauh HAM and a forest dynamics model to simulate the development of a forest stand, with the stand-level ecological/silvicultural model and the tree-level architectural representations integrated to provide the details of the individual trees at the stand level.

With the HAMs used as a linking bridge, ALS can supply more information on the structural, morphological, and biophysiological properties of trees. Theoretically, although ALS cannot make the same performance on tree structure reconstruction as TLS does, it can reveal the sketchy features such as the spatial pattern of laser points, the penetrations for different crown layers and crown integral symmetry. These three case features correspond to the external shapes of trees and crown-internal structures (i.e., leaf distributions and branching modes), which are of the potentials for deriving the appropriate HAMs. Actually, in the previous ALS-based endeavors, it has already been realized that grasping more detailed structural features of crowns can better the depth of tree understanding (Hyde et al., 2005; Wang and Glenn, 2008; Ferraz et al., 2012). With the promotion of ALS performance, particularly on the sampling density, the trend of making use of crown-internal structure is getting more and more distinct. For instance, tree genera classification has been explored with the geometric traits of branches measured with high-density ALS (Ko et al., 2013). The HAMs proposed by Hallé et al. (1978) were defined in a descriptive (qualitative) way; hence, no explicit attempts, so far, have emerged to connect these theoretical models to real ALS point clouds to derive more about tree growth habits in structure. Consequently, a question can be asked—Investigating tree growth habits in structure: using traditional HAMs, state-ofthe-art ALS, or both?

To fill this gap, this study attempted to recruit the conventional tree architecture models into the state-of-theart ALS based investigation of tree-level properties, here tree growth habits in structure. As suggested by Hallé et al. (1978), HAMs depend on tree species. Accordingly, the used strategies were to classify tree species based on the ALS-collected data, identify the HAM for each of the targeted tree species, and derive the properties of the targeted trees by referring to their related HAMs. To validate this solution plan, this work needed to implement the following three tasks: (1) converting the descriptive HAM-indicated structural traits into a set of quantitative parameters by following the characteristics in the ALS-based tree representation; (2) classifying tree species based on the proposed crown- and tree-associated feature parameters; and (3) identifying the HAM for each of the tree species to supply the information about tree growth habits in structure.

# MATERIALS AND METHODS

#### Study Area and Data Collection

The study area was located on the Seurasaari Island, Helsinki, Southern Finland (N60◦ 10′ 52′′, E24◦ 53′ 4 ′′). The island is a wooded park and has rocks, hills, wetlands, and herb-rich forests, covering approximately a total of 46 hectares. The tree species include Norway spruce (Picea abies), Scots pine (Pinus sylvestris), European rowan (Sorbus aucuparia), European aspen (Populus tremula), alder (Alnus sp.), birch (Betula sp.), English oak (Quercus robur), Norway maple (Acer platanoides), and smallleaved lime (Tilia cordata), in a descending order of abundance. The studied trees lie in the middle of the forest with varying understory vegetation.

The airborne LiDAR data were collected by using an Optech ALTM 3100 laser scanner (Optech Inc., Ontario, Canada) in September 2010. The flight altitude was set at ∼400 m. One laser pulse triggered 1, 2, 3, or 4 returns. The density of the collected point clouds is ∼10 points per m<sup>2</sup> , which can be classified into an intermediate point density compared with the very-high density reported by Ko et al. (2013). To acquire the reference data, the TLS data were collected with a Leica HDS6100 system (Leica Geosystem AG, Heerbrugg, Switzerland) in September 2010. HDS6100 is a 690 nm phase-based continuous-wave laser scanner with a 360◦× 310◦ Field-of-view; its data acquisition rate is 508, 000 points per second. The distance measurement accuracy is ±2 mm at a range distance of 25 m. The point spacing is 6.3 mm at 10 m. The TLS measurements were finished in a multi-scan mode (Liang and Hyyppä, 2013), which can result in good point coverage and further supply tree structure representations as the reference data. More detailed specifications of the test settings were described by Holopainen et al. (2013). The field inspections were conducted in January 2012, and the species of the target trees were visually identified.

# Data Preparation

Based on the pre-set reference markings, the ALS and TLS data were registered by using the Cyclone software (Leica Geosystem AG, Heerbrugg, Switzerland). The procedures of tree isolation and stem segmentation were interactively accomplished by using the Terrascan software (Terrasolid Oy, Helsinki, Finland), and the detailed operations of these isolation and segmentation were delineated by Holopainen et al. (2013). The stem axes of the isolated trees were determined by principal component analyses (PCA) of the extracted laser points on stems (Lehtomäki et al., 2010). The results can help calculate tree feature parameters in a uniform manner.

After data preprocessing, a total of 40 trees with their structural representations simultaneously by the ALS and TLS data were chosen for the tests. The trees belong to four typical boreal tree species, including nine Norway spruces (P. abies, PA), 14 Scots pines (P. sylvestris, PS), 7 European aspens (P. tremula, PT), and 10 English oaks (Q. robur, QR). The specimens represented by the ALS and TLS data are illustrated in **Figure 1**. Their descriptive statistics in terms of tree height and crown length (i.e., crown thickness Hauglin et al., 2014) are listed in **Table 1**.

#### Schematic Program

The schematic program of combining the HAMs and ALS was designed (**Figure 2**). The dashed arrow line marks the status that it is hard to directly derive tree growth habits in structure from ALS data. After all, different to TLS that is even available for tree architecture modeling at the branch scales (Côté et al., 2011), most contemporary ALS systems show low point densities. The HAMs are used to work as a bridging link to fill this technical gap. Specifically, the HAM-delineated tree structural organization rules are used to guide tree species classification; then, the HAM that can best model each tree species is identified; finally, the structural growth habits for each of the tree species of interest are derived from its associated HAM. In this schematic framework, tree species classification serves as a kernel procedure; hence, seeking how to add the accuracy of this procedure is very important.

For LiDAR-based tree species classification, a large number of approaches has been proposed, even aiming at the scenario

sylvestris (PS) (C,D), Populus tremula (PT) (E,F), and Quercus robur (QR) (G,H).

that merely limited-size sample data are collected (Puttonen et al., 2010). Their specific methodologies have proved to work well on the corresponding ALS data (Dalponte et al., 2008; Heinzel and Koch, 2012; Ko et al., 2013) and TLS data (Lin and Herold, 2016) that are of varying conditions. However, almost all of these previous studies suggested that both distinctive feature variables and high-performance classifiers are still needed. This schematic program was aimed at these two aspects, i.e., comprehensively proposing feature parameters and classifiers. For example, the parameters capable of reflecting tree external (TE) structures such as tree height can be intuitively extracted, but the feature parameters reflecting crown internal (CI) structures such as tree symmetry need to be proposed by referring to the structural characteristics presented by the associated HAMs. The classifications based on these two kinds of parameters correspond to the routines [1] and [2] in **Figure 2**, respectively. Tree species classification can be deployed on the two sets of feature parameters separately or together ([1]+[2]) to enhance the overall accuracy. The following sub-Section Tree Species Classification specifically describes how to derive feature parameters from the HAMs and the selected classifier appropriate for the associated classification scenario in the present study. Next, the sub-section HAM Identification and Information Derivation delineates how to determine the related HAMs, via referring to the expert knowledge supplied by the potential HAMs, and derive tree growth habits in structure.


TABLE 1 | Descriptive statistics for the sample trees.

FIGURE 2 | Workflow of the proposed schematic program in this study.

#### Tree Species Classification TLS-Assisted Analysis of HAM Signatures

For the four tree species, their HAMs were visually interpreted in advance, via referring to Prusinkiewicz and Remphrey (2000), in order to obtain the ground-truth data that were used to test the performance of the proposed schematic program. The manuallyidentified HAMs are demonstrated in **Figure 3**. The Massart model shows plagiotropic structure, with the main branches being in whorls and rhythmic growths. The Rauh model shows different orthotropic structure, with its main branches being morphogenetically equivalent to its trunk, all in a rhythmic growth mode. The Roux model demonstrates plagiotropic, monopodial, and non-phyllomorphic structure and continuous growths. The Attim model exhibits orthotropic structure and more or less continuous growths, also with its main branches being morphogenetically equivalent to its trunk. More specific information about tree growth habits in structure can refer to the relevant literature (Hallé et al., 1978; Fisher, 1984; Costes et al., 2006). Given that the focus of this study was on examining the proposed schematic program rather than on revealing more growth habits of the sample trees, the above-listed structures such as plagiotropic structure and continuous growth were referred to for proposing more appropriate feature parameters for tree species classification and deriving tree growth habits in structure.

#### Proposal and Derivation of Tree Structure Feature Parameters

The performance on exploiting tree structure growth habits is largely dependent on whether the assumed feature parameters are suitable for tree species classification. However, none of the previous studies have generalized each of the 23 HAMs into a set of distinguished feature parameters. To fill this gap, the first work was to propose feature parameters effective for operating the step of tree species classification, with the HAM-indicated descriptive signatures and the characteristics of ALS-based tree representation considered in a comprehensive way. That is, the derived tree structure feature parameters from the ALS-based tree representations need to characterize the marked differences of orthotropic vs. plagiotropic branching and rhythmic vs. continuous growth reflected by the four models as much as possible.

First, the CI structure feature parameters were proposed. For different branching patterns within crowns, the related ALScollected point clouds demonstrate different spatial-distribution modes. For example, the orthotropic branches often correspond to a symmetric distribution of point clusters that represent branches, but the plagiotropic branches more often mean high laser penetration in trees. Such knowledge comprises the rules linking between the structural features and the HAMs. In accordance to this rule set, the HAM-associated CI structure parameters were extracted by applying the 8-segments framework developed by Lin and Hyyppä (2016). Next, a total of 13 CI parameters were extracted, with their derivations from the specific characteristics of the HAMs and their quantitative definitions listed in **Table 2**. For instance, the fourth parameter (P4) is defined as the largest voxel density within 1 m super-voxels to reflect the HAM situation of leaf clustering. These parameters can allude to different branching and leaf distribution modes, which are characteristic for different HAMs.

Then, the TE structure feature parameters were also proposed. The procedures of deriving the HAM-related TE structural features followed the workflow proposed by Lin and Herold (2016). In addition to the common structural parameters (Lin and Herold, 2016), additional structural parameters were defined (**Table 3**), by seeking the corresponding more robust parameters or by combining two common parameters. For the first case, the diameter of the circle with area equal to the crown cover (DEA) was defined, which can reduce the effects of crown shape variability. For the second case, the ratios between a couple of feature parameters and tree height were also examined, as shown by the parameter of LcHt (**Table 3**). This operation can handle the influences of varying tree ages on tree species classification. The specific 10 TE parameters can also characterize HAMs, and their combinations are of the potential for reflecting the structural differences between the four tree species.


FIGURE 3 | Illustrations of the HAMs proposed by Hallé et al. (1978): (A) Massart, (B) Rauh, (C) Roux, and (D) Attim.


Where L<sup>c</sup> is crown length, S<sup>i</sup> is the ith segment, NG<sup>i</sup> is the point number within the ith voxel, HG<sup>i</sup> is the height of the ith voxel, all of the voxels at the same height compose the related crown layer, RG<sup>i</sup> is the radius of the ith voxel, and n indicates the total number of the voxels for each tree.

Then, the performance about the derivation of the CI and TE feature parameters was assessed. Tree height (TH) and diameter at breast height (DBH) were deemed as the representative variables. After correlation analysis of the estimated and reference TH sets, the performance of the derivations was quantified in terms of the coefficient of determination (R<sup>2</sup> ) and

**13**


TABLE 3 | TE feature parameters derived from the ALS point clouds by referring to the HAMs.

Where L<sup>c</sup> is crown length, ABase indicates the crown base area, HHLS is the height of the highest branch within crown lower surface (HLS), HLLS is the height of the lowest branch within crown lower surface (LLS), L<sup>s</sup> is the longest spread of crown cover (LS), Lcs is the longest cross-spread of crown cover (LCS), HG<sup>i</sup> is the height of the ith voxel, A<sup>G</sup> is the side area of each voxel, nGXZ , nGYZ and nGXY each means the number of voxels in the XZ, YZ, and XY projection plane, AXZ CV , AYZ CV and AXY CV each means the area of the convex hull of the voxels in the XZ, YZ, and XY projection plane, respectively, and n indicates the total number of the voxels for each tree. For each tree, the variable of DTM (digital terrain model) relates to the height of the location relating to its root.

root mean squared error (RMSE), which are defined as

$$R^2 = 1 - \left(\sum\left(d\_i - \hat{d}\_i\right)^2\right) / \underbrace{\left(\sum\left(d\_i - \bar{d}\right)^2\right)}\_{},\tag{1}$$

$$RMSE = \sqrt{\sum \left( d\_i - d\_i^{\mathbb{R}} \right)^2} / n \,, \tag{2}$$

where d<sup>i</sup> is the estimated TH value for the ith tree; d R i is its reference TH value; their difference d<sup>i</sup> − d R i is termed as TH deviation; ˆd<sup>i</sup> is the TH value estimated by following the estimation-reference TH regression relationship revealed by correlation analysis; and ¯d = Pdi/n .

#### Classification Implementation

Before the classification, a non-parametric test was manipulated to evaluate the distinctions between the four tree species, in terms of all of the calculated CI and TE feature parameters. The significance of the differences between the four tree species was handled through conducting the multivariate analysis of variance (MANOVA) (http://www.statsoft.com/Textbook/ ANOVA-MANOVA#multivariate). The specific operation was run in the Data Processing System software environment (Tang and Zhang, 2013), with the resulting p-values used to characterize the differences.

The classification of the four tree species was based on LIBSVM, i.e., a support vector machine (SVM) package (Melgani and Bruzzone, 2004; Pfeifer and Briese, 2007; Dalponte et al., 2008; Chang and Lin, 2014). The sample trees were classified by using the SVM classifier in a leave-one-out cross-validation (LOOCV) way (Lin and Herold, 2016; Lin and Hyyppä, 2016). Specifically, each tree was classified with the SVM classifier that was trained based on the remaining trees. The assessment of their accuracies was operated in terms of recall and precision, i.e., for any tree species, the ratio between the number of the correctlyclassified trees and the number of its trees and the ratio between the number of the correctly-classified trees and the number of the classified-into-that-species trees, respectively.

To seek the optimal classification result, the LOOCV-SVMbased classification was deployed on all of the cases of combining the CI and TE feature parameters. The numbers of the cases that combine 1 to 13 CI feature parameters are 13, 78, 286, 715, 1287, 1716, 1716, 1287, 715, 286, 78, 13, and 1, respectively, and the numbers of the cases that combine 1 to 11 TE feature parameters are 11, 55, 165, 330, 462, 462, 330, 165, 55, 11, and 1, respectively. The resulting recalls and precisions were compared and the optimal accuracies were achieved. The CI and TE parameters corresponding to their optimal results (termed as CIOpt and TEOpt variables) were combined together as the input variables (equivalent to the case of [1]+[2] in the schematic program) to explore the best classification result. In addition to the recall and precision, Cohen's kappa coefficient (κ) (Cohen, 1960) was calculated to assess the performance of the classifications individually based on the CI, TE or the combination of the CIOpt and TEOpt feature parameters. κ measures the interrater agreements between categorical items. As a more robust measure than the simple percent agreement calculations like recall and precision, κ also considers the agreement occurring by chance.

# HAM Identification and Information Derivation

#### HAM Determination and Qualitative Information Retrieval

After tree species classification, two best-performing parameters were extracted. Their related feature parameters for HAM identification included CI alone, TE alone, or the combination of CI and TE. With the assistance of expert knowledge about the HAMs in terms of these feature parameters, the distribution of CI and TE value-pairs (i.e., the combination of CI and TE) was assumed to accurately determine the HAM corresponding to each of the four tree species. Specifically, the scatterplots of those calculated value-pairs were divided into four local quadrants, in accordance to their relative ranges for the four HAMs, e.g., the Massart model relating to low P1 values and low P13 values, the Rauh model to high P1 and low P13, the Roux model to low P1 and high P13, and the Attim model to high P1 and high P13. Next, the HAM for each group of the classified trees was identified according to its related center of the value-pairs lying in which one of the local quadrants. By this means, the influences caused by some special specimens can be reduced. Based on the determined HAMs, the knowledge about tree growth habits in structure can be derived, and the specific operations are listed as follows.

#### Quantitative Information Derivation and Assessment

The feature parameters proposed for HAMs determination can also be assumed as the quantitative indicators of tree growth habits in structure. After the steps of tree species classification and HAMs identification, the derived feature parameters based on the classified specimens, in an isolated or integrated manner, were used to supply the information on the quantitative characteristics of tree growth habits.

The mis-classified or -identified specimens, no doubt, may introduce errors into the derived feature parameters; hence, the accuracies of the derived parameters need to be quantitatively assessed. Given that the objective of this study was to investigate tree growth habits in structure at forest plot or stand rather than individual tree levels, statistics based on Gaussian distribution were used to generate a limited number of indices, in favor of efficient accuracy assessment. For each of the prescribed feature parameter, its derived values for all of the specimens of each classified species were fitted to a Gaussian distribution function. The resulting expectation (µ) and standard deviation (σ) were compared to the derivations from the ground-reference data, and the numerical divergence was used to quantitatively evaluate the performance of the proposed method for retrievals of tree growth habits in structure.

#### RESULTS

#### ALS-Based Tree Characterization

Given that TH and DBH were selected as the representative variables to assess the derivations of the CI and TE feature parameters, their derivations from the ALS and TLS data were examined. The performance of TLS-based DBH derivations and ALS-based TH estimations are shown in **Figure 4**, with high-value R<sup>2</sup> and low-value RMSEs but with linearly-varying estimation biases at different value ranges that are principally caused by the systematic errors of the scanning systems. The comparison between the TLS-derived and manually-measured DBHs can testify the capability of the used TLS for detailed tree representation, and the comparison between the ALS- and TLS-derived THs can verify the availability of the ALS for tree characterization. After all, TLS has been used for collecting the ground-truth data (Côté et al., 2012), and ALS data can derive other feature parameters like DBH via allometric relations (Hauglin et al., 2014; Calders et al., 2015). More assessments of the derivations of other structural feature parameters can refer to Holopainen et al. (2013). All of the results validated the premier of this study, i.e., the used ALS and TLS are applicable for characterizing the basic structures of the sample trees.

#### Tree Species Classification

We first derived the proposed CI feature parameters (Figure S1), based on which the MANOVA p-value was calculated to be 0.0001 (Table S1). This meant significant differences exist between the four tree species, but the CI-based classification results (Figure S1) told that using any one of the 13 CI parameters individually cannot fulfill distinguishing the four tree species. Then, classifications were conducted for all of the cases of combining 1–13 CI parameters (Figure S2), which relates to the operations of the routine [1] marked in **Figure 2**. The overall performance firstly went better and then leveled off. The highest classification accuracy is 82.5% (κ = 0.794), in the case of inputting the five CI parameters—P3, P4, P6, P7, and P8—as the dependent variables into the classification (Table S2).

Then, the TE parameters were derived (Figure S3), and their related MANOVA p-value was 0.0001 (Table S3). The classifications for all of the cases of combining 1–10 TE structure parameters were carried out (Figure S4), which corresponds to the routine [2] in **Figure 2**. The highest classification accuracy is 85.0% (Table S4), in the case of combining the LcHt, LcDEA, Gc, DEA, and σ TE parameters (κ = 0.764).

The combinations of the above-sought CIOpt and TEOpt parameters were input to implement the classification (MANOVA p-value = 0.0001; Table S5). The accuracies for all of the cases of combining the optimal five CI parameters and the optimal five TE parameters were calculated (**Figure 5**), and this step relates to the routine [1]+[2] in **Figure 2**. The highest accuracy is 85.0%, which corresponds to the optimal case of TE-based classification (κ = 0.794; Table S6). Further, for the task of searching two feature parameters to determine the appropriate HAMs for the four tree species, the best case of combining the CI-typed parameters of P4 and P7 was found, exhibiting the highest classification accuracy of 75.0% (κ = 0.658).

# HAM Determination and Information Retrieval

#### Qualitative Derivation

After tree species classifications, the HAM for each of the four tree species was identified based on the expert knowledge, assisted by referring to the distributions of the two P4 and regression analysis.

P7 variables in pairs (**Figure 6**). Specifically, for the four tree species, the centers of their P4–P7 pair distributions were derived, and a comparison of these centers to the opposites calculated by referring to the manually-trained crown-HAM links was operated to determine the HAM for each of the four tree species. The links include that the Massart model relates to low P4 and low P7 values (corresponding to the PA species), the Rauh model exhibits high P4 and low P7 (PS), the Roux model shows moderate P4 and low P7 (PT), and the Attim model presents low P4 and high P7 (QR). By this means, the a-priori knowledge about tree growth habits in structure was acquired for the four tree species (**Table 4**).

#### Quantitative Derivation

The information about tree growth habits in structure, further, was quantitatively derived, as illustrated by the boxplots of the derived parameters P4 and P7 (**Figure 7**). The biases between the boxplots show the distinctions between the four tree species. For each tree species, the derived parameters are approximate to their values from the ground-truth data, as evidenced by their comparative statistics (**Table 5**). Specifically, their absolute deviations (|1µ|) of the expectations of the fitted Gaussian distribution before and after the proposed tree species classification and HAM identification are <7% of the actual values for any of the four tree species, and their absolute deviations (|1σ|) of the standard deviations (σ) are all <16% of the reference values. These results suggested that based on the proposed method, the quantitative characteristics about tree growth habits in structure for the four tree species can be derived for the trees at plot or stand scales.

# DISCUSSION AND CONCLUSION

The results of the case study based on the four tree species growing in the boreal environment validated the hypothesis that the recruitment of conventional descriptive tree architecture models such as the HAMs into the state-of-the-art ALSbased forest mapping facilitates more efficiently deriving the typically hard-to-discern tree growth habits in structure. The contributions of this study are 2-fold. With ALS data used as a bridge, the application range of the HAMs can be expanded to the boreal forests; meanwhile, with the HAMs used as a link, the application discipline of ALS can be extended to tree growth habits in structure. In other words, the answer to the question posed in the Introduction, no doubt, is "both." Overall, this study developed a novel pan-solution strategy, i.e., by recruiting the previously-established tree architectural, physiological, and/or biophysichemical models into the cutting-edge forest mapping technologies, for more efficiently understanding various forest growth habits, which facilitate investigating forest ecologies and projecting forest dynamics and processes under climate changes.

with pluses indicate the centers of the parameter-pair distributions for the correctly-classified tree specimens; and two dash lines segment the four local quadrants.

TABLE 4 | Tree growth habits in structure derived for the four tree species.


#### Application Extending

First, the implications of this study require more elucidation, because the results in **Table 5** may be misleading, i.e., the results appear simple. Theoretically, the HAM related to each individual tree can be identified by the associated experts in the field (Negrón et al., 2013), and in such a sense, the development of a complicated method as shown in this study seems unnecessary. This reasoning, however, is only rooted in the results that were derived from the limited number of sample trees, without taking the broader-sense context of this study into account. This study tried to advance the field of forest mapping based on ALS, which generally can cover large areas (Ferraz et al., 2012). In this situation, manual determination of the HAM for each single tree is unadvisable. Besides, the concerned targets in practical ALS-based forest mapping tend to be tree communities, seldom single trees (Koetz et al., 2007). The HAMs can give descriptive "plagiotropic or not," instead of quantitative "plagiotropic degree," and they are useful for learning the whole structural traits of communities. Hence, the combination of ALS and the HAMs can help people to have a sketchy understanding of tree structural attributes for forest stands; namely, the first implication of this work is to present a solution of reflecting largescale properties of forest dynamics, further advancing studies on plant geography and spatial ecology.

Then, for each specimen of the four tree species measured by the ALS, its HAM identification means not only grasping its growth habits in structure but also learning the rules for its symbolic delineation and graphical representation (Prusinkiewicz and Remphrey, 2000). Specifically, the proposed feature parameters can be adopted as the geometric/topologic restrictions to briefly determine the sequencing of apices and the configuration of branches; then, tree geometric models can be derived from ALS data, a difficult task for long but of extensive interest (Calders et al., 2013). The developed tree geometric

TABLE 5 | Derivations of the (a) P4 and (b) P7 CI parameter (DP) values after the classification and HAM identification, compared with their values in the ground-truth (GT) data.


More information is listed in the Supplementary Data file.

models can be effectively in favor of the simulation-based analyses of forest-environment interaction effects (Lovelock et al., 2006). The botanic knowledge derived from the HAMs, further, can help to push forward the stage of tree geometric modeling to tree comprehensive biophysical modeling for better understanding forest physiologies.

### Influence Analysis

The performance of the proposed method is restricted to many influence factors, and analyses indicated that there are three kinds of primary ones. First, the calculated values of the proposed feature parameters were unavoidably attached with errors. The sources of these errors may involve biased geometric modeling, incomplete structure representations, and uncertainties of parameter derivations. When the errors are large enough, such feature parameters may play negative roles in tree species classification (Lin and Herold, 2016). This limitation is evidenced by the maximum total accuracy, which initially improved and then leveled off, as indicated by the top whiskers of the boxes in **Figure 5**. In other words, this effect occurs in the cases involving the CI and TE parameters separately (Supplementary Figures S2 and S4) and their optimal-accuracyrelated combination (**Figure 5**), respectively.

Second, the proposed feature parameters maybe cannot efficiently reflect some particular features of the tree species of interest. This influence factor is exemplified by the P. tremula trees, which showed relatively lower accuracies in all of the three steps of determining their HAMs in comparison to the other three tree species. In fact, as illustrated in **Figure 1**, the ALSrepresented morphologies of the P. tremula tree and the Q. robur tree are similar, and it is difficult to distinguish them even by visual interpretation. This just verifies the hypothesis that the feature parameters proposed in this study may better present the characteristics of Q. robur than P. tremula.

Third, one tree species may not be strictly related to one HAM. At its different growth stages, a tree may satisfy different HAMs. Some tree species show different developments in HAMs during their ontogeny (Millet et al., 1998). Some trees may behave as satisfying multiple HAMs. Although this diversity issue can be somehow handled in the context of ALS-based large-area forest mapping as mentioned above, errors may be introduced into the results of tree species classification. This suggests that this problem may limit extending the application range of the proposed method; in addition, in the following studies more attention shall be paid onto the CI feature parameters, which were proposed by following the characteristics of HAMs.

# Potential Improvements

Note that although the derivations were on the basis that the used ALS system can represent trees in a relatively complete manner (as illustrated in **Figure 1**), the proposed CI and TE feature parameters are far from enough for representing the real structures of the targeted trees, let alone the scenarios of other tree species and other HAMs regarded. Compared to the parameters considered in this study, multiple other structural feature parameters can be extracted by exploring other morphological or topological features of the trees. For instance, the spatial distribution mode of the primary branch centers (Lin and Herold, 2016) can be extracted to supply more information. Simultaneously, the proposal of other structural parameters needs also to regard the characteristics of ALSbased tree representations. When a crown is dense, less information is available from the related trunk (Puttonen et al., 2010); therefore, the parameters derived from such trunks cannot be directly compared in some cases. With such detailed influence factors as many as possible considered, the results will be improved. In addition, strengthening the automatic degree in deriving these feature parameters is highlighted.

The proposed approach for identification of the HAMs for different tree species did help to exploit the unreadilyobservable tree growth habits in structure hidden in the ALScollected point clouds, but this approach is still principally under qualitative interpretation. In other words, the derived knowledge of tree growth habits cannot be quantitatively used for precision managements of forests. Subsequent studies need to emphasize how to propose new feature parameters and to attempt other kinds of tree architecture models (e.g., Weber and Penn, 1995) for quantitative characterization of tree growth habits.

The used ALS system is a currently-pervasive airborne LiDAR, and its sampling density can reflect the main-stream laser scanning performance. Hence, the proposed method shall be highly applicable for practical applications. Nevertheless, many ALS systems produced at earlier times are still in use, and consequently, the sensitivity of the proposed method in response to laser point density needs to be further examined in the subsequent work, e.g., via checking its performance based on the ALS data of different sampling densities. The feasibility of the proposed method for different tree ages also needs to be examined, e.g., via applying it to the stands at varying growth stages and comparing its performance relative to different morphologies of trees.

#### REFERENCES


## AUTHOR CONTRIBUTIONS

YL: designed this study and developed the method; MJ: made the data analysis; PP: analyzed the results and revised the manuscript; JH: reviewed and revised the manuscript.

#### FUNDING

This work was financially supported in part by the National Natural Science Foundation of China (Grant No. 31670718, 41471281, and 41401515) and in part by the SRF for ROCS, SEM, China.

#### ACKNOWLEDGMENTS

Thank the Mobile Mapping group at the Finnish Geospatial Research Institute for distributing the data. This work was financially supported in part by the National Natural Science Foundation of China (Grant No. 31670718, 41471281, and 41401515) and in part by the SRF for ROCS, SEM, China. The cooperation was made possible by China SAFEA (State Administration of Foreign Experts Affairs) supported program (Grant No: 20161100067).

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpls.2018. 00220/full#supplementary-material


cultivars with different tree architectures. J. Am. Soc. Hortic. Sci. 138, 407–415.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Lin, Jiang, Pellikka and Heiskanen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Not All Trees Sleep the Same—High Temporal Resolution Terrestrial Laser Scanning Shows Differences in Nocturnal Plant Movement

András Zlinszky 1, 2 \*, Bence Molnár <sup>3</sup> and Anders S. Barfod<sup>2</sup>

<sup>1</sup> Balaton Limnological Institute, Centre for Ecological Research, Hungarian Academy of Sciences, Tihany, Hungary, <sup>2</sup> Ecoinformatics and Biodiversity Section, Department of Bioscience, Aarhus University, Aarhus, Denmark, <sup>3</sup> Department of Photogrammetry and Geoinformatics, Budapest University of Technology and Economics, Budapest, Hungary

#### Edited by:

Yann Guédon, Centre de Coopération Internationale en Recherche Agronomique pour le Développement (CIRAD), France

#### Reviewed by:

Franck Hétroy-Wheeler, Université de Strasbourg, France Frédéric Boudon, Centre de Coopération Internationale en Recherche Agronomique pour le Développement (CIRAD), France

> \*Correspondence: András Zlinszky zlinszky.andras@okologia.mta.hu

#### Specialty section:

This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science

> Received: 09 June 2017 Accepted: 05 October 2017 Published: 20 October 2017

#### Citation:

Zlinszky A, Molnár B and Barfod AS (2017) Not All Trees Sleep the Same—High Temporal Resolution Terrestrial Laser Scanning Shows Differences in Nocturnal Plant Movement. Front. Plant Sci. 8:1814. doi: 10.3389/fpls.2017.01814 Circadian leaf movements are widely known in plants, but nocturnal movement of tree branches were only recently discovered by using terrestrial laser scanning (TLS), a high resolution three-dimensional surveying technique. TLS uses a pulsed laser emitted in a regular scan pattern for rapid measurement of distances to the targets, thus producing three dimensional point cloud models of sub-centimeter resolution and accuracy in a few minutes. Here, we aim to gain an overview of the variability of circadian movement of small trees across different taxonomic groups, growth forms and leaf anatomies. We surveyed a series of 18 full scans over a 12-h night period to measure nocturnal changes in shape simultaneously for an experimental setup of 22 plants representing different species. Resulting point clouds were evaluated by comparing changes in height percentiles of laser scanning points belonging to the canopy. Changes in crown shape were observed for all studied trees, but clearly distinguishable sleep movements are apparently rare. Ambient light conditions were continuously dark between sunset (7:30 p.m.) and sunrise (6:00 a.m.), but most changes in movement direction occurred during this period, thus most of the recorded changes in crown shape were probably not controlled by ambient light. The highest movement amplitudes, for periodic circadian movement around 2 cm were observed for Aesculus and Acer, compared to non-periodic continuous change in shape of 5 cm for Gleditschia and 2 cm for Fargesia. In several species we detected 2–4 h cycles of minor crown movement of 0.5–1 cm, which is close to the limit of our measurement accuracy. We present a conceptual framework for interpreting observed changes as a combination of circadian rhythm with a period close to 12 h, short-term oscillation repeated every 2–4 h, aperiodic continuous movement in one direction and measurement noise which we assume to be random. Observed movement patterns are interpreted within this framework, and connections with morphology and taxonomy are proposed. We confirm the existence of overnight "sleep" movement for some trees, but conclude that circadian movement is a variable phenomenon in plants, probably controlled by a complex combination of anatomical, physiological, and morphological factors.

Keywords: circadian plant movement, laser scanning, tree canopy movement, 3-dimensional modeling, tree physiology, turgor, chronobiology, circadian rhythm

# INTRODUCTION

Circadian movement of leaves includes the process of nyctinasty, already known to the ancient Greeks (Bretzl, 1903) but first described by Darwin (Darwin and Darwin, 1880). The leaves of many shrubs and trees actively change position during the night. In some fabaceous trees this phenomenon is particularly striking. In a recent series of experiments by Puttonen et al. (2016) terrestrial laser scanning (TLS) was applied to detect welldefined circadian movement rhythm of the branches and the trunk of the temperate tree Betula pendula. TLS (Dassot et al., 2011) allows for detailed measurement of the distance from sensor to a target object by measurement of the travel time of short laser pulses emitted from a given position in a known direction. Since the measurements are conducted during very short time (in the order of milliseconds), a high repetition rate is possible, which together with an equidistant scan pattern allows regular sampling of the target objects. The output of such a measurement is a set of points with positioning coordinates and the amplitude of the reflected signal as an attribute. By employing TLS, Puttonen et al. (2016) created detailed 3-dimensional models of trees at hourly intervals during the night. Their results show that tree branches moved up to 8 cm downwards during the night and returned to their original position by daybreak. By scanning two Betula trees nearly 2,000 km from each other, they proved that this phenomenon is probably an attribute which is unique to individual plant species. This seminal discovery opened up many new questions. What is the adaptive advantage of sleep movement (as Darwin called it and as we will also call nocturnal changes in plant shape)? What are the processes governing sleep movements and how are these linked to ambient light, temperature, humidity, and soil water content? Are the movements driven by an internal circadian clock or induced externally by photoperiodism? Are patterns of sleep movement characteristic for certain taxonomic groups?

Our aim is to gain an overview of the variability of plant circadian movement in seed plants. Our method measures movement as the concerted displacement of branches and leaves. Under conditions monitored for light, humidity, temperature, and (lack of) wind, we surveyed a wide taxonomic range of trees and shrubs grown in containers. The experiment was set up to compare sleep movement across taxonomic groups and different types of phenology and morphology. Advancing the methodology of nocturnal tree movement quantification was outside the scope of this mainly qualitative study.

#### MATERIALS AND METHODS

#### Measurement Setup

#### Timing

The study was carried out on 16–17 September 2016, close to the autumn solstice when day and night are equally long. The first laser scanning was conducted at 6:30 p.m. and repeated every 30 min until 8:00 p.m. Hereafter, the scans were repeated every hour until 4:00 a.m. the next day. Finally a series of half-hourly scans were conducted until 7:00 a.m.

#### Location and General Environment

The experiment was carried out at a gardening shop at Balatonfüred, Western Hungary (N46◦ 56′ 16.91′′E17◦ 51′ 42.48′′), which has developed an open-air nursery for small-sized container grown woody plants. The shrubs and young trees were kept for at least half a year in the nursery, where they were watered on a daily basis. All the plants included in this study have thus been exposed to exactly the same weather and water conditions for at least 6 months. To avoid the expected effect of wind action we conducted the experiment in a greenhouse under construction, which had a glass roof, but no walls. During the full duration of the experiment weather conditions were stable with no wind and no precipitation or condensation. The nearest streetlights were more than 1 km away. During the overnight experiment we made great effort to exclude all outdoor lights and restrict the use of torch light to a minimum. A full moon allowed us to operate the scanner and read the instruments without an additional light source.

#### Experiment Design and Studied Plants

The choice of plants was somewhat restricted to what was available to us at site. Nevertheless, we were able to include in the experiment a broad range of growth forms (trees, shrubs, bamboos, and palms), leaf phenologies (evergreen and deciduous), leaf morphologies (entire, dissected, needle-shaped, and scaly), taxonomic groups, and ecoclimatic ranges (tropical, subtropical, temperate). Although high priority was given to include tall and dense plants, the height was restricted by the size of the containers. For most species we were unable to include more than one individual.

#### Measurement Techniques

#### Ambient Sensors and Instruments

Ambient air temperature and humidity were recorded by placing six climate loggers on the trunks of six selected trees and near the LIDAR sensor (Easylog USBTM, Lascar Electronics <sup>R</sup> , UK), and temperature loggers (HOBO PendantTM) in the soil containers of the same trees. These were set to collect data every 10 s. Wind speed was measured before each laser scanning measurement cycle using a simple hand-held ball anemometer (Dwyer Instruments, Michigan City, IND, US). At the same time ambient light was recorded using a LabSphereTM integrating sphere mounted on a Li-CorTM data logger, within the greenhouse frame as well as outside.

#### Terrestrial Laser Scanning Instrument and Measurement Parameters

Laser scanning can produce a high-density point cloud that allows for dynamic 3D modeling of complex shapes in few minutes. For 3D data acquisition, a phase-based Terrestrial Laser Scanner (TLS) was applied. FAROTM Focus S120 (FARO Technologies, Lake Mary, FL, US) operates a Class 3 laser with 905 nm wavelength. Acquisition parameters were set to capture 7 × 10<sup>9</sup> points in 15 min with highest quality. Based on instruments data sheet (FARO, 2013), the distance measurement has 0.95 mm accuracy on a solid, well-defined target while angular deviation has ± 2 mm effect on accuracy at 25 m


object distance (Note: in our case the furthest tree had an object distance of 15 m). Each raw scan requires 3 GB storage space, however, only ∼10% of the points are representing trees. Therefore, the first step of post-processing was to reduce the amount of data manually based on object distance and direction, removing all points that were outside the greenhouse frame. Reducing data amount allowed efficient data handling and comparison. The second step was to co-register the time series of the measurements. All measurements were captured from the same scanner position; however, the scanner battery and storage device (SD card) had to be replaced during the measurement session. Because this could potentially cause minimal changes in the scanner position relative to the object, we choose target based registration with reference spheres, but also including

the steel girder frame of the greenhouse as an additional source of reference surfaces. Altogether 18 scans were processed; maximum mean tension of registration between the reference (first) scan and any other scans is 2.3 mm. The first scan was selected as a fixed position, but global registration was applied for network adjustment. Assuming that all the scans were captured from the same position, no error propagation was expected from the network.

#### Data Analysis and Visualization

After removal of terrain and roof points we viewed the point clouds belonging to individual plants in 3D using FugroViewer <sup>R</sup> and in 2-dimensional height models which were obtained by rasterizing the maximum height in each cell in 10 cm intervals. Zlinszky et al. Not All Plants Sleep the Same

The 2D outlines were digitized in GIS with a margin of at least 10 cm, and further refined with the point cloud viewer to exclude all points belonging to the greenhouse structures and the plant container. Wherever the canopy started more than ∼1 m above the soil level of the container, we also excluded the lower part of the trunk from subsequent analysis via a minimum height limit in order not to misinterpret canopy height changes. For each plant, this frame defined by the horizontal and the vertical limits of the canopy was kept constant for individual exports of the point clouds acquired during the scannings. No further editing of the raw data was undertaken. We used the procedure described by Puttonen et al. (2016) to subdivide individual point clouds into height percentiles that were exported to Excel. The height of each percentile during the first survey (6:30 p.m.) was used as baseline, and the changes of percentile heights with respect to this baseline were plotted during the course of the experiment. All changes in percentile height exceeding 0.5 cm were interpreted as movement. Percentiles are height thresholds that separate the point cloud horizontally into sets of equal counts of points amounting always to 10% of the total point count for each survey. Where the branches and/or leaves of a plant are displaced, this is followed by the height distribution of the point cloud and thus also the height percentiles. However, depending on the crown shape, the effect of canopy displacement on the changes in height percentiles may vary considerably. For trees with a uniform (near-cylindrical) crown shape extending from the ground right upwards, the interpretation of percentile height changes is straightforward. For trees with other shapes, the narrower parts such as the trunk or crown apex have the same number of points between percentiles, but spread across a wider height interval. Height changes in any part of the canopy affect all percentiles, and such narrow parts amplify these effects, creating artifacts of high displacement of these percentiles. To keep track of such effects, we indicated in gray the curves pertaining to the percentiles of the narrow tree parts. To render figures comparable we excluded data from percentiles pertaining to the most narrow parts of the tree, where the extent across was less than 50% of the maximum crown diameter. The movement for the percentiles that had more than 50% of the maximum crown diameter was also averaged to create a mean displacement value for each studied point in time. The extent and nature of canopy movement was visually interpreted from these graphs, checking time distances between local and general minimum and maximum points and the vertical extent of displacement, without further numeric data analysis, in accordance with Puttonen et al. (2016).

### RESULTS

#### Ambient Conditions

Wind speed was measured to be 0 m/s during all measurements. Ambient light level inside the greenhouse was 63 mmol/s/m<sup>2</sup> at 6:30 p.m., 9.3 at 7:00 p.m., and dropped to 0.05 at 7:30 p.m., remaining at or below this value until 6:30 a.m., when 2.4 mmol/s/m<sup>2</sup> was measured, increasing to 33 mmol/s/m<sup>2</sup> by the last measurement at 7:00 a.m. Air temperature decreased gradually from 24◦C at 6:30 a.m. to 20◦C at midnight and further to 17◦C around 3:00 a.m., where it slowly increased to 18◦C by 7:00 a.m. All air temperature sensors recorded the same temperature within ± 1 ◦C. Relative humidity was around 65% for all sensors placed on tree trunks (and 55% for the sensor near the LIDAR), and increased continuously throughout the studied timeframe with small fluctuations of 1–2% that were closely similar for all trees, up to 95% at 7:00 a.m.

# Observations of Tree Position Changes

All the plants measured showed movement of at least part of the crown that exceeded our noise threshold defined here as 0.5 cm. Most trees showed periodic movement. Either they revealed true sleep movement by returning to the early evening position by next morning, or they showed shorter term periodic movement, or both. Additionally, some trees showed unidirectional movement and did not return to their starting position within the measurement timeframe.

The most striking result obtained in this study was the large variation revealed in nocturnal tree movement patterns across the seed plants. Betula utilis (Figure S7) and Olea europea (**Figure 4**) displayed the least movement, where the mean of all crown height percentiles never exceeded the preset threshold of 0.5 cm for noise. The legume tree Gleditschia triacanthos (**Figure 14**) represented the opposite extreme by moving part of its crown up to 9 cm during the scannings. Practically all of the remaining trees showed oscillations in movement and shape of the crown of 0.5–1 cm with cycles that were shorter than the observed timeframe, typically a few hours.

Nerium oleander (**Figure 1**) revealed a pattern reminiscent of the overnight movement recorded in uncultivated B. pendula trees by Puttonen et al. (2016). Our scanning captured a slight downwards movement between 6:30 p.m. and 7:00 p.m. followed by a phase of upwards movement peaking at 9:30 p.m. Hereafter, the crown moved downwards with minor oscillation until 4:30 a.m. The remaining part of the morning the crown moved upward to resume the position recorded at the start of the experiment. Although only minor movements below 1 cm were captured during the scanning, the recorded patterns were very consistent across the crown height percentiles. It should be noted that the overall crown shape of this shrub was relatively even with no pronounced narrow parts.

Acer palmatum (**Figure 2**) showed a pronounced circadian cycle as well, but in the exact opposite direction: crown height percentiles were observed to move upwards reaching the highest point (+1.5 cm) at 4:30 a.m. Hereafter, the percentiles gradually moved downwards until next morning when the canopy had almost resumed the starting position.

The scanning of Aesculus hippocastanea (**Figure 3**) revealed a similar pattern of upward movement in the evening peaking at 04:00 a.m. and gradual return to the start position at 07:00 a.m. However, this overall pattern was interfered by rapid changes in height, which caused the mean of the crown height percentiles to shift more than 1 cm downwards between 04:00 a.m. and 05:00 a.m.

Although sleep movements were detected in the individual crown height percentiles of O. europea (**Figure 4**) the displacement of the mean of all percentiles never exceeded

our preset detection threshold of 0.5 cm. The lower branches moved downwards before 07:00 p.m., upwards between 09:00 p.m. and 11:00 p.m., downwards between 01:00 a.m. and 05:30 a.m. and upwards again hereafter. The upper branches moved in quite the opposite pattern, reaching the highest point at 05:00 a.m., and resuming the starting position in the morning. The mean of the percentiles revealed an overall upward trend in the early evening, a more or less constant height during the night and returned to the start by morning; however, it should be noted that the amplitude was only 0.3 cm.

Viburnum pragense (**Figure 5**) moved similarly but with an even longer upward phase and perhaps some short-term periodicity superimposed. The lower branches moved downward whereas the upper branches seemed to move up almost continuously until around 4:30, when the movement was reversed and both the lower and the upper branches of the canopy returned close to their starting position. This results in the displacement amplitude of the mean of all used percentiles barely reaching 0.5 cm.

Photinia x fraseri (**Figure 6**) also showed movements similar to the upward sleep pattern detected in the species above: most crown height percentiles increased in height until 11:00 p.m. (compared to 10:00 p.m. recorded for most other plants), then slowly moved downwards until a minimum was reached between

4:00 and 5:00. However, prior to 8:00 p.m. and after 04:00 a.m., we recorded much more variation in point heights, which suggests that this shrub might have a shorter period of only about 8 h of sleep movement.

Quercus robur (**Figure 7**) seemed to carry out two cycles of movement. The first one started at 07:00 p.m., peaked at 11:00 p.m., and returned to the starting position around midnight. The second cycle started at midnight, peaked

at 5:00 a.m. and resumed the starting position around 7:00 a.m.

Magnolia grandiflora (**Figure 8**) apparently did not follow a diurnal cycle but moved in cycles of ∼3 h, peaking at 8:00 p.m., midnight, and 4:00 a.m. respectively. This pattern was quite distinct from the other species included in the experiment. The fact that the start and end positions of the branches were similar, and that the movement of the individual height percentiles was more or less synchronized strongly suggests that this recorded pattern was not just an

FIGURE 5 | Point cloud height percentiles and their displacement (m) in time relative to the starting position, for Viburnum pragense. Percentiles not used for calculating mean movement are shown in gray. Note that lower percentiles move downwards while higher ones move up.

artifact of noise. The maximum displacement of the mean of all percentiles is exactly −0.5 cm, which equals to our noise threshold. However, this is partly an effect of the lowest percentile, which moved in the opposite direction relative to the rest of the tree, thus resulting in lower average movements.

Among the Gymnosperms, Larix kaempferi (**Figure 9**) showed upward movement during the first hours of the

survey and then continued downward for the rest of the night, with two short peaks at 01:00 a.m. and 06:00 a.m. It returned relatively close to its start position by the end of the experiment in the morning. Nevertheless, the recorded movement pattern seemed to deviate from the sleep patterns of the trees above.

The smaller of the two Pinus nigra individuals included in the experiment, P. nigra 2 (**Figure 10**), moved upwards until 7:30 p.m., downward until 04:00 a.m., and then returned back to the original position, with some oscillation superimposed. More scannings are needed to clarify whether this pattern form part of a

sleep movement or rather represents random oscillations around a mean.

The larger P. nigra individual, P. nigra 1 (Figure S1), showed extreme fluctuations in the beginning of the scanning. We suspect this to be an artifact of mechanical disturbance at the beginning of the experiment and therefore not go into further detail. Compared to the other species included in the experiment the percentiles are highly variable in between scannings. This may be explained by the large size and long branches of the selected individual. The mean of the percentiles revealed that the tree kept a relatively constant height during most of the experiment, with two peaks at 8:00 p.m. and 05:00 a.m. respectively. It should be noted that this particular tree was almost twice as tall as the other trees included in the study and therefore had to be placed outside the greenhouse. For this reason great caution should be taken when making comparisons with the other species.

Cedrus libanica (Figure S2) fluctuated around a mean as well, with peaks around 07:00 p.m. and 3:00 a.m. It stood out, however, by not returning to its original position at the end of the measurement; in fact, the largest displacement from the starting position was observed at the last scan in the morning.

B. pendula (**Figure 11**) showed some abrupt downward movements compared to a seemingly constant baseline. Low points are occurred at 8:00 p.m., 04:00 a.m., and 07:00 a.m. This did not resemble classical downward sleep movement. Movement of most crown height percentiles was congruent. The tree did not return to its original position by 07:00 a.m.

Many plants belonging to the family Fabaceae with pinnately or double pinnately compound leaves display striking sleep movement of their leaves (nyctinasty). Among the trees included in our experiment Albizia julibrissin (**Figure 12**) showed the most conspicuous movement pattern, even though the leaves entered the sleep position before the scanning was initiated. A substantial part of this movement may be explained by the concerted action of the pulvini of consecutive orders. The scanning revealed upward movement of the crown until ca. 9:00 p.m., followed by downward movement until it reached the lowest point at 03:00 a.m. Hereafter, the crown gradually ascended until the morning. The leaves were observed to have reached their daylight configuration around 6:30 a.m., however, the tree reached its 06:30 p.m. start position already 1 h earlier (5:30 a.m.), and continued to move upward.

Another fabaceous tree included in the study, Styphnolobium japonicum, (**Figure 13**) stood out by having a pendulous crown, with an overall more or less cylindrical shape. The circadian movement of the leaves contributed only moderately to the overall crown movement in this species. The lower percentiles of the tree revealed most movement, but the broader lower part of the canopy top also moved more than 2 cm down. The crown movement had two low points, one at midnight and one between 4:00 and 5:00. It was not clear whether the night phase of the circadian cycle of this tree was longer than the duration of our experiment and the crown would return to its start position later, or whether the movement was not circadian.

Prunus virginica (Figure S3) showed a similar pattern, but the crown moved in the opposite, upward direction apparently right from the beginning of the experiment, with the highest point reached around 4:00 by most of the crown height percentiles. A return to the starting position was, however, not observed for the mean and some percentiles and it remains uncertain whether the branches would have resumed the starting position later during the day or whether the movement was induced by other factors than circadian rhythm.

Trachycarpus fortunei (Figure S4) stands out in its growth form by having an unbranched stem. Most crown movement must therefore be attributed to the leaves. This species was characterized by a short downward movement, followed by upward movement of most crown height percentiles until 01:00 a.m., then downward movement until 4:30 and finally upward movement again. It is not clear whether this pattern reflects a short circadian cycle similar to P. x fraseri, or whether it is the result of a longer cycle with a return to the starting position later during the day.

Some trees showed more or less unidirectional movement through the duration of the experiment. In such cases the observed patterns were not sufficiently clear to conclude whether the movements represented sleep or not. The leaves of the third fabaceous tree included in the experiment, G. triacanthos (**Figure 14**) were observed to have entered sleep configuration before the beginning of the experiment. The crown height percentiles of the canopy moved slightly downwards between 06:30 p.m. and 07:00 p.m., then moved upwards by 9:00 p.m. until 01:00 a.m., when a slight downward movement was detected, after which the percentiles showed a clear increase in height again. The last position scanned was situated 5 cm away from the starting position and we did not observe any signs that this would be resumed later.

The crown of Morus alba (Figure S5) moved constantly upward during the experiment, with only a slight downward movement around midnight. The duration of the experiment was not sufficiently long to determine whether this pattern was due to sleep movement initiated before the scanning started at 06:30 p.m. or some other processes. However, since this tree appeared to be generally unhealthy, the observed movements might have been determined by other factors than sleep movement.

The bamboo, Fargesia murielae is characterized by having multiple stems, each with a uniform modular structure and slender first order branches carrying the leaves (Figure S6). For the lower two crown height percentiles (0.5 and 0.6) we recorded a pattern similar to other plants with sleep motion: a short downward movement between 06:30 a.m. p.m. and 07:00 a.m. p.m., followed by upward movement until 9:00 p.m., down until 04:30 a.m. and back up, but with the original position reached at 5:30 and downwards movement afterwards. This species was excluded from further analysis since the uppermost part of the crown was partly in contact with and supported by the greenhouse frame.

In some of the tress studied, the movement observed can be interpreted as non-cyclic movement superimposed on an overall circadian cycle. This particularly applies to Chamaecyparis nootkatensis (**Figure 15**), which moved upwards during most of the night. At 10:00 p.m. the apical part of the canopy showed an abrupt downward shift (the crown height percentiles 0.6, 0.7, 0.8, and 0.9). An hour later at 11:00 p.m., it had returned to the pre 10:00 p.m. position. The tree reached the highest position at 02:00 a.m. and

moved downwards until 5:00 a.m. Hereafter, it moved upwards again but did not resume the staring position. It should be emphasized that this particular tree has small scaly leaves and that the patterns observed were solely a result of branch movements.

Similarly, the crown of B. utilis (Figure S7) moved upward as expected until 8:00 p.m. The lowest point was reached at 02:00 a.m., and the highest point at 06:00 a.m. with slight downward movement afterwards. The crown did not return to its original position.

mean movement are shown in gray. Movement of this tree might be interpreted as a circadian cycle returning to the original position before the end of the measurement.

calculating mean movement are shown in gray. Movement of this tree seems to be dominated by aperiodic displacement.

#### Summary of Results

Changes in crown shape were observed for all studied trees, but clear and well-measurable sleep movements are apparently rare. Ambient light conditions were continuously dark between 7:30 p.m. and 6:00 a.m., but most changes in movement direction occurred during this period, suggesting that they were not controlled by ambient light. Aesculus and Acer had the highest periodic circadian movement amplitudes of the crown amounting to 2 cm, whereas the highest aperiodic change in crown shape was observed for Gleditschia (5 cm) and Fargesia (2 cm). We detected some shorter-period oscillation with cycles of 2–3 h and amplitudes between 1 and 0.5 cm, close to the margin of our measurement accuracy. Movement patterns recorded can be attributed to in three major types: circadian movement in an 8–12 h cycle (Nerium, Acer, Aesculus, Viburnum, Photinia), periodic movement between 6 and 2 h (Quercus, Magnolia, Pinus, Cedrus), and non-periodic, unidirectional movements (Styphnolobium, Prunus, Gleditschia, Morus). The remaining trees studied (Olea, Betula, Larix, Trachycarpus, Fargesia, Chamaecyparis) either show a combination of these basic movement forms or low-amplitude movements close to the margin of our accuracy or an apparently random pattern.

# DISCUSSION

### Measurement Accuracies and their Effects on the Results

The scanning method used here has a nominal ranging accuracy of 0.003 m on solid, fixed targets. This signifies that theoretically the results presented here can be influenced by errors in position measurement. However, since these effects are probably random, their influence can be reduced by ensuring a large number of points in each height percentile. The differences in the number of points acquired for each tree during the individual scanning sessions fluctuate slightly (± 1%) mainly due to ghost points. This may constitute an additional source of error. Finally, the co-registration accuracy of the individual scans falls within 2.3 mm. including both horizontal and vertical errors. Horizontal co-registration errors have no effect on crown height percentile calculations since the point clouds representing the individual trees were cut out with the same 2D and height boundaries each time, allowing for at least 10 cm distance around the points observed to belong to the tree. Vertical co-registration errors might have an effect, but that is well characterized by the error observed at the co-registration targets since the reference spheres and the greenhouse structure were further from the sensor than the trees were. Co-registration errors would reveal themselves by spatial autocorrelation, which we did not detect in our data. In this study we followed the methodological approach used by Puttonen et al. (2016) as close as possible to make the results comparable. The relatively low number of measurement points on the trees limited more sophisticated analysis of position change, e.g., by segmentation or the iterative closest point algorithm (Glira et al., 2015).

The phase-based TLS successfully recovered sleep movement in some of the trees species studied, whereas no movement was detected in other of the trees. The small size of the trees included in the experiment probably constrained the magnitude of the movement. The scannings were not duplicated for practical reasons (for most species, the gardening shop had only one individual in stock) except for the two individuals of P. nigra, thus we cannot exclude that some of the canopy movements were undetected (false negatives). Also due to the small size of the trees it was decided not to affix standard reflectors to the branches that would have allowed for a more accurate tracking of movement. The draw-back of using such a method would have been that the weight of these reflectors could have influenced the movement patterns. Although the half-hour intervals between some scans represent a step forward compared to Puttonen et al. (2016), the timing of the survey was still not optimal. The duration of the experiment should ideally be 24 h, but for the interpretability of the results the scannings should at least have been initiated a couple of hours before and run for a few hours longer. Despite the exploratory nature of this study we have nevertheless revealed substantial variation of nocturnal plant movement across seed plants, which opens up a number of new research questions specifically addressing the dynamics of sleep movements in trees.

# A Conceptual Framework for Interpretation of Plant Movement

Our results clearly show that small trees move their canopy during the night in complex ways and that it is difficult to identify general patterns. Some of the movements detected were close to the preset noise threshold and although we find it unlikely, we cannot exclude effects caused by random noise. Based on the results of the present exploratory study we propose a framework that can be used for future reference of research focusing on the morphological and physiological implications of canopy movements. We propose that all movements are broken down in four basic and more or less interacting patterns, (1) sleep (circadian, light or internal clock controlled), (2) drift (non-circadian, a trend within the window of observation), (3) oscillation (short term fluctuation, typically superimposed on other types of movements) and, (4) noise (random fluctuation around a mean of zero within the expected noise limits).

As an initial rule of thumb we differentiated between canopies with and without circadian movement ("sleep") respectively by comparing the starting and end positions of these. Traditionally sleep movement has been defined as vertical, cyclic with approximately 12 h as period length (Darwin and Darwin, 1880; Puttonen et al., 2016). We classified movement as sleep if the tree returned to its starting position 8–12 h after the start of the experiment, and if one or two clear minimum/maximum height positions could be identified. Most of the trees included in this study returned to their evening positions the morning after thereby moving full circle, whereas others showed unidirectional movement and never resumed their position at the beginning of the experiment. Sleep movement governed by an internal circadian clock has already been described e.g., in Arabidopsis thaliana (Barak et al., 2000; McClung, 2013). Canopy movement may be an adaptation to light exposure, and may occur in both upward or downward directions during the night. The movement pattern during the day is expected to be the opposite (Mullen et al., 2006). The movement itself is linked to turgor pressure in cells situated in places of the plant body where their effect on organ motility is optimal. This applies e.g., to the bulliform cells in grass leaves, the lodicules in grass flowers and to cells in the pulvinus found in plant families such as Fabaceae and Maranthaceae that controls movement of the leaf lamina. When it comes to movement of branches, much less is known but this is most likely a consequence of the hydraulic dynamics of the conducting tissues. The simplified view that phloem and xylem constitute separate transport systems with exclusive functions has recently been abandoned and today, we have a more refined understanding of the complex interaction between tissues of the inner bark region and the xylem, facilitated by radially aligned rays (Pfautsch et al., 2015). The palm included in this study is of course excepted since it is a monocot without rays. It has been shown that the diurnal pulse of stem expansion and contraction is driven by the daily cycle of transpiration and to a lesser degree by photosynthesis (Zweifel et al., 2001). The mechanisms behind the minute nocturnal branch movements revealed in this study are probably more complex and linked to physiological dynamics.

The second type of movement we propose here is "drift," which implies unidirectional movement. Observed movement was classified as drift if no periodic changes in movement direction occurred and the position farthest from the starting point was measured at one of the last scans in the morning. Since we only ran our experiment for 12 h we do not know whether these movements form part of a long-term change in the tree shape. Such long-term changes may be linked to stress caused by disease, herbivory, or lack of water and will eventually lead to wilting. It may also be a result of the plants preparations for hibernation. The study was conducted near autumnal equinox to assure an almost even length of night and day and we cannot exclude that some of the trees were already preparing for hibernation.

The third type of movement, which we refer to as "oscillation" includes short-term movements in the canopy that occur in cycles of a couple of hours or shorter. Movement was interpreted as oscillation if at least three local maxima or minima could be observed, with comparable amplitudes. Although the biomechanics and ecophysiological basis of these are unknown, movements are most likely linked to tissue specific changes in turgor pressure caused by action of membrane borne ion pumps. We can only speculate of the molecular mechanism leading to these rather fast changes in turgor pressure, yet one possibility could be pulses in the way osmotically active metabolites are stored and released. In that case the movements could be regarded as consequence of physiologically processes that are vital to the plants and the movement per se can therefore be regarded as having no adaptational implications.

Lastly random noise should also be considered as a driver of movement recorded. Here, we used a preset threshold of 5 mm to filter out the noise, well knowing that we could lose valuable information in this way. For convenience and lack of knowledge we used the same filter across all the plants studied. Future studies of plant movement should consider other ways of reducing noise especially in those species that show moderate changes in the canopy.

Observed movements are a result of a combination of these four types, therefore their interpretation might still be ambiguous. Combinations of two or three main movement types may be superimposed and can in such cases only be interpreted through the main trend and local deviations from this trend with shorter or longer periodicity.

#### Interpretation of our Observations Based on the Conceptual Framework

Among the species included in this studies A. palmatum and A. hippocastanea most clearly represent the sleep type of movement. In both species drift is negligible, whereas oscillations are superimposed on the general sleep pattern. The latter is most obvious in A. hippocastanea. Interestingly, both species are classified in the Sapindaceae in the malvid clade, and both have palmate, deciduous leaves. However, the leaves of A. hippocastanea are much larger and therefore contribute more to the movement of the crown. V. pragense and O. europea are also primarily showing sleep movement. The crown moves upwards in the early hours of the night and back down by morning. We were unable to distinguish drift or oscillations from the background noise. This was also the case for P. x fraseri, which nevertheless seems to have a short period of active oscillation movement before and especially after an 8-h period of sleep. In our study, N. oleander showed downward circadian movement, reminiscent of the movement identified by Puttonen et al. (2016) in B. pendula. The crown of Albizia shifted upwards until 8:30 p.m. and then downwards until it reached the lowest point at 03:00 a.m. After it had resumed the baseline position at 5:30 a.m., it continued upwards. This overall pattern in combination with leaf movement suggests that this species has a pronounced sleep cycle, but that the window of observation in our study was too short to completely capture it. These findings suggest that sleep movement can occur in trees both in downward direction as revealed by Puttonen et al. (2016) in B. pendula, as well as in the upward direction, or downward for part of the branches and upward for the rest as we observed for Olea and Viburnum. We can only speculate about the mechanisms leading to these differences and more plant physiological and biomechanical studies are needed to fully understand the driving mechanisms.

Clear cases of drift were revealed in Morus alba and F. murielae, with Morus moving upwards and Fargesia moving downwards. We cannot rule out that these plants later returned to their starting position since the experiment only lasted 12 h. The Morus alba tree revealed several signs of being unhealthy, which suggests that the unidirectional movement was a result of wilting of the plant. However, this was not evident in the Fargesia murielae and furthermore this bamboo showed signs of reversal to the starting configuration of the crown between 4:30 a.m. and 5:30 a.m. The fabaceous tree Gleditschia started moving downward in the evening and reached a low point at 8:00 p.m. Then it moved upward during the remaining part of the night and reached the most elevated position in the morning. This movement pattern can best be explained by downward sleep such as in A. julibrissin, but combined with an upward drift. Similarly the results suggest that P. virginica has upwards sleep movement. During early morning, the leaves started resuming the evening position. It is not clear whether they complete the full circle of movement, or whether the upward movement is repeated resulting in overall drift. In case of the palm tree T. fortunei the movement of the crown can be entirely attributed to the leaves. Our observations suggest an upward sleep movement for the upper part of the crown combined with a downwards drift of some of the lower part.

The sleep cycle of B. utilis is reminiscent of that described for B. pendula by Puttonen et al. (2016). It differs from this by having a barely distinguishable upwards drift superimposed.

Some trees revealed cyclic movements of shorter duration than the sleep movement, which we refer to here as oscillation. In two species we captured crown movements in 3 or 4 h cycles. This was most evident in Magnolia alba and somewhat obscure in Q. robur, being barely above the preset level set for background noise.

The gymnosperms mainly differed from angiosperms trees by the short-period fluctuations, which were more pronounced and spatially decoupled in the sense that the crown height percentiles did not show concerted movement such as it is more or less the case with angiosperms, but moved rather independently of each other. The results indicate that "sleep" movements may be weak or entirely absent in gymnosperms. The large individual of Austrian pine (P. nigra 1) revealed the clearest patterns of short-period movement of which individual pulses reached 0.6 cm. The small Austrian pine tree (P. nigra 2) had a similar pattern of erratic pulses deviating from the starting configuration at night. The individual percentiles of C. libani showed distinct peaks, whereas the mean of the percentiles was less clear and difficult to interpret. L. kaempferi was the only gymnosperm with movements reminiscent of "sleep" cycle. Superimposed on this pattern we captured an oscillation effect, which exceeded the preset background noise level for the individual percentiles as well as for the mean of all percentiles.

# Comparison with Taxonomy, Morphology, and Anatomy

The two representatives for the malvid clade, A. hippocastanea and A. palmatum showed similar sleep patterns. Superimposed on this pattern A. hippocastanea showed conspicuous oscillation, which we attribute to the larger leaves of this species as compared to A. palmatum. Olea and Viburnum both have rigid, sclerophyllous leaves, which may explain why no short-period movement was observed in these plants. Photinia also seems to match this pattern of no short-period movement, at least during its sleep period, which is shorter than for other trees. The three Fabaceous trees probably have downward sleep cycles, but the duration of the experiment was probably too short so the drift observed may actually instead be part of a longer sleep cycle. This may also apply to the drift recorded in P. virginica and B. utilis, which may have sleep cycles lasting longer than the duration of the experiment.

Apparently gymnosperms have no sleep movement, except for the single case of the smaller L. kaempferi in which sleep movement cannot be cannot entirely ruled out. L. kaempferi stands out among the gymnosperm species investigated by being deciduous and having needles with less supportive tissue. In general we saw a gradual dimming of pulsation movement from P. nigra through C. libani to C. nootkatensis, which is probably correlated to decreasing length of the needles. Interestingly, M. grandiflora is the only angiosperm where shortterm oscillation is overriding the circadian rhythm. This species represents a basal lineage within the angiosperms and as such has an isolated taxonomic position.

To sum up: sleep is variable in both direction (upward or downward) and extent and apparently determined by ancestry to a certain extent. Oscillation, which may be a universal feature, is probably controlled by the amount of supportive tissue in the leaves. Finally, drift is probably related to physiological phenomena such as, growth, senescence, and health of the plant.

# New Finding and Hypotheses

The demonstration by Puttonen et al. (2016) of sleep movements in two provenances of B. pendula left the question outstanding whether it is a widespread phenomenon across seed plants. Our results indicate that several types of circadian crown movement occur and that some trees move more than others. Some of the movement patterns detected were quite unique but at the same time so close to the preset detection threshold that we abstain from sweeping conclusions, but instead encourage further studies. The fact that our experiment only lasted 12 h may in some instances have prevented us from capturing the full extent of the sleep cycles and instead led us to interpret the observed pattern as drift since the tree did not resume the starting position at the end of observation. Puttonen et al. (2016) hypothesize that the observed branch movements are independent from sunlight and governed by the internal circadian clock of the plant. Here, we also show ample cases of canopy movement taking place during the dark period of night after sunset and before sunrise. The results of the present study suggest that movements are somehow linked to ancestry. Biomechanics and ecophysiology are probably key drivers that interplay in complex ways, but this needs to be further explored before speculations are made.

Nevertheless nocturnal crown movement is a general characteristic of the plants monitored in this study. More specifically in nearly all the plants included we observed the same crown movement cycles of a few hours duration, which were also revealed (but not discussed) by Puttonen et al. (2016). Circadian movements in plants have been explained by attributes of the diurnal rhythm of photosynthesis and the resulting fluctuation in turgor pressure (Solomon et al., 2010). Does this imply that shorter cycles of movements can be attributed to short term fluctuations in turgor pressure of yet unknown causes? Sap flow measurements suggest that fluctuations in turgor pressure more or less follow the diurnal rhythm in light exposure of the plant (Chapotin et al., 2006; Steppe et al., 2015). However, Ehrenberger et al. (2012) revealed some instances of short cycle fluctuations in turgor pressures during the night.

# Future Outlook

Our experiment is a trade-off between small-sized trees grown in containers under controlled conditions and fully developed trees growing in their natural environment where larger movements are expected due both to physiological changes and to wind. Due to the extremely high point densities the datasets generated in this study allow for more sophisticated analysis and as a consequence we have made all point clouds available online for further studies. Based on our results we clearly recommend that the duration of future experiments is more than 12 h, ideally 24 h. A draw-back of the laser scanning technique that we employed here is that the point cloud is rather unstructured. At the given resolution it is not trivial to keep track of individual parts of the trees and compare these across the species studied. Puttonen et al. (2016) solve this by affixing standard measurement targets to some branches, which are then automatically located in postprocessing. Alternatively, several processing techniques have been developed with the purpose of creating a 3-dimensional model from point clouds of trees obtained by TLS (Xu et al., 2007; Raumonen et al., 2013; Hackenberg et al., 2015), and these could potentially allow determining and tracking similar parts of the tree in consecutive scans to track their movement. However, such methods typically focus on modeling the branch structure without the leaves, which might be a limitation for detecting canopy movement. Alternatively, leaf segmentation approaches are also promising for detecting change maybe even at the individual leaf level (Béland et al., 2014; Koma et al., 2017). It seems feasible to alternatively use reflective prism foil for marking specific points on the branches, which can then be detected based on signal amplitude. Such a setup would furthermore open up for using a geodetic total station that can automatically locate and accurately measure the position of the reflective markers only. This would probably be a cheaper and more practical approach while delivering data that allows for similar or even more detailed spatial statistical analysis.

From a plant physiological point of view, the most daunting challenge is to explain the short-term cycles of movement. This can be done by using non-invasive methods to measure oscillations in water potential. The question is whether these are sufficiently sensitive to measure relatively small differences within a short time span. Another option is semi-invasive, measurement of conductivity between electrodes inserted into the trunk has been proposed as a metric of sap flow (Koppán et al., 2000). High precision measurements of tree girth may also give an indication of sap flow and pressure regulation but the question is again whether it is sufficiently precise enough to reveal any patterns (Pesonen et al., 2004). Finally, more invasive methods could be considered involving anatomical sectioning of fresh wood sampled at different phases of crown movement and immediately frozen in liquid nitrogen for later microanalysis of sugar contents at the tissue level.

Recent advances in laser technology open up for detailed studies of spatial dynamics in plants such as presented here.

#### REFERENCES


Since studies like this are still in their infancy they raise more questions that they answer. However it is very clear that plant movement is understudied, and that a set of promising methods exist that, in combination with TLS, can be expected to provide fundamental new knowledge on the physiology of plants. Our results give an impression of complex relationships that are controlled by many drivers and perhaps when mapped in detail reveal information on the plants adaptation to the environment and its interactions with other species. We have tried to reduce this complexity by establishing a rough conceptual framework within which phenomena can more easily be described.

# AUTHOR CONTRIBUTIONS

AZ planned and coordinated the study, performed data processing and analysis and wrote most of the main text. BM carried out the laser scanning measurements and the preprocessing of the point clouds, and wrote the methodology of the scanning. AB contributed to the analysis and interpretation of the results and to the discussion section. All authors worked together to produce the final version of the text.

# FUNDING

AZ was supported by NKFIH OTKA PD 115833 and joint work on data analysis was allowed by a study visit to Aarhus University funded by the Department of Ecoinformatics and Biodiversity and OTKA ERC\_16\_M 122670.

# ACKNOWLEDGMENTS

We gratefully acknowledge the collaboration of the "Sylvestris" garden shop staff in support of our measurements, and the assistance of Katharina Prager (BOKU, Vienna) during the experiment.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpls.2017. 01814/full#supplementary-material


pendula) branches and foliage with short interval terrestrial laser scanning. Front. Plant Sci. 7:222. doi: 10.3389/fpls.2016.00222


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer FB and handling Editor declared their shared affiliation.

Copyright © 2017 Zlinszky, Molnár and Barfod. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Aerial Images and Convolutional Neural Network for Cotton Bloom Detection

Rui Xu<sup>1</sup> , Changying Li <sup>1</sup> \*, Andrew H. Paterson<sup>2</sup> , Yu Jiang<sup>1</sup> , Shangpeng Sun<sup>1</sup> and Jon S. Robertson<sup>2</sup>

<sup>1</sup> Bio-Sensing and Instrumentation Lab, College of Engineering, University of Georgia, Athens, GA, United States, <sup>2</sup> Plant Genome Mapping Laboratory, Department of Genetics, University of Georgia, Athens, GA, United States

Monitoring flower development can provide useful information for production management, estimating yield and selecting specific genotypes of crops. The main goal of this study was to develop a methodology to detect and count cotton flowers, or blooms, using color images acquired by an unmanned aerial system. The aerial images were collected from two test fields in 4 days. A convolutional neural network (CNN) was designed and trained to detect cotton blooms in raw images, and their 3D locations were calculated using the dense point cloud constructed from the aerial images with the structure from motion method. The quality of the dense point cloud was analyzed and plots with poor quality were excluded from data analysis. A constrained clustering algorithm was developed to register the same bloom detected from different images based on the 3D location of the bloom. The accuracy and incompleteness of the dense point cloud were analyzed because they affected the accuracy of the 3D location of the blooms and thus the accuracy of the bloom registration result. The constrained clustering algorithm was validated using simulated data, showing good efficiency and accuracy. The bloom count from the proposed method was comparable with the number counted manually with an error of −4 to 3 blooms for the field with a single plant per plot. However, more plots were underestimated in the field with multiple plants per plot due to hidden blooms that were not captured by the aerial images. The proposed methodology provides a high-throughput method to continuously monitor the flowering progress of cotton.

#### Edited by:

Norbert Pfeifer, Technische Universität Wien, Austria

#### Reviewed by:

Maria Balota, Virginia Tech, United States Patrick Tutzauer, University of Stuttgart, Germany

> \*Correspondence: Changying Li cyli@uga.edu

#### Specialty section:

This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science

> Received: 30 September 2017 Accepted: 19 December 2017 Published: 16 February 2018

#### Citation:

Xu R, Li C, Paterson AH, Jiang Y, Sun S and Robertson JS (2018) Aerial Images and Convolutional Neural Network for Cotton Bloom Detection. Front. Plant Sci. 8:2235. doi: 10.3389/fpls.2017.02235 Keywords: cotton, flower, bloom, unmanned aerial vehicle, point cloud, convolutional neural network, phenotyping

# INTRODUCTION

Improving cotton yield is one of the main objectives in cotton breeding projects and cotton growth and production management. Yield can be defined as the product of the number of bolls produced per unit area and the mass of lint per boll (a common measure of boll size). Cotton yield is associated with many physiological variables and environmental factors. However, an increase in cotton yield is generally associated with an increase in the number of bolls regardless of genotype or environment (Wells and Meredith, 1984; Pettigrew, 1994). Flower and boll retention will affect the final number of bolls produced. Therefore, processes that affect flower and boll retention will have a significant impact on yield.

**46**

Flowering is important to cotton yield because pollinated flowers form cotton bolls. Within a given genotype, seasonal flower production per unit area is more closely related to yield than boll retention percentage (Heitholt, 1993, 1995). Since a cotton flower is white (cream-color for some upland germplasms and yellow for Pima) on the first day it opens and turns pink within 24 h, it is unlikely to mistake an old bloom for a new bloom on separate days. Therefore, if bloom counts are obtained daily then the seasonal total counts can be calculated. The flowering time (the time of the first flower opening) and the peak flowering time can be determined accordingly, both of which are critical to production management. The timing and duration of the flowering stage also reflect the difference in growth habits of different genotypes, which can help breeders select specific genotypes, for example, short-season or long-season genotypes. Although manual counting is perhaps the simplest and most reliable way to count flowers, it is tedious and inefficient and requires a massive amount of labor, which is not practical for large fields.

Imaging methods can improve the efficiency of manual counting. Many studies have been done on flower detection and classification using color images (Adamsen et al., 2000; Siraj et al., 2010; Hsu et al., 2011; Biradar and Shrikhande, 2015; Seeland et al., 2016; Thorp et al., 2016). Flowers usually have distinct color and texture from the background; therefore, traditional image processing methods such as color and texture analysis can be used to segment flower pixels (Thorp et al., 2016). The number of flowers can be calculated using morphological operations on the segmented flower pixels or correlated with flower pixel percentage (Biradar and Shrikhande, 2015; Thorp et al., 2016). Flower features can be extracted from flower images and are used to recognize flower species (Hsu et al., 2011). Machine learning techniques can be used to classify different flower types, which could be useful to determine the age of cotton flowers based on differences in color and shape (Seeland et al., 2016). Deep learning methods such as the convolutional neural network (CNN) have been demonstrated to be effective in recognizing flower species (Liu et al., 2016). CNN showed advantages over traditional machine learning methods because it does not require extraction of image features.

Although imaging methods are efficient in detecting flowers, the image collection throughput limits its usage in agriculture because agriculture usually deals with large fields. To improve the data collection throughput, the use of an unmanned aerial vehicle (UAV) is usually preferred over a ground vehicle or a robot because a UAV can provide superior data collection speed and larger spatial coverage. UAVs also do not interact with the plants, so constant data collection will not cause soil compaction and plant damage, which can happen when using a ground vehicle. Although many researchers have used UAVs for agriculture studies in recent years, only a few used UAVs to count flowers for crops. For example, aerial multispectral imaging has been used to calculate flower fraction in oilseed rape and monitor the peach flower (Fang et al., 2016; Horton et al., 2017). However, those studies only segmented flowers from the canopy rather than counting the flowers. To our knowledge, no study has been done to count flowers using aerial images. The main reason for this is the low image resolution compared to images taken from ground platforms, making image processing challenging.

In this paper, we aimed to develop a methodology for counting cotton blooms using aerial color images. The overall objective of this paper was to develop a data processing pipeline to detect and count the number of newly opened cotton flowers using aerial images. Specific objectives were to: (1) build and train a CNN to classify flowers, (2) construct dense point clouds from raw images and evaluate the quality, (3) develop an algorithm to register flowers, and analyze its accuracy and efficiency using simulated data, and (4) evaluate the performance of the data processing pipeline compared with manual counting. To avoid ambiguity, we used the term bloom to refer to a newly opened flower to distinguish it from its other growth stages.

# MATERIALS AND METHODS

#### Test Fields

Two test fields were used, both of which were located in the Plant Science Farm in Watkinsville, GA (33◦ 43′ 37.80′′N, 83◦ 17′ 57.52′′E) (**Figure 1**). Field 1 had 132 plots with a single cotton plant in each plot, arranged in 12 rows and 11 columns with a 1.5 m (5 ft) distance in both row and column directions. The cotton in field 1 was planted on May 25, 2016. Field 2 had 128 plots of cotton, arranged in 16 rows and 8 columns (**Figure 1**). Each plot was 3 m (10 ft) long and 1.5 m (5 ft) wide, with a 1.8 m (6 ft) alley between each plot. Fifteen cotton plants were planted in each plot with equal spacing of 0.15 m (6 inches) on June 13, 2016. The plant density varied because of the different number of plants germinated, resulting in some empty plots in field 1 and field 2. The number of germinated plants in each plot for field 2 was recorded 2 weeks after planting (June 8, 2016) to be used to calculate the average number of blooms for each plant.

#### Data Collection

Aerial color images of the test fields were collected using an octocopter (s1000+, DJI, China) with a color camera (Lumix DMC-G6, Panasonic, Japan) on August 12, August 19, August 26, and September 9, 2016 (**Table 1**). The color camera was directly mounted on the bottom of the drone with the lens facing downward. An inertial measurement unit (IMU) with GPS was mounted on the drone to record the location and orientation of the camera. Raspberry Pi 2 was used to trigger the camera and record the IMU/GPS measurement. During the flight, the camera took images at a frequency of 1 Hz. The exposure time and aperture were manually set for the color camera according to the light conditions of the field, and auto-settings were used for other parameters. The drone was flown at a height of 15 m above ground level (AGL) to reduce the ground sample distance (GSD). One flight was used to collect data from both of the test fields.

The number of blooms in each plot was manually counted as a reference for the image method. To count the blooms in each plot, only white flowers were counted in each plot on the same day that aerial images were collected. Although manual counting is reliable overall, it is possible for some blooms to be counted multiple times or not counted. The judgment of whether a flower is white is subjective and varies from person to person, which

TABLE 1 | Data collection summary.


could cause human counting errors. For field 1, blooms in all of the plots were counted on all 4 days. For field 2, blooms were counted in all plots on August 12, and in 32 random plots on the other 3 days.

#### Data Processing Pipeline

The overall data processing pipeline for bloom counting can be divided into four key steps: (1) dense point cloud generation, (2) plot images extraction, (3) bloom detection, and (4) bloom registration (**Figure 2**). The final output of the pipeline is the bloom count for each plot. Other information such as bloom position can be obtained too.

#### Dense Point Cloud Generation

A dense point cloud of each test field was generated in PhotoScan (PhotoScan Professional 1.3.1, Agisoft, Russia) using the raw color images and the IMU/GPS measurements. The high accuracy and reference preselection settings were used for photo alignment, the high-quality setting for dense point cloud build, and default for other settings. After photo alignment, PhotoScan constructed a tie point cloud—often called a sparse point cloud in other software—from the feature points detected in the images. Before constructing the dense point cloud, the tie point cloud was used to calculate the accuracy of the dense point cloud.

After building the dense point cloud, it was found that field 1 was not fully covered by the images collected on August 19 and those uncovered plots were excluded from further data analysis. Additionally, some plots had an incomplete point cloud for the canopy due to low side-overlap on August 19, August 26, and September 9, when the focal length was enlarged to reduce the ground pixel size. Point cloud coverage, which is the percentage of the plot constructed with valid point cloud, was used to evaluate the completeness of the point cloud. The plots with point cloud coverage less than 80% were excluded from further analysis. The method to calculate point cloud coverage is explained in section Quality of the dense point cloud.

#### Plot Images Extraction

The main purpose of the plot images extraction step is to export plot images from the raw images such that one plot image contains only one plot or part of the plot. Since one plot can be imaged several times, one plot has multiple plot images from different raw images. This is helpful for detecting blooms inside the canopy because different raw images provide views of each plot from different angles, which greatly improves the chance of one bloom being imaged compared to the ortho-image, which only shows the top view of the canopy. To extract the plot images, plot boundaries were first manually drawn for each plot and stored as quadrangle shapes in PhotoScan. Then the four vertices of each shape were projected to each raw image to get the pixel location of the vertices in each image. One image was considered to cover part of the plot if two or three vertices were in the image (pixel location is within the boundary of the image), and the whole plot if four vertices were in the image. The pixel location of the four vertices was recorded and plot images were extracted from the raw image accordingly. The step was processed in PhotoScan using built-in Python functions. The image file name, plot number, and pixel locations of the four vertices were saved in a text file and imported into MATLAB (MATLAB 2017a, MathWorks, USA) to extract plot images from raw images.

#### Bloom Detection

The bloom detection process contains two steps. The first step is to screen out the locations that most likely contain cotton blooms based on the fact that cotton blooms usually have the highest pixel intensity because of their white color. The plot image was first transferred into CIELAB color space and then the screening was performed using the L channel. A threshold of 0.75 on the normalized L channel (normalized by the maximum value in L channel) was used to extract bright objects. The number of pixels for each object was counted and noisy small objects less than 15 pixels were removed. Objects separated by a distance of less than the diameter of a flower—which is about 20 pixels for our dataset—were combined into one object because some flowers were split into two objects by the leaves. Images with a size of 36 by 36 pixels around the center of the remaining objects were extracted from the plot image. Those images were treated as potential bloom images.

The second step is to classify the potential bloom images into bloom and non-bloom class using the pre-trained CNN (detailed information on the CNN and training process is described in section Convolutional neural network). After classification, the images classified as bloom class were kept and the pixel location of those images in the raw color images was recorded.

#### Bloom Registration

Since the same bloom can be detected in several images, it is necessary to register the bloom before counting the blooms to prevent counting the same bloom more than once. Registration methods based on image features were not useful because very few features could be detected from the bloom images due to the resolution limit. Therefore, we first projected the pixel location of each detected bloom into the 3D location in the dense point cloud, and then clustered the blooms based on the locations because the same bloom should have the same 3D location in the dense point cloud. The projection was done in PhotoScan. Because each pixel in the image has a corresponding image ray, any point in the image ray will appear as the same pixel in the image, and the 3D location of that pixel in the point cloud is the intersection between the image ray and the point cloud. If there is no intersection between the image ray and the point cloud, PhotoScan will return the closest point to the image ray.

Due to the accuracy of the point cloud and pixel location, the 3D location of the bloom may deviate from the true location, which increases the chance of incorrect bloom registration solely based on the location. However, since multiple blooms detected in the same image are different blooms and cannot be in the same cluster, the bloom registration based on the 3D position can be generalized as a constrained clustering problem as follows:

Given the set of data points and the set of their corresponding classes, form the data points into clusters so that no data points in the same class will be in the same cluster and the distance between any two clusters are not smaller than a threshold.

In the case of bloom registration, the data point is the bloom position and the class is the image number to indicate in which image the bloom was detected. Although existing constrained clustering algorithms can solve the problem, most of the algorithms are for general clustering problems and are not efficient for this specific problem. Therefore, inspired by hierarchical clustering, a deterministic clustering algorithm was designed specifically for the bloom registration problem. The algorithm initializes with each data point as one cluster. The algorithm involves cluster selection and merging. First, for each cluster i, a set of clusters, S, was selected from all of the clusters such that no data points in S had the same class as any data point assigned to cluster i. Second, the distance between cluster i and every cluster in S was calculated and cluster i was merged to the closest cluster in S if their distance was smaller than the threshold λ. The algorithm repeats the selection and merging until no merging happens. The algorithm has a similar effect as the regular hierarchical clustering algorithms when the distance between two clusters is measured using Equation (1).

$$\text{dist}\left(\mu\_i, \mu\_j\right) = \begin{cases} \text{dist}'\left(\mu\_i, \mu\_j\right), \text{ if } E\_i \cap E\_j = \bigotimes \\ +\infty, & \text{otherwise} \end{cases} \tag{1}$$

because the distance calculation is only performed on a subset of the clusters.

Euclidean distance was used to measure the cluster dissimilarity. However, because of the large error of the 3D position projection on the z-axis, the Euclidean distance was modified by adding a weight term on the z-axis to adjust the influence of z-axis on the distance. The weighted Euclidean distance between a = ax, ay, a<sup>z</sup> and b = bx, by, b<sup>z</sup> was calculated using Equation (2). The final program used 0.5 for w.

$$dist\left(a, b\right) = \sqrt{\left(a\_x - b\_x\right)^2 + \left(a\_\circ - b\_\circ\right)^2 + \nu^2 \left(a\_z - b\_z\right)^2} \quad \text{(2)}$$

#### **Bloom registration algorithm**

**Input:** Set of data points D = {xi} n i , class E = {ei} n i , and distance threshold λ. **Result:** The assignment variables for the data points, z = [z1, z2, . . . , zn]. **Initialization:** each point is assigned as one cluster, z<sup>1</sup> = 1, z<sup>2</sup> = 2, · · · , z<sup>i</sup> = i, · · · , z<sup>n</sup> = n **while** not converged **do for** i = 1 : n **do** Find cluster i and its center µ<sup>i</sup> **if** cluster i exists **then** S ← ∅ ; Given the current assignment of points, find the set of classes of the data points assigned to cluster i, E<sup>i</sup> = {e<sup>k</sup> ∈ E|z<sup>k</sup> = i} ; **for** j = 1 : n **do** Find cluster j and its center µ<sup>j</sup> **if** cluster j exists **then** Given the current assignment of points, find the classes of the data points assigned to cluster j, E<sup>j</sup> = e<sup>k</sup> ∈ E <sup>z</sup><sup>k</sup> <sup>=</sup> <sup>j</sup> ; **if** E<sup>i</sup> ∩ E<sup>k</sup> = ∅ **then** // Add the current cluster to the selected clusters. S ← S∪µ<sup>k</sup> ; **end end end end** - dmin, imin ← min dist µi ,**s1** , · · · , dist µi ,**s**|S<sup>|</sup> ; **if** dmin < λ **then** //Merge cluster i with imin and update the assignment variables; Assign imin to the assignment variables for the data points in cluster i; **end end for** k = 1 : n **do** Given the current assignment of points, find the set of points assigned to cluster k, D<sup>k</sup> = x<sup>i</sup> ∈ D <sup>z</sup><sup>i</sup> <sup>=</sup> <sup>k</sup> : **if** |D<sup>k</sup> | > 0 **then** µ<sup>k</sup> ← P **<sup>x</sup>**i∈D<sup>k</sup> **x**i |D<sup>k</sup> | **else** Remove the cluster and apply the changes to the related variables ; **end end end**

where <sup>µ</sup><sup>i</sup> and <sup>µ</sup><sup>j</sup> are the center of the cluster <sup>i</sup> and <sup>j</sup>, dist′ µi ,µ<sup>j</sup> is the distance metric used in the hierarchical clustering algorithms, and E<sup>i</sup> and E<sup>j</sup> are the set of classes of the data points assigned to cluster i and j. Compared to the regular hierarchical clustering algorithms, the efficiency of our algorithm is improved

#### Convolutional Neural Network

The CNN was used to classify potential bloom images. The network had seven layers, one input layer, two convolutional layers, two max pooling layers, and two fully connected layers (**Figure 3**). Since the classification was performed on the potential bloom images, the potential bloom images were first extracted from individual data sets and manually labeled into bloom and non-bloom classes. Then the labeled potential bloom images were used to construct the image database. In total, ∼14,000 images for the bloom class and ∼60,000 images for the non-bloom class were extracted. Two thirds of the bloom images (roughly 9,000 images) and the same number of non-bloom images were randomly selected as training samples. The remaining images were used as testing samples. The CNN was trained for 30 epochs. The learning rate was set to 0.01 at the first epoch and decreased by a factor of 10 every 10 epochs. The mini batch size was set to 256. The regularization factor was set to 0.01 to prevent overfitting.

# Quality of the Dense Point Cloud

The quality of the dense point cloud of the canopy greatly affects the accuracy of the 3D position of each bloom, and thus the clustering results, and ultimately the bloom numbers. The quality of the dense point cloud can be evaluated from two aspects—the accuracy and the completeness of the dense point cloud.

The position accuracy of each point in the point cloud is difficult to calculate because the geometry of the canopy is unknown. Instead, the projection and reprojection error were calculated for the tie points to estimate the overall accuracy of the dense point cloud. The projection error was calculated as the distance between the projection of the feature points to the point cloud and their corresponding tie points. The reprojection error was calculated as the pixel distance between the projection of the tie points on the images and their original corresponding feature points. The projection error was analyzed on each single axis (easting, northing, and elevation) and the summation of three axes.

The completeness of the dense point cloud quantifies how completely the point cloud represents the real object or scene, and can be measured by the density of the point cloud. The completeness of the point cloud on the easting-northing plane was calculated since the camera only captures the top view. First, the point cloud was rasterized into a 2D elevation map using grid steps of 1.3 cm without interpolating the empty cells. The elevation map was divided into each plot. For each plot, the ground surface was calculated using the Maximum Likelihood Estimation SAmple Consensus (MLESAC) (Torr and Zisserman, 2000). The plot was divided into ground and canopy using the elevation map based on the distance to the ground surface with a threshold of 0.1 m. The point cloud coverage (PCC) for the canopy is defined in Equation (3).

$$PCC = \frac{Area\ of\ the\ canopy}{Area\ of\ the\ canopy + Area\ of\ the\ empty\ cells} \tag{3}$$

The area of the empty cells was included in the denominator because the empty cells were usually part of the canopy. The PCC estimates the completeness with which the canopy is represented by the dense point cloud, which can greatly affect the bloom registration result.

## Evaluation of the Bloom Registration Algorithm

To evaluate the efficiency and accuracy of the bloom registration algorithm, artificial data was generated in three steps. First, a data point with three dimensions was generated by randomly drawing integers from 0 to 9 for each axis. This step was repeated until 125 different data points were generated. The same class number was assigned to the 125 data points to simulate that they were from the same image. Second, the first step was repeated 10 times, generating in total 1,250 data points. A new class number was assigned to the 125 data points at each repeat to simulate that the data points from each repeat were from different images. The data points with the same coordinates were set in the same cluster, which is the true clustering result. Third, a random error that followed N 0, √ 3 3 σ was added to each axis of each data point, thus the position error (vector summation of the errors of the

three axis) followed N (0, σ). Different σ was used to simulate the noise level of the 3D position of the bloom. The 1,250 data points were clustered using the bloom registration algorithm and compared with the true clustering result.

Since the clustering result will depend on the noise level of the data and threshold for the clustering algorithms, 10 different σ values from 0.1 to 0.5 with an interval of 0.05, and 5 threshold values were tested. The threshold values were σ, 1.5σ, 2σ, 2.5σ, and 3σ and they covered from 68 to 99.7% of the position error. To acquire statistics of the result, the third step of generating artificial data was repeated 10 times to generate 10 sets of artificial data for each noise level and threshold. The mean and standard deviation of the clustering error and runtime on the 10 data sets were analyzed. The misclustering rate was defined using equation 4.

$$m \text{isclustering rate} = \frac{number \text{ of misclassified points}}{total \text{ number of points}} \tag{4}$$

Since it is difficult to relate the clustering result with the ground truth, the misclustered points were considered as the points that were in the same cluster from the clustering result but not in the same cluster from the ground truth. The number of misclustered points was calculated with two approaches. The first approach referred to as misclustering rate by reference—used the true clustering result as the ground truth, and the second approach referred to as misclustering rate by result—used the clustering results as the ground truth. The first approach was good at evaluating the clustering error when the clustering algorithm over clustered the data points, while the second approach was suitable for under-clustering errors.

#### RESULT

#### Quality of the Dense Point Cloud

The point cloud accuracy was analyzed using the dataset collected on 8/12. More than 99% of the reprojection errors of the tie points were less than 1.7 pixels and only a few were larger than 2 pixels (**Figure 4**). The mean reprojection error was 0.5 pixels. The tie points had overall larger projection error on the elevation than the easting and northing (**Figure 5**). The mean projection error for the easting and northing was 0 m because the positive and negative errors were canceled out, whereas the mean projection error for the elevation was 0.014 m, which is reasonable since the depth generated from multi-view stereo is usually the least accurate. The elevation also had larger variation than the easting and northing. The mean projection error for the three axes combined was 0.022 m, and 99% of the errors were between 0 and 0.127 m (**Figure 5D**). The large projection error on the elevation validates the rationale to assign a weight to the elevation when calculating the cluster distance in bloom registration.

In the event of an incomplete dense point cloud for the canopy, some blooms may not be able to get 3D point positions and thus cannot be counted. If the blooms equally distribute over the canopy, the point cloud coverage can be considered as the probability of a bloom having a valid point in the dense point cloud. Assuming a canopy has n blooms and point cloud coverage

of p, then k (k = 1, 2, . . . , n) blooms not having valid points (which cannot be counted using the imaging method) follow a binomial distribution B n, 1 − p . The mean count error is n 1 − p and the relative error is 1 − p . Therefore, the point cloud coverage should be close to 1 to make the counting error small.

**Figure 6** showed that the 8/12 dataset had good point cloud coverage (> 0.9) on most of the plots for both test fields. However, the other three datasets showed low point cloud coverage on both fields, except that the 8/19 dataset had some plots with good point cloud coverage. The low point cloud coverage was mainly due to the insufficient image side-overlap (<60%); the PhotoScan was unable to construct valid 3D points for areas that were covered by less than three images. Therefore, increasing the image overlap can improve the point cloud coverage. The depth filter inside the PhotoScan removed noisy points due to the movement of the plants and image noise, which was another reason for the low coverage. To achieve low image counting error, the plots with point cloud coverage lower than 0.8 were excluded from data analysis, which removed 16 plots in field 1 for the 8/12 dataset, 18 plots in field 1 and 103 plots in field 2 for the 8/19 dataset, all the plots in field 1 and 112 plots in field 2 for the 8/26 dataset, and 127 plots in field 1 and 97 plots in field 2 for the 9/9 dataset.

#### Training Result of the Convolutional Neural Network (CNN)

With more than 28,000 training images and mini batch size of 256, the CNN was trained about 114 iterations every epoch. Because of the large training sample size and relatively simple structure, the training process converged quickly and the training accuracy reached 0.94 in the first epoch. As shown in **Figure 7**, the training accuracy for bloom and non-bloom class increased in the first few epochs. The training accuracy for the non-bloom class reached a maximum on the 8th epoch, and the training

accuracy for the bloom class reached a maximum on the 12th epoch. The accuracy for both classes decreased after reaching the maximum value, fluctuated a bit, but then reached a stable accuracy with small variation after 20 epochs. The training loss decreased quickly over the first few epochs, which indicates the quick convergence of the training. After 10 epochs, where the learning rate changed from 0.01 to 0.001, the training loss kept decreasing but the change rate was small. After 20 epochs, the training loss reached the minimum with a certain level of oscillation, which indicates that the training process should

stop at the 20th epoch because no further improvement of the CNN can be gained from the last 10 epochs. Therefore, the training result at the 20th epoch was used to classify the potential bloom images. The testing accuracy was smaller than the training accuracy for both classes after 7 epochs but the difference between them was less than 0.01, which showed the CNN worked well on both the training and testing sets.

The classification result for each dataset using the trained CNN varied due to the different lighting conditions and flowering stages (**Table 2**). The classification result showed high precision (>0.9) for both classes across all datasets. However, the recall (true positive rate) for the bloom class was low for the 8/19 and 9/9 datasets. The CNN can cause overestimation or underestimation when comparing the number of predicted blooms and the number of actual blooms. For example, in the 8/12 dataset, the number of predicted bloom images was 789, but the actual number of bloom images was 719, thus the classification result overestimated the number of blooms by 10%. Similarly, the classification result overestimated the number of blooms by 23 and 29% in the 8/19 and 9/9 datasets, and underestimated the number of blooms by 2% in the 8/26 dataset. Objects such as cotton bolls, specular highlights on leaves, and pink flowers were misclassified as blooms because their shape and color appeared like a bloom in the aerial image due to the limited resolution (**Figure 8**). The misclassified blooms caused by objects (e.g., label sticks) on the ground could be removed based on the height from the ground, but the misclassified blooms caused by the plants (e.g., leaves, pink flowers, and cotton bolls) were difficult to eliminate using the height. Small blooms or partly hidden blooms can easily be misclassified as non-blooms because of their size. Blooms in the shade can also be misclassified as non-blooms because their intensity is reduced. Including those misclassified images into the training set may further improve the CNN performance.

# Efficiency and Accuracy of the Bloom Registration Algorithm

To cluster the 1,250 data points, the bloom registration algorithm used 13.3 to 48.4 s, which is 0.0106 to 0.0387 s per data point (**Figure 9A**). Using a larger threshold and larger noise level reduced the runtime because they resulted in a smaller number of clusters and thus the number of distance calculations. The bloom registration algorithm produced a near-zero misclustering rate by result at smaller noise level no matter what threshold was used because the distance between two reference clusters was still larger than the spread of the data points due to the noise (**Figure 9B**). As the noise level increased, the misclustering rate by result increased and larger threshold values had larger misclustering rates. This was mainly because of over-clustering when the noise of the points became large enough that certain portions of two clusters overlapped with each other and the algorithm could cluster them into one cluster. The misclustering rate by reference did not change significantly as the noise level


increased (**Figure 9**). Smaller thresholds had larger misclustering rates by reference.

At low noise level, the under-clustering takes the main effect because the bloom registration algorithm can split one cluster into smaller clusters at low noise level, generating a larger number of clusters compared to the real cluster number (**Figure 9D**). As the noise increased, a smaller number of clusters than the real number of clusters was generated, resulting in over-clustering. The bloom registration algorithm had a smaller cluster number with larger threshold at the same noise level, indicating that it is more prone to under-clustering but less prone to over-clustering.

#### Bloom Count Result

This section shows the bloom count results after removing the plots with point cloud coverage less than 0.8. The image count and manual count had the same trend for both fields (**Figure 10**). The error of the image count was between −4 and 3 for field 1, and the between −10 and 5 for field 2, showing that field 2 had more underestimated plots than field 1 (**Figure 11**). This may be due to the single plant per plot layout in field 1. The single plant per plot layout allows the plant to be seen from all directions without being blocked by other plants in the plot. Therefore, blooms in field 1 were more likely to be captured in the aerial images.

The overestimation was caused for two major reasons. One reason is the classification error of the CNN. For example, some leaves with specular highlights were classified as blooms (**Figure 12A**). The other reason is the quality of the point cloud, which causes the error of the bloom's 3D position and thus make the bloom registration incorrect (the same bloom from different images was registered as different blooms) (**Figure 12B**). The underestimation was caused by hidden blooms that were not shown in the aerial images, or blooms that were shown in the images but were not classified correctly by the CNN.

With enough datasets, it is possible to monitor the development of the flowers over time, which is one of the advantages of the proposed method. **Figure 13** demonstrates the trend of the bloom development. It shows that the number of blooms was low at the early flowering stage, reached the peak in the middle flowering stage, and then decreased at the late flowering stage. This trend is also consistent with the trend from manual counting (Figures S1–S5).

#### DISCUSSION

This paper demonstrated a high-throughput methodology to count cotton blooms using aerial color images. Continuously monitoring the cotton blooms over time can provide information about the cotton growth status, such as the flowering time and peak flowering time, which can be used for production management and yield estimation. It is also helpful for breeding programs to identify short season or long season genotypes.

Unlike sorghum or corn, whose flowers open at top of the stem, cotton flowers open from the bottom and progress up the plant. At the early cotton flowering stage, blooms at the bottom can be covered by the leaves and may not be able to be captured by the aerial images. Therefore, it is expected that image counting

can underestimate the real bloom count. Instead of using the ortho-images which only take the top view of the canopy, we utilized all the raw images that take different views of a plot to get the bloom count in order to improve the underestimation to a certain extent. The inability to detect hidden blooms inside the canopy from aerial images is the major limitation of the proposed methodology. This issue could be addressed by using oblique aerial images or ground side-view images. The underestimation could be improved at the middle and later flowering stages since the flowers that open at the middle and top of the canopy are more likely to be imaged in the aerial images.

Besides the inability to detect hidden flowers, the proposed bloom counting method can generate errors from two aspects: the bloom detection error and the bloom registration error. The bloom detection error was affected by the image quality, the threshold to select the potential bloom images, and the accuracy of the CNN. The image quality was affected by the illumination condition at the time of data collection. Ideally, the best image quality can be obtained when the illumination is uniform from all directions so there are no shadows or specular highlights. The ideal condition is hard to obtain in the field, but the approximate ideal condition can be obtained on a cloudy day. When the sunlight directly shines on the canopy without being scattered by clouds, shadows and specular highlights can be found in the aerial images, which cause non-uniformed intensity changes of the plot. Those changes will affect the selection of the potential bloom positions.

The threshold to select the potential bloom positions was arbitrary and mainly based on the images. A high threshold can eliminate some blooms that have low intensity due to shadows. A low threshold can include more non-bloom objects (such as the specular highlights), which can increase the processing time and be misclassified by the CNN. The classification of the potential bloom images relied on the CNN but misclassifying a bloom image as a non-bloom image affected the result differently from misclassifying a non-bloom image. When a bloom was classified as a non-bloom in one image, it was possible to be correctly classified as a bloom in other images given a different perspective of the flower; therefore, this flower could be included in the final result. However, when a non-bloom image was classified as a bloom, this false bloom was counted and there was no approach to remove it in the current method. The training samples for the CNN were selected from images collected on only 4 different days, so the trained CNN may not be suitable for data collected in different growth stages, especially when the color of the cotton leaves changes over the season. Therefore, including more training samples from different dates over the growth season could improve the accuracy and robustness of the CNN. The different appearances of the cotton flower caused by genotype differences—for instance, the Pima cotton has yellow bloom in contrast with white bloom for some upland cotton varieties—should also be considered when constructing the training samples.

The accuracy of the bloom registration was affected by the 3D position of the bloom and the registration algorithm. The 3D position of the bloom was affected by the pixel location and the dense point cloud. Therefore, the accuracy of the dense point cloud has an important impact on the accuracy of a bloom's 3D

position, and thus affects the bloom registration results. A larger error of the 3D position can cause the bloom registration to under-cluster or over-cluster the blooms. Under-clustering can count the same bloom more than once and over-clustering may count several blooms as one bloom, making the bloom count unreliable. The completeness of the point cloud also affects the registration result. If the dense point cloud cannot cover the whole canopy, some blooms that are detected in the images may not have a valid projection on the dense point cloud. Those blooms will not be registered and underestimation of the bloom count will occur. Therefore, adequate image overlap is critical in data collection to capture the whole canopy. Increasing the image overlap can improve the completeness, but also requires more data collection and processing time. Oblique imagery can also improve the completeness of the point cloud by providing more views of a plot and potentially can image more occluded flowers.

Although the proposed bloom counting methodology usually provides an underestimated bloom count compared to manual counting, it has the advantage in throughput over manual counting. It saves manual labor and makes continuously monitoring the flowering possible. Without such throughput, it is impossible to continuously monitor the flowering stage and determine the flowering time, peak flowering time, and seasonal bloom count. For farmers and breeders, it is helpful to estimate the fiber yield because some studies have shown the fiber yield is correlated with seasonal bloom count (Heitholt, 1993, 1995). However, additional studies on how well the proposed methodology can estimate fiber yield are needed. The methodology is also helpful for differentiating the growth behavior among different genotypes, which can be used to select certain genotypes, such as short-season or long-season genotypes. Compared to other flower detection methods that are

based on the percentage of flower pixels, the method proposed in this study can directly provide flower count without exploring the correlation between pixel percentage and flower count (Adamsen et al., 2000). The proposed method also can provide the locations of flowers as byproducts, which could be used to correlate with the cotton bolls (**Figure 14**).

To implement the proposed method, farmers and breeders can use commercial aerial photogrammetry systems or build custom systems to collect aerial images. The data processing pipeline can be used as long as the image quality (such as the ground resolution) meets the requirements. This study used small fields of 920 m<sup>2</sup> (0.22 acre) and the data collection throughput is not enough for commercial farms or breeding programs with large fields. Although the throughput can be increased by increasing the flight altitude, the reduced ground resolution may not meet the requirement for the pipeline to correctly recognize cotton flowers. Alternative solution is to use high-resolution cameras to maintain the ground resolution when imaging at higher altitudes.

#### CONCLUSION

This study developed a high throughput methodology for cotton bloom detection using aerial images, which can be potentially

blooms and right images show the corresponding blooms in the raw images.

used to monitor cotton flowering over the season for cotton production management and yield estimation. The method generally underestimated the bloom count due to the inability to count hidden flowers, but the bloom count for the single plant layout was less likely to be underestimated than the multiple plant layout. The accuracy and completeness of the dense point

cloud has an impact on the bloom count result, so generating a good dense point cloud can improve the results significantly. The bloom registration algorithm developed in this study was efficient in terms of runtime and was more prone to underclustering but less prone to over-clustering. The trained CNN correctly classified more than 97% of the training and test images, and more than 90% of the potential flowers extracted from individual datasets. Since the false classification from the CNN can result in false bloom count, designing a robust CNN that can handle images taken under different field illumination conditions and cotton growth stages will be included in future studies. In addition, oblique imagery will be explored to improve the quality of the dense point cloud.

# AUTHOR CONTRIBUTIONS

RX: performed the data analysis and wrote the main manuscript text; CL and AP: designed the experiments and wrote the manuscript; YJ: designed plot layout and contributed to the data collection; SS and JR: contributed to the data collection and field management; All of the authors have read and approved the final manuscript.

#### FUNDING

This study was funded jointly by the Agricultural Sensing and Robotics Initiative of the College of Engineering, and the College of Agricultural and Environmental Sciences of the University of Georgia. The project was also partially supported by the National Robotics Initiative grant (NIFA grant No. 2017-67021-25928) and Georgia Cotton Commission.

#### ACKNOWLEDGMENTS

The authors gratefully thank Mr. Jeevan Adhikari, Dr. Tariq Shehzad, Mr. Zikai Wei, Mr. Jamal Hunter

#### REFERENCES


and Mr. Shuxiang Fan for their great assistance in planting, data collection and harvesting. The authors also want to thank Mr. Joshua Griffin for his great assistance in field management. The authors thank Dr. Peng Wah Chee for selecting cotton lines and providing cottonseeds.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpls.2017. 02235/full#supplementary-material


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Xu, Li, Paterson, Jiang, Sun and Robertson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Fast High Resolution Volume Carving for 3D Plant Shoot Reconstruction

Hanno Scharr\*, Christoph Briese, Patrick Embgenbroich, Andreas Fischbach, Fabio Fiorani and Mark Müller-Linow

Institute of Bio- and Geosciences, IBG-2: Plant Sciences, Forschungszentrum Jülich GmbH, Jülich, Germany

Volume carving is a well established method for visual hull reconstruction and has been successfully applied in plant phenotyping, especially for 3d reconstruction of small plants and seeds. When imaging larger plants at still relatively high spatial resolution (≤1 mm), well known implementations become slow or have prohibitively large memory needs. Here we present and evaluate a computationally efficient algorithm for volume carving, allowing e.g., 3D reconstruction of plant shoots. It combines a well-known multi-grid representation called "Octree" with an efficient image region integration scheme called "Integral image." Speedup with respect to less efficient octree implementations is about 2 orders of magnitude, due to the introduced refinement strategy "Mark and refine." Speedup is about a factor 1.6 compared to a highly optimized GPU implementation using equidistant voxel grids, even without using any parallelization. We demonstrate the application of this method for trait derivation of banana and maize plants.

#### Edited by:

Alexander Bucksch, University of Georgia, United States

#### Reviewed by:

Suxing Liu, University of Georgia, United States Dimitrios Fanourakis, Institute of Olive Tree, Subtropical Plants and Viticulture, Hellenic Agricultural Organization "Demeter" (NAGREF), Greece

#### \*Correspondence:

Hanno Scharr h.scharr@fz-juelich.de

#### Specialty section:

This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science

> Received: 18 August 2017 Accepted: 12 September 2017 Published: 28 September 2017

#### Citation:

Scharr H, Briese C, Embgenbroich P, Fischbach A, Fiorani F and Müller-Linow M (2017) Fast High Resolution Volume Carving for 3D Plant Shoot Reconstruction. Front. Plant Sci. 8:1680. doi: 10.3389/fpls.2017.01680 Keywords: image processing, 3D from silhouettes, visual hull, octree, integral image, refinement strategy, performance analysis

### 1. INTRODUCTION

Complementary to genomics, the quantitative description of plant phenotypes is at the core of basic research for the analysis of plant development and physiological responses to abiotic and biotic challenges as well as for applications in plant genetic improvement and precision agriculture. An increasing amount of phenotypic data are generated using digital images and time series experiments using a variety of methods and sensors both in controlled environment and in the field (reviewed e.g., in Furbank and Tester, 2011; Fiorani and Schurr, 2013; Mulla, 2013; Araus and Cairns, 2014). Most of these methodologies, ranging from RGB to spectral imaging, are based in high-throughput phenotyping pipelines primarily on 2D spatial analyses for the estimation from image analysis of plant traits such as total leaf area, crop coverage and leaf biomass (Homolova et al., 2013; Chen et al., 2014), leaf color (Hu et al., 2013), plant height measurements using light courtains (Fanourakis et al., 2014) or root morphological features (Das et al., 2015). The use of optical imaging with RGB cameras for estimation of shoot area and above-ground biomass is an established method deployed in indoor phenotyping platforms imaging stations for large-scale studies (Al-Tamimi et al., 2016). These methodologies are based on estimation of leaf area and leaf biomass by acquisition and image analysis of a series of RGB projections from multiple view-angles (Golzarian et al., 2011). Issues that may arise from this approach that generally use a limited number of view-angles are related to the likely underestimation of total leaf area for relatively complex shoot architecture and for large plants at advanced developmental stages. In many cases and for the same

**63**

reasons, quantifying the growth rates and leaf angles of individual leaves from global shoot images remains challenging. Optical 3D reconstruction of plant shoots using a variety of methods assists in alleviating these issues. For example, studies were conducted in Arabidopsis using light-field cameras for depth reconstruction of rosettes (Apelt et al., 2015), in tobacco using high-resolution 3D imaging and a mesh approach (Paproki et al., 2012), and also in the field using stereo imaging with consumer cameras (Müller-Linow et al., 2015). Progress in this field is still limited by the required computational power and time investment for image analysis (Minervini et al., 2015). In this respect, improvements are required both for methods using 3D reconstruction from silhouettes and 3D imaging.

#### 1.1. Related Work

Measuring plant geometry from single view-point 2D images often suffers from insufficient information, especially when plant organs occlude each other (self-occlusion). In order to achieve more detailed information and recover the plants 3D geometric structure volume carving is a well established method to generate 3D point clouds of plant shoots (Koenderink et al., 2009; Golbach et al., 2015; Klodt and Cremers, 2015), seeds (Roussel et al., 2015, 2016; Jahnke et al., 2016), and roots (Clark et al., 2011; Zheng et al., 2011; Topp et al., 2013). Volume carving can be applied in high-throughput scenarios (Golbach et al., 2015): For the reconstruction of relatively simple plant structures like tomato seedlings image reconstruction takes ∼25–60 ms, based on a well though out camera geometry using 10 cameras and a suitably low voxel resolution 240 × 240 × 300 voxels at 0.25 mm voxel width. Short reconstruction times are achieved by precomputing voxel to pixel projections for each of the fully calibrated cameras. However, precomputing lookup-tables is not feasible for high voxel resolutions due to storage restrictions (Ladikos et al., 2008). Current implementations popular in plant sciences suffer from high computational complexity, when voxel resolutions are high. We therefore implemented and tested a fast and reliable volume carving algorithm based on octrees (cmp. Klodt and Cremers, 2015) and integral images (cmp. Veksler, 2003), and investigate different refinement strategies. This work summarizes and extends our findings presented in Embgenbroich (2015).

Visual hull reconstruction via volume carving is a well-known shape-from-silhouette technique (Martin and Aggarwal, 1983; Potmesil, 1987; Laurentini, 1994) and found many applications. Also octree as multigrid approach and integral image for reliable and fast foreground testing have been used successfully with volume carving in medical applications (Ladikos et al., 2008) and human pose reconstruction (Kanaujia et al., 2013). Realtime applications at 512<sup>3</sup> voxel resolution have been achieved where suitable caching strategies on GPUs can be applied e.g., for video conferencing (Waizenegger et al., 2009). Here we demonstrate that even higher spatial resolutions are achievable on consumer computer hardware without prohibitively large computational cost. Subsequent octree-voxel-based processing allows extraction of plant structural features suitable for plant phenotypic trait extraction.

# 2. RECONSTRUCTING SHAPES FROM SILHOUETTES

# 2.1. Voxelbased Volume Carving

Here, we revisit voxel-based volume carving, closely following the description found in Roussel et al. (2016). In the subsequent sections we will then extend this formulation using octrees as multigrid extension.

Consider an imaging setup with a set of fully calibrated cameras, and a turn-table allowing to rotate our plant of interest around its vertical axis. Golbach et al. (2015) use 10 cameras and no turn-table, Roussel et al. (2016) a single camera and a robot instead of a turn-table; here we use different setups combining up to three cameras and a turntable (see section 3.1).

For each camera c we obtain the 3 × 3 intrinsic camera matrix **K**c , 3×3 rotation matrix R<sup>c</sup> and translation vector Et<sup>c</sup> with respect to the reference camera (cmp. Hartley and Zisserman, 2004), and the distance between the origin of our working volume and the reference camera center from calibration (cmp. section 3.2). The origin of the working volume is selected to be the rotation center of the turn-table (cmp. section 3.1).

We acquire N images, showing the plant of interest under (equidistantly spaced) rotation angles α<sup>i</sup> where i ∈ {1, . . . , N}. We segment each image into a binary mask **M**<sup>i</sup> being one at the foreground, i.e., plant, and zero at background locations. For segmentation, we either use HSV color thresholding (Walter et al., 2007) or a support vector machine (SVM) based learning algorithm (cmp. e.g., Hearst et al., 1998; Wang et al., 2011; Li et al., 2013). Both methods are parameterized offline, where SVM parameters are learned using suitable training data. HSV parameters are hand-tuned by setting 6 threshold values on the 3 color channels. Computational effort as a preprocessing step for carving is negligible, as it typically takes fractions of a second per image for both methods. Subsequently, small objects like noise are removed and small holes filled. Suitable filling sizes depend on the imaging setup and need to be determined empirically.

For each image and thus segmentation mask we calculate the homogeneous camera projection matrix **P**<sup>i</sup> , from the rotation angle α<sup>i</sup> by

$$\mathbf{P}\_{i} = \mathbf{K}\_{c}(\mathbf{R}\_{c}|\vec{t}\_{c} + \vec{t}\_{0}) \begin{pmatrix} \mathbf{R}\_{i} & \mathbf{0} \\ \mathbf{0} & 1 \end{pmatrix} \tag{1}$$

where **R**<sup>i</sup> is the rotation matrix corresponding to the given angle αi , and translation vector Et<sup>0</sup> is calculated using the distance of the world origin to the reference camera center, also known from calibration (see e.g., Hartley and Zisserman, 2004). By this, the world coordinate frame rotates with the object, i.e., the plant.

We define an equidistantly spaced, cubic voxel grid around the world origin, being large enough to contain the plant. Such a working volume depends on the plant size in order to keep voxel number and thus complexity low. We can relax this requirement later, when using octrees.

Each voxel center with homogeneous world coordinates XE is projected to a point xE<sup>i</sup> in each mask **M**<sup>i</sup> by

$$
\vec{\boldsymbol{x}}\_{l} = \mathbf{P}\_{l} \vec{\boldsymbol{X}} \tag{2}
$$

If XE is projected to the background region of at least one of the N masks **M**<sup>i</sup> , then this voxel does not belong to the foreground object and its value **V**(XE ) is set to 0, i.e.,

$$\mathbf{V}(\vec{X}) = \prod\_{i=1}^{N} \mathbf{M}\_i(\vec{x}\_i) \tag{3}$$

Thus, if a voxel belongs to the foreground object, its value **V**(XE ) is set to 1.

When high voxel resolution is desired, and thus runtimes increase, parallelization of the carving algorithm (Ladikos et al., 2008; Brenscheidt, 2014) is feasible. However, as complexity increases linearly with the voxel number and thus cubically with voxel resolution, equidistant voxel discretization quickly reaches its limits for any computer hardware. High resolutions become available on current desktop computer hardware, when hierarchically representing the voxel grid, e.g., as an octree as described next (Szeliski, 1993; Ladikos et al., 2008; Klodt and Cremers, 2015).

#### 2.2. Octrees

An Octree (Meagher, 1980, 1995; Szeliski, 1993) is a hierarchical tree data-structure. Each node corresponds to a cube in 3d space (i.e., a large or small "voxel") where each node not being an endnode or "leaf " is subdivided in eight child nodes. The child nodes have each half the size, i.e., edge length, of their parent node and are non-overlapping such that they fill the same volume as their parent. (cmp. **Figure 1**). Octrees are designed to efficiently store voxel grids, where large, unstructured volumes can be stored using only few nodes (i.e., large voxels), whereas small structures or surfaces in space can be finely resolved using nodes further up in the tree (i.e., small voxels), i.e., higher "level" nodes. To this end, when building up a representation of an object, an octree node is kept as an end-node, when the corresponding 3d volume is either empty, or completely filled by the object. When the corresponding volume is only partly filled by the object, the node is split into its eight children. This is iterated until the finest resolution is reached. Then leaf nodes at finest resolution can be considered as partly filled and represent surface voxels of the object.

Consequently, memory investment for storing a working volume only marginally depends on its size, but rather on the size and complexity of the object of interest. For example storing an equidistantly sampled working volume of 1 m<sup>3</sup> using voxels of 0.25 mm edge length requires 6.4 · 10<sup>10</sup> voxels or ∼64 GB when representing each voxel by a single byte without any compression. Storing the same working volume takes only a few bytes, when it is empty, and when occupied by an object, still much less than storing an equidistantly sampled version of the occupied volume only (see section 4.1 for an example). We can therefore select our initial working volume such that it well contains the overall visible volume of our setup, without compromising on runtime.

#### 2.3. Integral Image

The occupancy test in Equation (3) assumes, that a each voxel XE corresponds to a single pixel or point xE<sup>i</sup> in a mask image **M**<sup>i</sup> . This can be sufficiently well fulfilled when image resolution is low, such that pixel size at working distance is well above voxel size. For large voxels as in the lower levels of an octree, this assumption is not sufficiently fulfilled. We need to adapt the occupancy test, such that we can decide if a voxel is empty, completely or only partly filled. Consider a large voxel being projected to a mask image (see **Figure 2**, left). In general, the 2d central projection of a cube is a hexagon and we needed to test every pixel within the region of this hexagon. When foreground is 1 and background 0, this corresponds to integrating the mask image over the region of the hexagon and compare the result Rhex with the area of the hexagon Ahex. Then only when Rhex/Ahex is exactly 1 or 0, the octree node would not be split. However, integrating over a hexagon for every voxel is time consuming. Instead of this, we integrate over the bounding-box of the hexagon, yielding Rbbx, as this can be done very efficiently using the integral image approach. A bounding-box is a rectangle with sides parallel to the coordinate axes, being spanned by xEmin as the upper left corner and xEmax as the lower right corner<sup>1</sup> . Here xEmin = (xmin, ymin) contains the minimum coordinate values<sup>2</sup> of all corner points of the voxel projected to the mask image, and xEmax their maximum<sup>3</sup> , accordingly. In case the bounding-box is completely filled by the object, i.e., Rbbx = Abbx or completely empty, Rbbx = 0, it will be the same for the hexagon.

$$A\_{\rm bbx} = (\chi\_{\rm max} - \chi\_{\rm min}) \cdot (\jmath\_{\rm max} - \jmath\_{\rm min}) \tag{4}$$

is the area covered by the bounding box. For a partly filled bounding-box we will split the corresponding octree node, in order to be on the safe side, even though there may be cases when the bounding box is partly filled, but the hexagon actually is not.

The integral over the bounding box can be evaluated very efficiently using the summed area table (Crow, 1984) or Integral Image **I**<sup>i</sup> of the mask **M**<sup>i</sup> (see **Figure 2**, right, and Veksler, 2003; Viola and Jones, 2004). Every pixel of **I**<sup>i</sup> contains the integral of **M**<sup>i</sup> in the rectangle being spanned by xEmin = 0 and E xEmax being the current pixel.

$$\mathbf{I}\_{i}(\mathbf{x}, \boldsymbol{\mathcal{y}}) = \sum\_{\mathbf{x'} \le \mathbf{x}, \mathbf{y'} \le \mathbf{y}} \mathbf{M}\_{i}(\mathbf{x'}, \mathbf{y'}) \tag{5}$$

**I**<sup>i</sup> can be derived efficiently by a recursive convolution scheme starting at the upper left corner and proceeding row- or columnwise, where we calculate the current pixel value from the value of its upper and left neighbors, i.e.,

$$\mathbf{I}\_{i}(\mathbf{x},\boldsymbol{\uprho}) = \mathbf{M}\_{i}(\mathbf{x},\boldsymbol{\uprho}) + \mathbf{I}\_{i}(\mathbf{x}-1,\boldsymbol{\uprho}) + \mathbf{I}\_{i}(\mathbf{x},\boldsymbol{\uprho}-1) - \mathbf{I}\_{i}(\mathbf{x}-1,\boldsymbol{\uprho}-1) \tag{6}$$

with considering **I** = 0 outside the mask image. From the integral image the sum over any rectangular region can be calculated using four values

$$\begin{aligned} R\_{\text{bbx},i} &= \mathbf{I}\_i(\mathbf{x}\_{\text{min}}, \mathbf{y}\_{\text{min}}) - \mathbf{I}\_i(\mathbf{x}\_{\text{min}}, \mathbf{y}\_{\text{max}}) - \mathbf{I}\_i(\mathbf{x}\_{\text{max}}, \mathbf{y}\_{\text{min}}) \\ &+ \mathbf{I}\_i(\mathbf{x}\_{\text{max}}, \mathbf{y}\_{\text{max}}) \end{aligned} \tag{7}$$

<sup>1</sup>The image coordinate origin is assumed to be in the upper left corner and x- and y-axes pointing to the right and downwards, as usual for images.

<sup>2</sup>Rounded to lower values, using the C++ operation floor.

<sup>3</sup>Rounded to higher values, using the C++ operation ceil.

For border handling, bounding-box corners outside the current field of view, are moved to the nearest pixel in **I**<sup>i</sup> .

# 2.4. Volume Carving with Octrees

When using octrees instead of equidistantly spaced, cubic voxel grids, the occupancy test from Equation (3) needs to be exchanged. To do so, we first calculate for every mask image **M**<sup>i</sup> the corresponding integral image **I**<sup>i</sup> using the scheme from Equation (6).

We start with an octree containing only a single coarse voxel covering the whole working space, i.e., the trunk node of the octree. This node is initially marked as "object," i.e., it is assumed it contains an object. Then, for all images, all nodes are iteratively updated, when a node N is marked as "object" and

$$R\_{\text{bbx},i}/A\_{\text{bbx},i} = \begin{cases} 1 & \text{then } N \text{ is marked as } \text{"object"}\\ 0 & \text{then } N \text{ is marked as } \text{"empty"}\\ \text{else } N \text{ is marked as } \text{"reference"} \end{cases} \tag{8}$$

Voxels marked as "refine" are split, marked as "empty" and their Children are marked as "object." This update scheme is iterated until the desired finest resolution is reached. In the following, "one iteration" means projecting all current octree voxels on one image once. We have several options, how to organize the iterations:


We investigate memory usage and runtimes for these different refinement strategies in our experiments (see section 4.1).

For our experiments with the banana plants we used a finest resolution of 0.244 mm, being a bit smaller than the pixel size at working distance of ∼0.3–0.4 mm depending on the camera distance to the voxel of interest. The volume of the visual hull of a plant is estimated by summing over the volumes of all nodes marked as "object" where nodes at the finest resolution, i.e., with unknown filling percentage, are accounted for half their volume in order to get an unbiased estimate. Clearly, as the visual hull includes convex volumes or other occluded volumes, its volume only coarsely reflects the volume of a complex object such as a plant.

For visualization (cmp. e.g., **Figures 7**, **8**) we assigned to each "object" voxel a single RGB color value for simplicity. To this end, we calculated for each voxel its nearest camera<sup>4</sup> and selected the color of the center pixel of the voxel's bounding box. No visibility or plausibility tests have been performed. Please note, that this approach is good for visualization purposes, only, and is not well suitable for color-based trait evaluation.

#### 3. MATERIALS AND METHODS

#### 3.1. Imaging

Two setups for image acquisition were used for this study, a semiautomatic setup combining a turntable and up to two cameras, as well as a fully automated system including a turntable and three industrial cameras.

The semi-automatic setup (see **Figure 3**) needs manual initial positioning of a plant, which is then imaged fully automatic. This setup includes an automated turntable (Steinmeyer Mechatronic, Dresden Germany, DT180-SM, 0.1◦ accuracy) with color markers for image-based rotation angle estimation, and two industrial cameras (Allied Vision Technologies, Stadtroda Germany, AVT MANTA G-504, 5MP color camera). Images can be taken either in a stop-and-go fashion, where the plant is turned to the next pose, stopped, and then images are acquired, or alternatively, the turntable can rotate and images are taken in equidistant time intervals corresponding to equidistant rotation angles.

The fully automated setup called Screenhouse at IBG-2 is used for screening of the shoot structure and function of different mono- and dicotyledonous plant species in a greenhouse environment (cmp. Nakhforoosh et al., 2016). Plants are automatically fed to the system being equipped with an imaging station for automated imaging (see **Figure 4**, top and left). Imaging is routinely performed with three cameras (Point Gray Grasshopper2, 5MP color camera, by FLIR Integrated Imaging Solutions Inc., Richmond, British Columbia, Canada) which are located at different positions to efficiently assess the plant structural properties for diverse shoot architectures. A turntable is used for stop-and-go imaging at 4 positions at 0, 90, 180, and 270◦ . For the top camera only the image acquired at 0◦ is used. This means that 9 images per plant are used for reconstruction in our experiments.

#### 3.2. Camera Calibration

One of the main drawbacks of visual hull computation with fixed calibrated cameras is its sensitivity to imprecise external camera calibration and lens aberrations. When a mask **M**<sup>i</sup> is misaligned and thus does not well overlap with the "true" object volume, the non-overlapping parts are deleted from the volume without further testing or corrections. Geometric precision of the calibration needs to be in the same range as the desired precision in object space, i.e., when reconstructing relatively large objects sufficient precision is reachable using well established methods. When dealing with small objects like plant seeds, an online image-based camera pose calibration step may be needed (see e.g., Roussel et al., 2016). In our case, plants are in the size range of tens of centimeters and precision needed for calibration

<sup>4</sup>We used as camera position the camera center, as usual.

FIGURE 4 | Imaging. (Top) Screenhouse setup. (Bottom left) Camera setup in Screenhouse measurement chamber. Three 5 MP RGB cameras (red circles) with different view angles and rotating table (green circle). (Bottom right, top) Original RGB images taken from 3 different view angles; (Middle) Binary masked images; (Bottom) Intermediate carving step overlaid on images.

is in the range of ≈0.3 mm, which is approximately the lower limit of pixel size at working distance. We use the calibration approach introduced by Bouguet (1999), where multiple, here 20, images of arbitrarily posed flat calibration targets are used. Laserprinted calibration targets glued to flat material are sufficient for our requirements. We use the OpenCV ver. 2.4.9 (Bradski, 2000) implementation calibrateCamera of the method presented in Zhang (2000).

#### 3.3. Software Implementation

The software framework was implemented in C++ on a Windows 7 operating system with Microsoft Visual Studio 2013.

#### 3.4. Datasets Acquired

#### 3.4.1. Maize Plants on the Semi-automatic Setup

We imaged 5 maize plants (Zea Maize, Genotype "Badischer Gelber") at 6 different time points over 17 days from seedling stage with 2–8 leaves stage (see **Figure 5** for an example). Using both cameras of the semi-automatic setup this resulted in 30 datasets of 60 images each, where we had to remove 7 sets due to imaging problems, e.g., plants being already too large for our setup.

#### 3.4.2. Banana Seedlings on the Automatic Setup

Seven groups of five banana seedlings, cultivar Khai Thong Ruang, were imaged and harvested for fresh weight determination immediately after imaging. Imaging and harvesting took place 6, 10, 14, 17, 20, 28, and 34 days after transplantation of the initial four leaf plantlets. This resulted in 35 datasets of 9 images each (see **Figure 6**). In addition some larger banana seedlings have been imaged in the same way, but not been harvested (see **Figure 7**).

# 4. EXPERIMENTS

## 4.1. Performance of Different Refinement Strategies

We investigate the runtimes needed for the four different refinement strategies Depth first, Breadth first, Refining resolution, and Mark and refine described in Section 2.4. We performed these test on a computer with 48GB RAM and two Intel Xeon E5540 @ 2.53 GHz as some refinement strategies use a lot of memory. The Mark and refine strategy can also be run on a current laptop (8GB RAM, Intel Core i5-5300U @ 2.30 Ghz) at comparable speeds. All runtime tests were done without any parallelization using a single CPU only, unless stated differently.

In **Figure 7A** an image from the dataset used for this investigation is shown, one of the larger banana plants being 31 cm high and 29 cm wide. **Figure 7B** shows the corresponding carving result. Its final 3d grid contains Noct = 5.4 · 10<sup>6</sup> foreground octree voxels from 0.244 to 15.6 mm. We observe that finest structures, like the tip of the top leaf or the tip of the large leaf on the left are lost. Missing of finest structures in volume carving typically, and also here, has two main reasons: segmentation errors and calibration inaccuracies. The main effect in the data set used here is lens imperfections being not fully compensated by radial distortion correction. In addition, the brownish part of the tip of the top leaf is missing in one segmentation mask and therefore carved away. For the purpose of this publication, i.e., performance evaluation in terms of runtime and resolution, such inaccuracies are of no interest. However for practical applications needing such detail, one may want to invest in better calibration procedures and more precise segmentation algorithms.

A uniform 0.244 mm grid requires 6.9·10<sup>10</sup> voxels to represent the same 1 m<sup>3</sup> working volume (corresponding to 64 GB, when using 1 byte per voxel), where Nequi = 3.3 · 10<sup>7</sup> voxels are foreground i.e., plant. We see that while using an equidistant grid during carving requires a large amount of memory, this is not necessarily the case for the final result. In our case the difference is a factor of Nequi/Noct = 6.1, allowing to perform subsequent processing using equidistantly sampled foreground voxels, if needed.

Complexity for deriving this carving result heavily depends on the refinement strategy used. **Figure 7C** shows the number of octree voxels generated when iterating through the images of the dataset. As before, "one iteration" means projecting all current octree voxels on one image once.

For the classic equidistantly sampled voxel grid approach, we would need 9 iterations as we have 9 images in the banana dataset. We derived the foreground voxel number plotted in **Figure 7C** from the Depth first results using octrees.

For the octree-based method, in order to reach the finest resolution of 0.244 mm, the initial 1 m<sup>3</sup> voxel needs to be refined

FIGURE 6 | Images of the Banana experiment. The images show one banana seedling of each of the seven groups. Imaging was done using the fully automatic Screenhouse facility (see section 3.1).

12 times. As some final surface voxels may only be seen from a single image, we have to project the voxels to each image 12 times to make sure to reach the final resolution; and one more time to remove the unneeded (background) voxels at final resolution. I.e., 117 iterations are needed overall.

The first recommendation for speed-up, or to be able to finish at a usable result at all, is that the final octree resolution needs to be restricted to the desired resolution. If this restriction is relaxed, many octree voxels are refined unnecessarily. This can be seen in **Figure 7C**, case "breadth first, max. depth 20," where the number of voxels keeps increasing exponentially after 18 iterations, in contrast to the case "breadth first," where we restricted resolution. The unrestricted calculation leaded to memory overflow and ended before reaching the final result.

The second recommendation for speed-up is to preallocate memory wherever possible. The effect of allocation on the fly<sup>5</sup> is demonstrated in **Figure 7D**, cases "depth first" and "depth first,

<sup>5</sup> Such allocation is as offered e.g., in C++ using the vector-class from the standard template library.

no preallocation" where a factor of 1.3 longer final runtimes can be observed.

Generally the Depth first strategy shows by far longest runtimes. The reason is that in the first iterations the number of voxels generated increases exponentially, in our case to 75 M voxels after 12 iterations. This is when the finest resolution is reached for the first image. Then, with every new image voxel number drops with the first iteration, as many voxel s are carved away, and then again increases with the next iterations. Due to the large number of voxels to process, runtime is very high (2,997 s).

The Breadth first strategy is about a factor 2.6 faster needing 1,155 s to finish. Restricting the reachable resolution with every loop through all images, i.e., using the Refining resolution strategy again yields a speed-up of a factor 4.0, i.e., a factor of 10.4 compared to Depth first.

Mark and refine is the fastest refinement strategy, as it produces fewest voxels during processing. It reaches a resolution of 0.98 mm after 7.7 s ( ˆ=1, 024<sup>3</sup> voxel grid), 0.49 mm after 32.7 s ( ˆ=2, 048<sup>3</sup> voxel grid), and the final 0.244 mm after 148 s ( ˆ=4, 096<sup>3</sup> voxel grid). Using Mark and refine on the laptop described above the final result is reached after 85.7 s, i.e., a factor 1.7 faster. In Roussel et al. (2016) 218 s are reported for the usual equidistant grid method using a 1, 024<sup>3</sup> voxel grid and a CPU implementation, 12.56 s using a well optimized GPU implementation. We even outperform this GPU optimized version by a factor 1.6, using no parallelization at all.

We also tested a simple parallelization method for Mark and refine, by distributing the voxel-to-image projections over 8 CPU threads, but not threading the splitting step. The final result at 0.244 mm resolution is then reached after 37.7 s. Compared to the slowest (but still completing) method we observe an overall speedup of a factor 102.

### 4.2. Runtime for Different Plant Sizes

The overall runtime of octree-based volume carving strongly depends on the plant size and complexity, more precisely, the surface area of its visual hull, as the finest voxels occur there. Consequently, we observe that the number of pixels roughly increases by a factor of 4 after each pass through all images in Mark and refine (cmp. **Figure 7C**). This is consistent with the assumption that on average 4 of 8 of the new voxels are foreground, when the old voxel was a the surface and thus needed splitting.

Overall runtimes for the reconstruction of the 35 plants of the Banana datasets are shown in **Figure 8B**. We observe, that runtimes are faster for smaller plants, approximately linearly increasing with the estimated volume or, for banana plants, with leaf area. This is the case as the structure of banana plants is dominated by big leaves, i.e., flat objects with area being well correlated with volume or, equivalently, fresh weight (see **Figure 8C**).

### 4.3. Examples for Visual-Hull Based Plant Traits

We evaluated the general usability of this visual hull reconstruction approach with two scenarios using banana and maize plants (see section 3.4).

In **Figure 8A** reconstructions of the banana plants are visualized and in **Figure 8D** their fresh weight is plotted vs. the estimated volume. We observe that the estimated volume of the visual hull correlates with the fresh weight (R <sup>2</sup> = 0.8511), however differences in estimated volume for plants with

approximately the same weight are up to about a factor of 2. This makes the estimated volume useful as non-invasive surrogate measure for fresh weight in statistical analyses with replication, but not very reliable when comparing individual plants. This is not unexpected, as the visual hull may contain non-negligible non-plant volumes due to occlusions. Such non-plant volumes largely depend on the actual configuration of leaves, i.e., leaf poses.

The visual hull can be used more reliably to derive structural plant properties, like leaf numbers or leaf areas, when applying further processing. In **Figure 9** a leaf segmentation result is depicted for a maize plant. Here a reliable leaf-wise segmentation has been reached by a sequential cluster connecting algorithm and subsequent refinement steps. A detailed description of the algorithm is beyond the scope of this publication. In short, leaf segmentation is done by searching clusters in horizontal one final voxel thick slices, and connecting the found clusters over slices scanning from top to bottom. By this, cluster labels are transferred from top to bottom, forming plant parts. When two clusters merge in one slice, the smaller label is transferred to this slice cluster. By this strategy leaves are segmented, where the stem region still belongs to the top-most leaf. They are separated in a subsequent step using simple but heuristic geometry rules. This leaf segmentation allowed e.g., to count leaves reliably. For 19 of the 23 maize plant datasets leaf numbers were correctly estimated. For the other 4 plants the smallest top leaf was missed, comparable to the small bright green leaf in **Figure 9C**. For further details we refer to Embgenbroich (2015).

When individual leaves are segmented as point clouds, leaf properties can be derived. The length of a leaf may be determined as the shortest path through the point cloud connecting leaf base and tip. To do so, usual skeletonization algorithms may be used. In order to derive leaf area, the point cloud needs to be flattened. This can be done by piece-wise fitting planes to the leaves as depicted in **Figure 10**. In order to remove steps between fitted plane pieces (**Figure 10B**), a moving least squares method may be applied (**Figure 10C**). For further details we refer to Embgenbroich (2015).

# 5. CONCLUSION AND OUTLOOK

The presented volume carving using octrees and integral images allows fast and accurate visual hull computation on standard PC hardware with relatively low memory and CPU requirements. A key to efficient computations is the choice of a suitable refinement strategy, as unrestricted refinement leads to exponential "explosion" of voxel numbers and thus complexity. Speedup between efficient and non-efficient implementations can be more than 2 orders of magnitude and differences in memory requirements comparably dramatic. Efficient CPU implementations allow fast execution on usual laptop or desktop compute hardware, outperforming even GPU-optimized "brute force" implementations using equidistantly sampled voxel grids. Even faster implementations may be possible, when porting the presented multi-grid approach to GPUs. Usability of volume carving results represented in octree voxel grids is equivalent to using fine equidistant voxel grids, but at much higher reachable resolutions. Some examples how to derive traits of banana and maize plants have been presented to demonstrate

possible applications in plant phenotyping for deriving leaf level traits.

## AUTHOR CONTRIBUTIONS

FF, MM, and HS designed the study. PE, CB, AF, and HS implemented and tested the software. CB and PE performed all lab experiments. FF, MM, and HS drafted the manuscript. All authors contributed to text and figures of the manuscript and approved the final manuscript.

# REFERENCES


#### ACKNOWLEDGMENTS

The banana dataset has been kindly provided by Eric Opoku Mensah, Thomas Bodewein, and Tobias Wojciechowski and has been acquired in the BMBF funded GlobE BiomassWeb project. Parts of this work was performed within the German-Plant-Phenotyping Network which is funded by the German Federal Ministry of Education and Research (project identification number: 031A053).


phenotyping and physiological measurements. Front. Plant Sci. 7:1155. doi: 10.3389/fpls.2016.01155


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer SL and handling Editor declared their shared affiliation.

Copyright © 2017 Scharr, Briese, Embgenbroich, Fischbach, Fiorani and Müller-Linow. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Chlorophyll Fluorescence Imaging Uncovers Photosynthetic Fingerprint of Citrus Huanglongbing

Haiyan Cen<sup>1</sup> , Haiyong Weng<sup>1</sup> , Jieni Yao<sup>1</sup> , Mubin He<sup>1</sup> , Jingwen Lv<sup>2</sup> , Shijia Hua<sup>1</sup> , Hongye Li<sup>2</sup> and Yong He<sup>1</sup> \*

<sup>1</sup> College of Biosystems Engineering and Food Science, Zhejiang University, Hangzhou, China, <sup>2</sup> College of Agriculture and Biotechnology, Zhejiang University, Hangzhou, China

Huanglongbing (HLB) is one of the most destructive diseases of citrus, which has posed a serious threat to the global citrus production. This research was aimed to explore the use of chlorophyll fluorescence imaging combined with feature selection to characterize and detect the HLB disease. Chlorophyll fluorescence images of citrus leaf samples were measured by an in-house chlorophyll fluorescence imaging system. The commonly used chlorophyll fluorescence parameters provided the first screening of HLB disease. To further explore the photosynthetic fingerprint of HLB infected leaves, three feature selection methods combined with the supervised classifiers were employed to identify the unique fluorescence signature of HLB and perform the three-class classification (i.e., healthy, HLB infected, and nutrient deficient leaves). Unlike the commonly used fluorescence parameters, this novel data-driven approach by using the combination of the mean fluorescence parameters and image features gave the best classification performance with the accuracy of 97%, and presented a better interpretation for the spatial heterogeneity of photochemical and non-photochemical components in HLB infected citrus leaves. These results imply the potential of the proposed approach for the citrus HLB disease diagnosis, and also provide a valuable insight for the photosynthetic response to the HLB disease.

Keywords: citrus, Huanglongbing, chlorophyll fluorescence imaging, photosynthesis, feature selection, classification

# INTRODUCTION

Huanglongbing (HLB), also known as greening disease, is one of the most destructive diseases to citrus industry. It is caused by a non-cultured phloem-restricted bacterium Candidatus Liberibacter with three different species, Liberibacter africanus (Laf), Liberibacter asiaticus (Las), and Liberibacter americanus (Lam), and is transmitted by the Asian citrus psyllid Diaphorina citri (Bové, 2006). HLB has threatened citrus production worldwide, including Asia, Africa, the United States, and Brazil. The infected trees are asymptomatic at early stages and become symptomatic anywhere from months to years varied with the cultivars. It could serve as a source of inoculums for pysllids with 60 days from the initial infection until it is infective (Lee et al., 2015), resulting in a rapid spread of infection in a glove of citrus trees. The leaf yellowing with blotchy mottle is considered as the most typical diagnostic symptom of HLB disease, especially on sweet oranges. The severely infected trees have smaller leaves with mottled patterns that are often similar

#### Edited by:

Eetu Puttonen, National Land Survey of Finland, Finland

#### Reviewed by:

Alireza Pourreza, University of California System, United States Luis Gustavo Marcassa, University of São Paulo, Brazil

> \*Correspondence: Yong He yhe@zju.edu.cn

#### Specialty section:

This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science

> Received: 10 May 2017 Accepted: 15 August 2017 Published: 29 August 2017

#### Citation:

Cen H, Weng H, Yao J, He M, Lv J, Hua S, Li H and He Y (2017) Chlorophyll Fluorescence Imaging Uncovers Photosynthetic Fingerprint of Citrus Huanglongbing. Front. Plant Sci. 8:1509. doi: 10.3389/fpls.2017.01509

**75**

to some nutrient deficient symptoms. In addition, the production of the HLB infected trees would dramatically decline with small and lopsided fruits, leading to the enormous economic loss.

So far, no effective control and cure measures are available for the HLB disease. There is an urgent need to develop an efficient method for the rapid and early detection of HLB disease in both field and laboratory conditions, and thereby in need of removal of the infected trees to prevent the further spread. The current commonly used HLB diagnosis involves experiencebased inspection in the orchard and laboratory diagnostic tests. The former is usually performed by professional experts to recognize the typical symptoms of the disease in the field survey. Reported studies have shown that the accuracy of identifying an HLB infected tree by visual inspection is lower than 59% (Pourreza et al., 2015). Polymerase chain reaction (PCR) method that amplifies a DNA target sequence of the pathogen is usually employed as a standard laboratory-based diagnostic method. Although PCR can achieve a relative high accuracy for the HLB detection, it requires a time-consuming and labor-intensive sample preparation, and is not suitable for the continuous onsite detection and early disease warning. Therefore, a fast and non-destructive method is needed for growers to monitor their groves and control the spread of the disease.

Over the years, reflectance spectroscopy and imaging methods based on the selected or broad visible and near-infrared (VIS-NIR) electromagnetic wavelengths were investigated to detect plant diseases on the leaf and/or canopy levels (Inoue et al., 2012; Gnyp et al., 2014; Kusnierek and Korsaeth, 2015), and several studies have been reported on detection of citrus diseases. Qin et al. (2011) used two-band ratio images selected from the hyperspectral images to detect the canker disease of citrus fruit with the overall classification accuracy of 95.7%. Mishra et al. (2012) employed a VIS-NIR spectroscopy technique combined with three different classifiers for identifying HLB infected citrus trees. Although a reasonable classification accuracy was achieved by multiple spectral measurements of the tree canopy, the variation of sunlight and other environmental factors under real field conditions can add additional noises and reduce the detection accuracy. Later, they developed a vision sensor based on the polarization planar rotation of light by the starch at 591 nm, which improved the HLB detection accuracy in both zinc-deficient and non-zinc-deficient classes (Pourreza et al., 2014). However, the applicability of this method largely depends on the starch accumulation that varies in citrus cultivars and growing conditions (i.e., different orchards and seasonal factors). Sankaran et al. (2013) demonstrated the applicability of VIS-NIR and thermal imaging for detection of HLB disease in citrus trees, and they obtained an average overall accuracy of 87% for trees with symptomatic leaves. Li X. et al. (2012) applied multispectral and hyperspectral airborne imaging to detect HLB infected trees in citrus groves. However, the classification accuracies are relative low ranging from 28.7 to 90.2% by using different image classification methods due to the large positioning error of the ground truth. Garcia-Ruiz et al. (2013) compared two aerial imaging platforms for identifying HLB infected citrus trees with the classification accuracies in the range of 67–85% from unmanned aerial vehicle-based data, and 61–74% with aircraftbased data. In general, these results were not better than those from the ground-based remote sensing methods.

Fluorescence spectroscopy is also considered as a promising method for the rapid and early detection of biotic and abiotic stress response in plants. Fluorescence emissions from molecules of certain compounds such as plant pigments in leaves can be captured after the natural and artificial ultraviolet (UV) light excitation on a small sampling point (Pereira et al., 2011; Harbinson, 2013). Sankaran and Ehsani (2013) used a commercial handheld fluorescence sensor to collect fluorescence signal from healthy and HLB infected leaves of different citrus cultivars, and 97% classification accuracy in the discrimination of healthy and symptomatic HLB infected samples was achieved based on the bagged decision tree (BDT) classifier. However, the classification accuracy was reduced to about 81% when using the validation dataset from Hamlin and Valencia samples, and the overall accuracy of differentiating healthy from asymptomatic HLB infected samples was also very low with the best result of 48.2% when using the support vector machine. Due to the limitation of the sampling area, the spatial heterogeneity in a leaf caused by plant diseases or other physiological disorders may not be accurately described. Researchers proposed that the fluorescence imaging technique could be more useful in detecting the plant physiological response since it can obtain the fluorescence signal as well as the within-plant variation (Pinto et al., 2016; Song et al., 2016). Different fluorescence imaging techniques with different excitation modes have been developed. Steady-state UV light-induced fluorescence imaging has been used to estimate anthocyanin in strawberries and visualize systemic viral infections in Nicotiana benthamiana plants by monitoring the signals of the chlorophyll fluorescence and blue-green fluorescence, respectively (Pineda et al., 2008a; Yoshioka et al., 2013). Fluorescence imaging spectroscopy with 530 nm excitation that can obtain a fluorescence image at 690 nm, was also employed to discriminate symptomatic HLB infected leaves from healthy ones sampling from two different orchards (Wetterich et al., 2013). Results showed a good accuracy (90%) for Brazil samples but low (61%) for United States samples. Later, they applied another two excitation sources (405 and 470 nm) to detect citrus HLB disease from nutrient deficient leaves, as well as healthy ones (Wetterich et al., 2016, 2017). The combination of fluorescence bands from two excitations improved the classification accuracy in the range of 92–95% when discriminating HLB from zinc-deficiency by using different machine learning methods. However, more detailed knowledge about energy partition in photosystem II (PSII) should be understood in HLB infected leaves. Kinetic chlorophyll fluorescence imaging provides an efficient way to trace energy partitioning in the photosystem II (PSII) and monitor the electron transport pattern in photosynthesis. It plays an important role in understanding the fundamental mechanism of photosynthesis, pathology, and phenotypic plasticity to genetic variations and plant environmental changes (Murchie and Lawson, 2013; Choi et al., 2016; Sui et al., 2017). Kinetic chlorophyll fluorescence imaging has been widely used to evaluate freeze-thaw and drought tolerance in Arabidopsis (Ehlert

and Hincha, 2008; Bresson et al., 2015), salt stress response in wheat (Mehta et al., 2010), chronic ozone damage to soybean leaves (Chen et al., 2009), and plant virus infection (Pineda et al., 2011; Lei et al., 2017). However, few studies have used kinetic chlorophyll fluorescence imaging to study the HLB disease effects on plant photosynthetic functions and complex pathogenesis. Furthermore, the spatial heterogeneity during the disease development implies additional challenges for the rapid diagnosis of HLB disease.

Therefore, this research was aimed to characterize the photosynthetic function of the HLB infected leaves by measuring the fluorescence signals using kinetic chlorophyll fluorescence imaging, and extract the photosynthetic fingerprints that can uniquely identify the HLB disease by performing the advanced machine learning and statistical analysis. The findings in this research provide an important insight to understand the citrus HLB disease infection related to changes in plant photosynthetic activities, and demonstrate that kinetic chlorophyll fluorescence imaging could offer a rapid and non-invasive means for detecting HLB in citrus trees.

# MATERIALS AND METHODS

## Leaf Sample Collection

Citrus leaf sampling was carried out at a commercial orchard in Nanping, Fujian Province, China in March 2016. The branches were detached from the citrus trees from three categories including healthy, HLB infected (symptomatic and asymptomatic), and nutrient deficient (zinc and magnesium deficient) samples marked by the HLB experts, and immediately wrapped with wet cottons and placed inside a cooler to avoid desiccation. Leaves were then detached from the branches in the laboratory (Supplementary Figure S1), and chlorophyll fluorescence image acquisition was performed immediately after detachment.

# Kinetic Chlorophyll Fluorescence Imaging

Chlorophyll fluorescence images of leaf samples were measured by an in-house chlorophyll fluorescence imaging system (FluorCam FC800, Photon Systems Instruments, Brno, Czechia) (Supplementary Figure S2) after dark adaptation. A CCD camera with a prime lens (SV-H1.4/6, VS Technology, Tokyo, Japan) was used to capture chlorophyll fluorescence transients in a batch of images with the spatial resolution of 696 × 520. Four lightemitting diodes (LEDs) panels with the incident angle of 45◦ were installed as the light source, which include two red-orange LEDs (620 nm) panels (<0.1 µmol·m−<sup>2</sup> ·s −1 ) for flashes and actinic light 1 (0–250 µmol·m−<sup>2</sup> ·s −1 ), and two cool white LEDs (>8000 K) panels (0–1600 µmol·m−<sup>2</sup> ·s −1 ) for actinic light 2 and saturating flashes (0–3000 µmol·m−<sup>2</sup> ·s −1 ). A leaf sample holder with a manually adjustable vertical stage was used for positioning samples to an imaging distance of 20 cm from the lens.

Before the experiment, a preliminary study was performed to determine the optimal protocol for the kinetic chlorophyll fluorescence image acquisition of citrus leaves. The dark-adapted time was determined by checking the value of the maximum PSII quantum yield (Fv/Fm) of citrus leaves at different dark-adapted time as shown in Supplementary Figure S3. It was observed that Fv/Fm was stable with the value of 0.80 after 20 min dark adaptation, which was considered as the optimal time. Although this 20 min period may be a problem for practical applications, it is appropriate for achieving this research goal. In addition, the intensity of the saturating pulse was described in percentage in image acquisition software. 50% was considered as the optimal intensity of the saturating pulse when the value of Fv/Fm reached 0.81 (Supplementary Figure S4), which corresponded with the absolute intensity value of 800 µmol·m−<sup>2</sup> ·s <sup>−</sup><sup>1</sup> when measured at the position of 20 cm distance from the lens using a quantum meter (Model MQ-200, Apogee Instruments, Inc. United States). The detailed chlorophyll fluorescence quenching protocol is described in **Figure 1**. The minimum fluorescence in darkadapted state (Fo) was measured after 20 min dark adaptation, and followed by a strong flash of light of 800 µmol·m−<sup>2</sup> ·s −1 at 5.56 s that transiently reduces the plastoquinone pool and the primary quinone acceptor Q<sup>A</sup> so that the maximum fluorescence in dark-adapted state (Fm) was recorded. The leaf sample was then exposed to an actinic light with the intensity of 100 µmol·m−<sup>2</sup> ·s −1 for 70 s. Saturating flashes (800 µmol·m−<sup>2</sup> ·s −1 ) were applied to measure the maximum fluorescence during light adaptation (Fm\_Ln) at 32.24, 42.24, 52.24, and 72.24 s, respectively, and the maximum fluorescence at the light-adapted steady state (Fm\_Lss) was obtained during the saturating flash (t = 92.24 s) at the end of the actinic light period. The instantaneous maximum fluorescence signals during dark relaxation (Fm\_Dn) with the saturating flashes at 122.24, 152.24, and 182.24 s were also measured. Based on the measured fluorescence signals, the variable fluorescence in dark-adapted state (Fv = Fm − Fo), instantaneous non-photochemical quenching during light adaptation (NPQ\_Ln = (Fm − Fm\_Ln)/Fm\_Ln), steady-state nonphotochemical quenching (NPQ\_Lss = (Fm − Fm\_Lss)/Fm\_Lss), and instantaneous non-photochemical quenching during dark relaxation (NPQ\_Dn = (Fm − Fm\_Dn)/Fm\_Dn) can be also obtained.

# PCR Analysis

Polymerase chain reaction analysis was conducted to confirm the HLB status of leave samples at Plant Pathology Laboratory of Zhejiang University. Two sets of primers designed from conserved regions of 16S ribosomal DNA were synthesized for HLB detection at TSINGKE Biological Technology, China. DNA was extracted from the healthy, nutrient deficient, and HLB infected leaf samples using CTAB (Cetyltrimethylammonium Bromide) methods with the following protocol, respectively: 200 mg midrib tissue of each leaf was transferred to a 2 mL centrifuge tube and ground to the fine powder in liquid nitrogen using an automatic grinding miller (Tissuelyser-64, Shanghai Jingxin Industrial Development Co., Ltd.). 800 µL CTAB was then added into the tube, and heated at 65◦C in water bath for 30 min with intermittent agitation. 800 µL mixture (phenol: chloroform: isopropanol = 25: 24: 1) was added and the tube was centrifuged at 12,000 rmp for 10 min at room temperature.

500 µl supernatant was then transferred into a new 2 mL tube, and 1000 µL 95% alcohol and 150 µL NaAC were added. The homogenate was placed on the ice for 12 min and then centrifuged at 12,000 rmp for 2 min. DNA was re-precipitated with 400 µL of 75% alcohol at 12,000 rmp for 2 min. 100 µL ddH2O was added to the precipitation and DNA was stored at −20◦C at last. PCR amplification was conducted in a 20 µL mixtures reaction using 7.5 µL ddH2O, 1 µL Primer-F, 1 µL Primer-R, 10 µL green taq mix (Vazyme) and 0.5 µL DNA template. The detailed PCR amplification protocol can be found in the literature (Gouda et al., 2006).

# Data Analysis

A total of 26 images for each leaf sample related to the fluorescence quenching process were obtained from kinetic chlorophyll fluorescence imaging, which provide detailed information of dynamics of PSII activities about plant status. Commonly used fluorescence parameters such as Fv/Fo, Fv/Fm, and NPQ\_Lss can be easily calculated by averaging the intensity of the region of interest (ROI) for the corresponding fluorescence image. They are considered as the important parameters in the analysis of photosynthesis associated with plant physiological changes, and their capabilities of identifying the HLB infected leaves from the healthy or nutrient deficient leaves were investigated by advanced machine learning and statistical analysis. Feature selection methods were performed on all the fluorescence parameters to extract the unique fluorescence features that could develop a photosynthetic fingerprint of the HLB disease. Three feature selection methods including random frog (RF), sequential forward selection (SFS), and Monte Carlo uninformative variable elimination (MC-UVE) were used. RF algorithm is based on the reversible jump Markov Chain, and its output provides the selection probability of each feature that can be used as a measure of feature importance. The detailed information of RF can be found in the literature (Li H. et al., 2012). The SFS algorithm is a bottom-up process that starts with an empty feature subset and repeatedly adds the most significant feature selected by an objective function. The feature cannot be discarded at a later stage once it is retained. The fisher criterion is usually used as an objective function (Geng, 2014). MC-UVE develops multiple models with randomly selected calibration sample set produced by the MC method, and the variable is then evaluated with the stability of the corresponding coefficient of the model (Han et al., 2008).

The development of the HLB disease and the spread of the symptoms vary spatially in the leaf, which result in a large variation of fluorescence intensities in the image. Therefore, image features were also investigated for the further classification. Principal component analysis (PCA) was applied to reduce the dimension of the chlorophyll fluorescence image cubes and obtain an uncorrelated orthogonal basis set from the original image set. Scale-invariant feature transform (SIFT) was then used to detect the local features in principal component images. The feature descriptors were first computed as orientation histograms and then transformed to a 128 dimensional SIFT feature vector, which is invariant to the image scale and rotation and robust to local geometric distortions.

Both the mean fluorescence parameter-based and imagebased classification models were developed for detecting the HLB disease, and the inputs of the models were the features extracted from the mean fluorescence parameters and the principal component images, respectively. Two classifiers including partial least squares discriminant analysis (PLS-DA) and support vector machine (SVM) were employed to discriminate the HLB infected leaves from the healthy and the nutrient deficient ones. The PLS-DA algorithm is an extension of the PLS model, where the dependent variable is a vector that represents the class label values for each class (Aliakbarzadeh et al., 2016). The SVM classifier is developed based on the statistical learning theory to find a hyperplane that gives the largest distance between the margins of the training data set, and it can be achieved by solving a convex quadratic programming problem using a kernel function (Chapelle et al., 2002; Cen et al., 2016). The three-class classification was performed by labeling "1" for healthy samples, "2" for HLB infected samples and "3" for nutrient deficient samples. All the samples were divided into two groups with 60 for the training set and 30 for the validation set by using Kennard-Stone algorithm (Macho et al., 2001). The machine learning and the statistical analysis were performed using Matlab 2011a (The Mathworks, Inc., Natick, MA, United States) and IBM SPSS Statistics (version 20.0, IBM Corporation, Armonk, NY, United States).

# RESULTS

# PCR Result

The status of leaves was verified though amplifying target DNA template obtained from leaf midrib tissue. The sensitivity of

PCR analysis with different primer sets is shown in **Figure 2**. Two different sizes of amplicons (1160 and 161 bp) from the 16S ribosomal DNA represented the HLB bacterium DNA template amplified with two specific primers (OI1/OI2c and rplLAS-F/rplLAS-R), respectively. It is indicated that lanes 1 and 4 were PCR positive, and lanes 2, 3, 5, and 6 were PCR negative. Finally, citrus samples were divided into three classes with 30 healthy, 30 HLB infected, and 30 nutrient deficient samples.

# Effect of HLB Infection on Commonly Used Chlorophyll Fluorescence Parameters

**Figure 3** shows a statistical summary (i.e., mean, range, median, outliers, and quartiles) of four commonly used chlorophyll fluorescence parameters, Fo, Fv/Fo, Fv/Fm, and NPQ\_Lss, in healthy, HLB infected, and nutrient deficient leaves. There was no statistically significant difference in Fo among the three classes (**Figure 3A**); while the variation of Fo in the HLB infected leaves was larger than those in healthy and nutrient deficient samples. Compared with healthy leaves, Fv/Fo decreased by 48.9 and 51.1% in HLB infected and nutrient deficient leaves, respectively, indicating the decreased photosynthetic rate in unhealthy leaves. However, no difference in Fv/Fo between HLB infected and nutrient deficient classes was observed (**Figure 3B**). The parameters Fv/Fm and NPQ\_Lss showed significant differences among healthy, HLB infected and nutrient deficient leaves (**Figures 3C,D**). Fv/Fm of healthy samples was higher than those of other two classes, and Fv/Fm of HLB infected samples decreased less than those in nutrient deficient ones (**Figure 3C**). The values of NPQ\_Lss in HLB infected and nutrient deficient leaves increased compared with those in healthy leaves. This result provided the first screening of HLB disease using the commonly used chlorophyll fluorescence parameters.

Furthermore, a set of chlorophyll fluorescence quenching parameters (Fv/Fo\_Ln, Fv/Fm\_Ln, NPQ\_Ln, Fv/Fo\_Dn, Fv/Fm\_Dn, and NPQ\_Dn) related to the stress-induced changes in the photosynthetic process were analyzed using the spider plots as shown in **Figure 4**. It includes the parameters measured at the dark-adapted state, light adaptation, and dark relaxation. The distance from the center of the spider plot indicates the relative change of the fluorescence parameter with different leaf conditions. Generally, fluorescence quenching parameters of HLB infected and nutrient deficient leaves exhibited a strong contrast compared with those of healthy samples, which is consistent with the reported studies (Pineda et al., 2008b; Spoustová et al., 2013; Mishra et al., 2016). The instantaneous variable-to-initial fluorescence and the maximal PSII quantum yield during light adaptation (Fv/Fo\_Ln and Fv/Fm\_Ln) and dark relaxation (Fv/Fo\_Dn and Fv/Fm\_Dn) dramatically decreased in HLB infected and nutrient deficient leaves. The difference in instantaneous maximal PSII quantum yield at multiple phases between the HLB infected and nutrient deficient samples was also observed. The value of instantaneous non-photochemical quenching in HLB infected and nutrient deficient leaves during light adaptation increased significantly, particularly, a large difference between the HLB infected and the nutrient deficient leaves was observed during the third (NPQ\_L3) and fourth (NPQ\_L4) of the saturating flashes and the light-adapted steady-state (NPQ\_Lss). Generally, the results shown in **Figures 3**, **4** reveal that commonly used fluorescence parameters have the potential for identifying the unhealthy leaves from the healthy ones, but it might be a challenge to differentiate the HLB infected leaves from the nutrient deficient ones by only using these parameters.

# HLB Detection Based on Optimal Features Extracted from Mean Fluorescence Parameters

Commonly used chlorophyll fluorescence parameters have demonstrated that both HLB infection and nutrient deficiency could cause the change of fluorescence emission as presented in **Figures 3**, **4**. However, they may not be the best parameters that would provide an accurate detection of HLB disease. Feature selection methods were applied here to extract the optimal features from all the mean fluorescence parameters that could characterize HLB infected leaves, and their performances are shown in **Figure 5**. It was observed that the classification accuracies varied greatly with the number of selected features, and exhibited the increasing tendency with the increased feature number. In general, the SFS selected features clearly exceeded those obtained from RF and MC-UVE methods when using both SVM and PLS-DA classifiers for three-class classification; while the performance of the RF and MC-UVE was more influenced by the classifier. As shown in **Figure 5B**, SFS-SVM achieved the best classification result of 90% with the feature subset size of 11, denoting that these 11 features might involve the most important fluorescence signatures related to the photosynthesis process affected by the HLB disease and nutrient deficiency, resulting in a better classification among healthy, HLB infected, and nutrient deficient leaves. It was also noted that the classification performance with selected features by SFS was better than that with the full dataset.

The best classification results using three feature selection methods with two classifiers based on mean fluorescence parameters are presented in **Table 1**. The SVM classifier significantly outperformed PLS-DA with the classification accuracy of 90% when using the features from all three feature selection methods. Among the combinations of the two classifiers and three feature selection methods, SFS with each of the two classifiers tended to produce the smallest feature subsets of 6 for PLS-DA and 11 for SVM with the high classification accuracies. The result also showed that the non-photochemical quenching related parameters measured during light adaptation were all selected by three methods, which was consistent with the previous analysis in **Figure 4C**.

# Chlorophyll Fluorescence Image-Based Classification

Chlorophyll fluorescence images related to the first three optimal parameters selected by RF, SFS, and MC-UVE methods were shown in **Figure 6**. The variations among healthy, HLB infected

FIGURE 2 | Electrophoresis on 1% agarose gel of DNA amplified with two specific primers (OI1/OI2c and rplLAS-F/rplLAS-R).

FIGURE 3 | Box-and-whisker plots of commonly used chlorophyll fluorescence parameters including (A) Fo, (B) Fv/Fo, (C) Fv/Fm, and (D) NPQ of healthy, HLB infected and nutrient deficient leaves. Different letters indicated significant differences (p < 0.05) by using Duncan test. Values in the figure were mean ± standard deviation (SD).

FIGURE 4 | Spider plots of selected fluorescence parameters including (A) Fv/Fo, (B) Fv/Fm, and (C) NPQ at different states of healthy, HLB infected and nutrient deficient citrus leaves measured during fluorescence quenching process.

HLB infected, and nutrient deficient leaves.

and nutrient deficient leaves were clearly observed. Some parameter images also showed the heterogeneity within the leaf due to the HLB infection or nutrient deficiency. This is agreement with the reported study on a localized decrease in photosynthesis during the disease infection or other physiological disorder processes (Kenny et al., 2011; Calatayud et al., 2013).

fpls-08-01509 August 26, 2017 Time: 14:52 # 7


TABLE 1 | Classification results based on chlorophyll fluorescence parameter analysis with the optimal features.

RF, random frog; SFS, sequential forward selection; MC-UVE, Monte Carlo of uninformative variables elimination method; PLS-DA, partial least squares discriminant analysis; SVM, support vector machine.

The spatial heterogeneity of the HLB symptoms caused by the HLB bacterium presents more challenges for the disease identification. Hence, chlorophyll fluorescence imagebased analysis was also performed to improve the HLB detection accuracy. The first six principal component (PC) images representing over 95% variables were used for image feature extraction. Features extracted by SIFT from PC images were used as the SVM classifier input. **Table 2** summarized the three-class classification results of using SVM based on image features and image features combined with the mean fluorescence parameters, respectively. The overall accuracy based on image features was 77%, which was not as good as that using mean fluorescence parameters with the classification accuracy of 90%. However, the combination of the image features and the mean fluorescence parameters significantly improved the classification performance with the overall accuracy of 97%. This suggests that contrast information among these three classes was enhanced when using data fusion.

# DISCUSSION

Various studies have reported that chlorophyll fluorescence is a promising technique for non-invasive measurement of PSII activities, and could achieve a rapid screening of photosynthetic processes related to the plant status (Ehlert and Hincha, 2008; Rolfe and Scholes, 2010; Ivanov and Bernards, 2016). Commonly used chlorophyll fluorescence parameters, such as Fo, Fv/Fo, Fv/Fm, and NPQ\_Lss, obtained by exposing the leaf to a combination of darkness, actinic lights and a series of saturating


TABLE 2 | Support vector machine (SVM) classification results based on the principal component (PC) image features and the combination of image features and mean fluorescence parameters.

flashes as shown in **Figure 1**, are considered as the most useful parameters to interpret energy dissipation of chlorophyll in the thylakoid membrane. In our study, these four fluorescence parameters measured in healthy, HLB infected, and nutrient deficient leaves showed the potential of differentiating the unhealthy leaves from the healthy ones except Fo (**Figure 3**). Large variations in Fo might be one of the reasons that it is difficult to provide a straightforward explanation about the impacts of HLB infection and nutrient deficiency on the PSII reaction center and energy transfer. The parameter Fv/Fo is considered as an indicator of the number and the size of active photosynthetic reaction centers (Dan et al., 2000). The dramatic declines of Fv/Fo in HLB infected and nutrient deficient leaves indicate the change in the rate of electron transport from PSII to the primary electron acceptors with the reduction of the number and the size, which have also been reported in different plants exposed to the disease and the environmental stresses (Zhou et al., 2009; Martinazzo et al., 2012; Janka et al., 2013). The decrease of Fv/Fm in HLB infected and nutrient deficient leaves could be a result of the major damage to the photosynthetic apparatus in response to the HLB infection and nutrient deficiency. The value of NPQ\_Lss in HLB infected and nutrient deficient leaves might be related to the different degrees of tissue damage, which can also be reflected from the severity of the HLB and nutrient deficient symptoms. Reported studies have revealed that the increase of the NPQ\_Lss depends on the stimulated electron flow as a protection mechanism (Tatagiba et al., 2016). What's more, it indicates the excess excitation energy dissipated as heat in order to reduce the photooxidative damage, which is also coupled to a gradual loss of chloroplastic pigments in response to the HLB infection and nutrient deficiency, eventually resulting in blotchy yellowing symptoms in leaves. Meanwhile, the significant differences observed in Fv/Fm and NPQ\_Lss values between the HLB infected and nutrient deficient leaves demonstrated that it was possible to identify HLB infected leaves from the nutrient deficient ones, although the interpretation is not straightforward for a number of reasons.

Although previous research has reported a desired discrimination accuracy (92–95%) with adding fluorescence at 690 nm by using fluorescence imaging spectroscopy suggesting the significant contribution of emission from chlorophyll a molecule for HLB detection (Wetterich et al., 2016, 2017), more fundamental understanding about energy partition in PSII in the HLB infected leaves should be gained. The chlorophyll fluorescence quenching process represents the complex dynamics of plant photosynthetic reaction during a transition from a darkadapted to a light-adapted state (Govindjee, 1995). Some parameters measured at different time-courses during quenching process could offer photosynthetic signatures related to the plant disease (Berger et al., 2007). Further analysis of chlorophyll fluorescence parameters obtained during fluorescence quenching process as shown in **Figure 4** provided additional feature information related to the HLB disease, especially the consistent decrease in instantaneous non-photochemical quenching at several states in response to the HLB infection indicated a direct effect of HLB disease on PSII due to the irreversible damage of the leaf tissue.

Advanced machine learning and statistical analysis of finding the photosynthetic fingerprints of the HLB disease detection was shown to be superior to the commonly used fluorescence parameters. Designing an effective criterion to select an optimal subset of features is a challenging problem for data classification. For the mean fluorescence parameters analysis, SFS-SVM reached a best overall accuracy of 90% suggesting that features selected by SFS can better describe the pattern of the fluorescence data than the other two methods. Compared with the commonly used fluorescence parameters, feature selection methods provided a new insight to achieve a better interpretation of the original data by removing the redundant information and enhancing the most useful information related to the HLB infected characteristics. Although image features could provide the spatial information about the photochemical and non-photochemical components, the performance of the threeclass classification based on image features extracted from PC images combined with the SVM classifier, was not better than those from mean fluorescence parameters (**Tables 1**, **2**). This suggests that nutrient deficiency in leaves would induce similar image texture to HLB infected ones as shown in **Figure 6**. This is probably a reason why local image features had a poorer classification performance. The best classification accuracy of 97% using the combination of image features and mean fluorescence parameters revealed its capability to discriminate among healthy, HLB infected, and nutrient deficient leaves. In general, the results obtained in this study was comparable to those obtained using other spectroscopy and imaging techniques (Garcia-Ruiz et al., 2013; Sankaran et al., 2013; Wetterich et al., 2016, 2017). However, one should notice the differences

of experimental conditions and samples when comparing our classification accuracy with those from other techniques. It is true that the conventional method depending on the commonly used chlorophyll fluorescence parameters provided a better plant physiological interpretation, but it is challenging to perform a quantitative analysis of decision making for the HLB diagnosis in various conditions. Recent studies have approved that feature selection methods as well as data fusion have exploited a new way to interpret and extract the hidden information from the original data, such as image and spectral data (Li et al., 2015; Liu and Li, 2016; Fu et al., 2017). Better classification results by employing proper feature selection method and classifier (**Table 2**) demonstrated the capability of this data-driven strategy for the chlorophyll fluorescence image analysis.

In this study, we presented a comprehensive investigation on the photosynthetic characteristics of HLB infected citrus leaves and the possibility of the HLB disease detection using chlorophyll fluorescence imaging combined with feature selection. The combination of mean fluorescence parameters and images features significantly improved the classification performance with the accuracy of 97%, which indicated a better interpretation for the spatial heterogeneity of photochemical and non-photochemical components in HLB infected citrus leaves. The proposed approach cannot only achieve a better detection

#### REFERENCES


accuracy of the HLB disease, but also be developed as a new means for the plant photosynthetic analysis.

#### AUTHOR CONTRIBUTIONS

HC and HW designed the experiment. HW, JY, MH, JL, and SH performed the experiment. HC and HW contributed to the data analysis and wrote the manuscript. HL and YH provided suggestions on the experiment design and discussion sections.

#### ACKNOWLEDGMENTS

This work was supported by the Key Research and Development Program from the Science Technology Department of Zhejiang Province (2015C02007) and China Agriculture Research System (CARS-27).

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpls.2017.01509/ full#supplementary-material


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Cen, Weng, Yao, He, Lv, Hua, Li and He. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

fpls-08-01509 August 26, 2017 Time: 14:52 # 11

# Can Leaf Water Content Be Estimated Using Multispectral Terrestrial Laser Scanning? A Case Study With Norway Spruce Seedlings

Samuli Junttila1,2 \*, Junko Sugano<sup>1</sup> , Mikko Vastaranta1,2,3, Riikka Linnakoski 1,4 , Harri Kaartinen5,6, Antero Kukko2,5, Markus Holopainen1,2, Hannu Hyyppä2,7 and Juha Hyyppä2,5

#### *Edited by:*

Alexander Bucksch, University of Georgia, United States

#### *Reviewed by:*

Asish Kumar Parida, CSIR-Central Salt & Marine Chemicals Research Institute, India Ioannis Konstantinos Christodoulakis, National and Kapodistrian University of Athens, Greece András Zlinszky, Institute of Ecology Research Center (MTA), Hungary

> *\*Correspondence:* Samuli Junttila samuli.junttila@helsinki.fi

#### *Specialty section:*

This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science

> *Received:* 30 September 2017 *Accepted:* 21 February 2018 *Published:* 08 March 2018

#### *Citation:*

Junttila S, Sugano J, Vastaranta M, Linnakoski R, Kaartinen H, Kukko A, Holopainen M, Hyyppä H and Hyyppä J (2018) Can Leaf Water Content Be Estimated Using Multispectral Terrestrial Laser Scanning? A Case Study With Norway Spruce Seedlings. Front. Plant Sci. 9:299. doi: 10.3389/fpls.2018.00299 <sup>1</sup> Department of Forest Sciences, University of Helsinki, Helsinki, Finland, <sup>2</sup> Centre of Excellence in Laser Scanning Research, Finnish Geospatial Research Institute (FGI), Masala, Finland, <sup>3</sup> School of Forest Sciences, University of Eastern Finland, Joensuu, Finland, <sup>4</sup> Natural Resources Institute Finland (Luke), Helsinki, Finland, <sup>5</sup> Department of Remote Sensing and Photogrammetry, Finnish Geospatial Research Institute (FGI), Masala, Finland, <sup>6</sup> Department of Geography and Geology, University of Turku, Turku, Finland, <sup>7</sup> Department of Built Environment, Aalto University, Aalto, Finland

Changing climate is increasing the amount and intensity of forest stress agents, such as drought, pest insects, and pathogens. Leaf water content, measured here in terms of equivalent water thickness (EWT), is an early indicator of tree stress that provides timely information about the health status of forests. Multispectral terrestrial laser scanning (MS-TLS) measures target geometry and reflectance simultaneously, providing spatially explicit reflectance information at several wavelengths. EWT and leaf internal structure affect leaf reflectance in the shortwave infrared region that can be used to predict EWT with MS-TLS. A second wavelength that is sensitive to leaf internal structure but not affected by EWT can be used to normalize leaf internal effects on the shortwave infrared region and improve the prediction of EWT. Here we investigated the relationship between EWT and laser intensity features using multisensor MS-TLS at 690, 905, and 1,550 nm wavelengths with both drought-treated and Endoconidiophora polonica inoculated Norway spruce seedlings to better understand how MS-TLS measurements can explain variation in EWT. In our study, a normalized ratio of two wavelengths at 905 and 1,550 nm and length of seedling explained 91% of the variation (R 2 ) in EWT as the respective prediction accuracy for EWT was 0.003 g/cm<sup>2</sup> in greenhouse conditions. The relation between EWT and the normalized ratio of 905 and 1,550 nm wavelengths did not seem sensitive to a decreased point density of the MS-TLS data. Based on our results, different EWTs in Norway spruce seedlings show different spectral responses when measured using MS-TLS. These results can be further used when developing EWT monitoring for improving forest health assessments.

Keywords: terrestrial laser scanning, tree health, drought stress, multispectral laser scanning, leaf water content, forest damage, *Endoconidiophora polonica*

# INTRODUCTION

Measuring tree health is an increasingly important topic as the world's climate is warming and the human population is growing, putting the environment and forests under increasing stress (Williams et al., 2013). Stress and different disturbance events caused by, for example, insects are natural processes in forest ecosystems that can help maintain healthy and heterogeneous forests (Raffa et al., 2009). However, forests are facing new stresses and disturbance events such as drought, invasive pests, and air pollution, which are more intense than those they have previously adapted to (Trumbore et al., 2015). Forests provide many ecosystem services, such as carbon storage, recreational activities, timber, and non-timber forest products, which are jeopardized by declining forest health, increased forest mortality, and increased susceptibility to forest fires (Hanewinkel et al., 2013; McDowell and Allen, 2015; Morris et al., 2016). Managing declining forests efficiently in the face of climate change requires more information on the condition of forests to be able to monitor and control forest damage and fires (Allen et al., 2010); thus, new methods for mapping and monitoring forest health are needed.

Existing forest health mapping methods can be divided according to spatial, spectral, and temporal resolutions (Senf et al., 2017). Large-scale mapping of insect infestation has been conducted using single and multi-date Landsat data at 30 m resolution (Goodwin et al., 2008; Meddens et al., 2013). These methods are capable of capturing forest health information on a large scale and are suitable for countries where extensive forest management is applied, but spatially, more detailed approaches are necessary for capturing small-scale dynamics of forest decline in smaller countries with a more intensive approach to forest management. Spatially more accurate hyperspectral remote sensing data with 5 m resolution have been used to investigate the mapping of bark beetle induced tree mortality resulting in high overall accuracy (84–96%) in classifying dead trees (Fassnacht et al., 2014). Multispectral satellite imagery with 4 m resolution has been used to detect the red stage in mountain pine beetle infestation, resulting in high overall accuracy of 71–92% (White et al., 2005). However, none of the methods studied so far have been able to accurately detect early stages of bark beetle infestation and early tree stress when the trees have exhibited few visual symptoms (Fassnacht et al., 2014).

Leaf water content, typically measured as the weight of water per leaf area or equivalent water thickness (EWT), is an indicator of tree health that is affected by several tree stressors (Carter, 1993; Chaerle and Van Der Straeten, 2000). Drought, pest insects, and pathogens affect EWT, often before other symptoms appear, such as discoloration or defoliation; thus, detecting changes in EWT provides an early-warning signal of tree stress (Skakun et al., 2003; Zarco-Tejada et al., 2003). Leaf and canopy water content derived from Landsat data have been used as a proxy to detect large-scale insect infestation (Skakun et al., 2003; Wulder et al., 2006). Early detection of tree stress is crucial in providing timely information about the condition of forests, mitigating damage, and preventing the spread of the stressor (Wermelinger, 2004). For example, the European spruce bark beetle (Ips typographus L.) is a forest insect pest whose main host is Norway spruce trees. The bark beetle and a fungal pathogen have a mutual relationship causing disruptions in the flow of water and nutrients in the phloem and sapwood of the trees, resulting in decreased water content in the canopy and the sapwood (Horntvedt et al., 1983). Recently infested trees do not exhibit visual symptoms (green attack), but a reddish-brown coloring of tree crowns appears when the beetles have already left the host tree (Lausch et al., 2013). Thus, early detection of tree stress would greatly assist in mitigating and managing the damage caused by I. typographus. Efforts have been made to solve this issue but none have succeeded with reasonable accuracy thus far (Lausch et al., 2013; Fassnacht et al., 2014).

Improving reference data for assessing forest health has been identified as a key research area in need of development (Senf et al., 2017). Because field reference data or the ground truth is the gold standard of modeling, any errors in the data can lead to significant bias and errors in modeling. The high costs of collecting field data usually lead to a compromise between the number of plots and the spatial extent of the investigation. National forest inventories may collect forest health data, but forest inventory data are not available in all countries (Gschwantner et al., 2016). Visual assessment of forest health in field data collection is prone to error and bias due to the subjective nature of the estimation, and it requires substantial expertise. The early stages of tree stress can be visually difficult to determine because trees may not exhibit visual symptoms at this stage. Thus, objective and reliable methods for collecting forest health related reference data in the field are needed.

Equivalent water thickness is closely related to another metric for calculating vegetation water content—fuel moisture content (FMC)—that is more frequently used in forest fire danger literature (Chuvieco et al., 2004). Fuel moisture content is expressed as the ratio between the weight of water and sample dry weight and is an important parameter in fire ignition (Dimitrakopoulos and Papaioannou, 2001). A multitemporal analysis of NOAA-AVHRR images has been used to derive FMC for Mediterranean grasslands and shrub species with R 2 -values over 0.8 (Chuvieco et al., 2004). Generally, studies aiming to estimate FMC use spatially coarse resolution satellite data that can provide frequent imagery, important in forest fire danger estimation (Sow et al., 2013).

Multispectral terrestrial laser scanning (MS-TLS) measures the range and reflectance of the target at several wavelengths simultaneously, providing three-dimensional estimates of reflectance. Several MS-TLS instruments have been developed recently mainly for laboratory use (Douglas et al., 2012; Hakala et al., 2012; Wallace et al., 2012; Wei et al., 2012; Niu et al., 2015; Li et al., 2016). These instruments utilize a supercontinuum laser source that is powerful enough to not be eye-safe, limiting their operation in the field, with the exception of a terrestrial laser scanner operating at two wavelengths, which is also portable (Douglas et al., 2012). Recently, a commercial multispectral laser scanner—the Optech Titan (Teledyne Optech, Canada)—with three channels has become available for airborne measurements (Matikainen et al., 2016). The development of

multispectral laser scanning instruments widens the possibilities of using laser scanning data as more information about the target's reflectance becomes available in a three-dimensional format.

Alterations in EWT affect leaf reflectance in the shortwave infrared region (Ceccato et al., 2001), which can be measured using MS-TLS. A number of studies have investigated the use of MS-TLS and terrestrial laser scanning (TLS) in the estimation of EWT. Gaulton et al. (2013)studied the potential of estimating leaf EWT with a dual-wavelength terrestrial laser scanner at 1,063 and 1,545 nm wavelengths, resulting in a significant correlation (R <sup>2</sup> = 0.8, root mean square error [RMSE] = 0.0069 g/cm<sup>2</sup> ) between leaf EWT and a normalized ratio of the two wavelengths, but with a low number of samples and species. The estimation of leaf EWT has been recently studied using MS-TLS at 690 and 1,550 nm wavelengths with single leaves, resulting in a strong correlation (R <sup>2</sup> = 0.93, RMSE = 0.004 g/cm<sup>2</sup> ) between EWT and a normalized ratio of the two wavelengths, but the selection of wavelengths was concluded to be suboptimal (Junttila et al., 2016). Zhu et al. (2015) showed that TLS intensity data at 1,550 nm were able to explain 76% of the variation in EWT at leaf level after radiometric correction of the TLS intensity data. Canopy EWT has been estimated with full-waveform single wavelength TLS at 1,550 nm wavelength, resulting in a significant correlation between canopy EWT and backscatter coefficient (R <sup>2</sup> = 0.66, RMSE = 0.001 g/cm<sup>2</sup> ) (Zhu et al., 2017). However, single wavelength data seem to be sensitive to confounding factors, such as incidence angle, requiring complicated correction of the intensity data, and they are also affected by structural variables of the leaf other than EWT (Zhu et al., 2015, 2017). The requirement of an incidence angle correction limits the utilization of the single wavelength method to deciduous species with distinguishable leaves. Coniferous forests comprise a large part of the world's forest biomes, but thus far only Junttila et al. (2016) have looked into the estimation of EWT with coniferous species using MS-TLS. The use of spectral ratios calculated from dual-wavelength intensity data has been shown to be insensitive to incidence angle effects thus enabling higher accuracy in the estimation of EWT (Hancock et al., 2017). A second wavelength in the near infrared region coupled with a wavelength in the shortwave infrared region is also capable of normalizing leaf structural effects on the estimation of EWT (Ceccato et al., 2001).

The aim of this study was to investigate the capability of multisensor MS-TLS in monitoring leaf EWT, with Norway spruce (Picea abies L.) seedlings. First, the dependencies between EWT and laser intensity features at different wavelengths (and calculated spectral ratios) of segmented point clouds are examined to find explanatory variables for predicting EWT. Secondly, we compare the combination of 690, 905, and 1,550 nm wavelengths in predicting EWT using different sensors. Thirdly, the importance of point density in estimating EWT is evaluated using random sampling of the point clouds to decrease the point density. This research contributes to the first steps in developing a method for the accurate estimation of leaf EWT in forests using MS-TLS.

# MATERIALS AND METHODS

#### Experiment Design

Two-year-old commercial Norway spruce seedlings (n = 145) were grown for 12 weeks in a greenhouse between May and August 2016. The seedlings were subjected to different treatments to induce drought and variation in EWT. The seedlings were divided into five groups for different treatments. Three of the groups received 75% (28 ml), 50% (20 ml), and 25% (12 ml) of "normal" watering (watered as required to maintain moist soil) three times a week (groups D75, D50, and D25, respectively) (**Table 1**). The fourth group was grown with a sufficient amount of water (36 ml) for 10 weeks until irrigation was completely stopped, and the seedlings then went without water for 3 weeks (group D\_total). The fifth group of seedlings was inoculated with Endoconidiophora polonica (isolate CBS 142283), a fungal pathogen associated with the European spruce bark beetle (I. typographus) that disturbs the flow of water and nutrients in the phloem and sapwood (group F). The amount of water given to the seedlings in the drought treatment groups (D75, D50, and D25) was adjusted during the experiment because the seedlings showed resistance to drought (**Table 2**). After 34 days from the start of the experiment, irrigation was reduced to twice a week; after 46 days, the amount of water given was cut to half; and finally after 56 days from the start of the experiment the irrigation was reduced to once a week. During the 12 weeks, 5–16 seedlings were randomly collected for TLS measurements at eight time intervals. Two or three seedlings were collected from each of the drought treatment groups (D75, D50, and D25) and eight seedlings from the fungal treatment group (F) for each MS-TLS measurement. Seedlings that showed severe symptoms during the final stages of the experiment, and which were unlikely to survive until the next measurement time were prioritized to avoid the loss of research material, resulting in uneven sample numbers toward the end of the experiment. Details of the measurements can be found in **Table 3**. The mean length of the seedlings was 30.3 cm (std ± 5.34 cm) during the experiment.

#### Terrestrial Laser Scanners

The seedlings were scanned consecutively with three different terrestrial laser scanners from the same position from a distance of 5.2 m inside the greenhouse. The terrestrial laser scanners used were a Leica HDS6100 (Leica Geosystems AG, Heerbrugg, Switzerland), a FARO S120 (FARO Europe GmbH & Co. KG, Korntal-Münchingen, Germany), and a FARO X330, utilizing wavelengths of 690, 905, and 1,550 nm, respectively. The technical specifications of the terrestrial laser scanners can be found in **Table 4**. The varying beam diameter at exit and beam divergence resulted in different spot sizes at target distance, which were 4.14, 3.99, and 3.24 mm for the Leica, the S120, and the X330, respectively. The scanning was done using the highest resolution available for each instrument, resulting in a point spacing of ∼1 mm; thus, there was significant overlap between consecutive laser measurements.

Three white spheres were used as common targets to register the scans to the same coordinate system as each other, facilitating the segmentation of point clouds in the processing of the data. A four-grade Spectralon (Labsphere, North Sutton, NH, USA) panel was used as a reference target for intensity normalization. The size of the panel was 460 mm x 460 mm consisting of four different reflectance panels (nominal reflectances of 99, 50, 25, and 12%), each sized 115 × 460 mm. Approximately 65,000 points were received from each of the panels.

#### Ecophysiological Measurements

After scanning, the seedlings were transported to the laboratory for EWT measurements (**Table 3**). A sample of 30–40 needles was randomly collected from each seedling, constituting a single measurement of EWT. As the seedlings were growing during the experiment, only previous year needles were sampled to avoid the influence of needle development on the EWT measurements. The needles were weighed (with a precision of 0.0001 g) to measure fresh weight, scanned with an Epson V370 Photo Scanner (Epson America, Inc., CA, USA) at 800 dpi resolution to measure leaf area, and dried in an oven at 60◦C for 48 h to measure dry weight. The images were analyzed with the open source software EasyLeafArea (Easlon and Bloom, 2014) to segment the needles and calculate leaf area based on the number of segmented pixels. The EWT was then calculated according to Danson et al. (1992):

$$\frac{FW - DW}{A} \left(\frac{\text{g}}{cm^2}\right),\tag{1}$$

where FW is the fresh weight of the needles (g), DW is the weight of the dried needles (g), and A is the surface area of the fresh needles (cm<sup>2</sup> ).

#### Terrestrial Laser Scanning Data Processing

The TLS scans were registered to a common coordinate system using the Z + F LaserControl software package (Zoller + Fröhlich GmbH, Wangen im Allgäu, Germany), resulting in a mean accuracy of 2.7 mm between targets. The registration

TABLE 1 | Treatment groups and related statistics.


TABLE 2 | Irrigation of the drought experiment groups.


process was based on Helmert transformation and three sphereshaped targets with known locations (Watson, 2006). The registration of the scans was visually inspected after the registration process to ensure the alignment of the point clouds at the range of the seedlings. Because the terrestrial laser scanners utilize a phase shift measurement technique to determine the range to the target, a large number of ghost points were produced in the scans (Balduzzi et al., 2011). To reduce the number of ghost points, the point clouds were filtered with a statistical outlier algorithm based on the distance between points by using the CloudCompare v.2.8.1 (Girardeau-Montaut, 2011) software. The algorithm calculates the maximum distance for a point to be included using the following equation:

$$MaxD = MeanD + n \\ Sigma \ast std,\tag{2}$$

where MaxD is the maximum distance for a point to be included, MeanD is the mean distance of the neighboring points, nSigma is a standard deviation multiplier threshold, and std is the standard deviation of the distance of the neighboring points. Thus, the algorithm requires two parameters: the number of neighbors and nSigma. The parameters were determined though iteration and visual estimation of the result. Fifty neighbors were used for every scan and the nSigma was set to 1.0 and 0.65 for the Leica and FARO scanners respectively.

Then, cloud-to-cloud distances were calculated for the point clouds in the CloudCompare software package where the FARO X330 was used as a reference cloud for calculating the distances because of its expected sensitivity to EWT (Junttila et al., 2016). The tool calculates the distance of each point to the nearest point in the reference cloud. A maximum distance of 2 mm (which is also the ranging accuracy of the terrestrial laser scanners) was used to filter the point clouds and match the geometry of the point clouds produced by different terrestrial laser scanners with each other.

Points from each seedling were then manually clipped from the georeferenced point clouds for further processing. The point clouds from each seedling were further manually segmented into stem and leaf points to evaluate their effect on the estimation of EWT.

Because the point clouds were produced using a very highresolution scanning setting, the point clouds were randomly sampled to reduce the amount of points in each point cloud and investigate how the point density affects the estimation of EWT. The point clouds were randomly sampled to four different point cloud sizes: 2,000, 1,000, 500, and 250 points per seedling. The mean number of points for each seedling was 14,702 in the unsegmented point clouds; thus, the amount of points was greatly reduced.

#### Intensity Calibration

Terrestrial laser scanners are primarily designed for range measuring; thus, they are not optimized for reflectivity measurements. Some terrestrial laser scanners have been shown to have a nonlinear intensity scale (Kaasalainen et al., 2011); thus,


TABLE 3 | EWT for each measurement and treatment group, and their respective statistics.

TABLE 4 | Technical specifications for the terrestrial laser scanners.


DN, digital number.

the calibration of intensity data may be required. The stationary four-grade Spectralon panel was used as a reference target for the calibration procedure using the mean of ∼65,000 points from each of the reflectance panels. The standard deviation of raw intensity values of these samples varied between 4.7 and 8.9. The panel's reflectance was also measured with a FieldSpec Pro FR (Analytical Spectral Devices, Inc., Longmont, CO, USA) field spectrophotometer. The relationship between the spectrophotometer measured reflectance and the laser scanner measured intensity was then investigated to detect any nonlinearities in the intensity response.

The relationship between measured intensity and reflectance was shown to be linear for the Leica scanner and logarithmic for the FARO scanners (**Figure 1**). The measured intensity from the FARO X330 scanner was shown to be saturated in the 99% panel because the maximum intensity of the digital number was reached; thus, the calibration of FARO X330 data was done using only the 50, 25, and 12% panels. To correct for the logarithmic effect of the FARO scanners, Equation 3 was used (Kaasalainen et al., 2009):

$$y = 10^{\frac{(\chi - A\_1)}{A\_0}},\tag{3}$$

where A<sup>0</sup> and A<sup>1</sup> are empirical parameters determined by fitting the raw intensity measured by the FARO scanners to the spectrophotometer measured reflectance. Additionally, a linear regression model was developed to model the relationship between the raw intensity and reflectance for the Leica scanner to convert the raw intensity values to calibrated intensity. The fitted parameter values of A<sup>1</sup> and A<sup>0</sup> were 1833.8 and 446.9 for FARO S120, and 2018.7 and 379.9 for FARO X330. The linear regression model of the Leica scanner had a coefficient of 0.00119 and an intercept of −0.57186.

The effect of ambient temperature, temperature of the scanner, and background illumination on measured intensity was then reduced by normalizing each scan with the mean intensity of ∼20,000 returns from the 99% Spectralon panel in the case of the Leica and FARO S120, and from the 50% Spectralon panel for the FARO X330 due to the saturation of intensity values in the 99% panel. The intensity of the X330 points from the seedlings had values below the mean of the 50% Spectralon panel; thus, the intensity was within the calibrated range.

The result of this process is referred to as the calibrated intensity within this paper. Because the samples were scanned within a narrow range bin of less than 1 m, no range corrections were conducted. A range correction is required when the targets are located at different distances from the scanner, i.e., scanning trees in a forest environment (Kaasalainen et al., 2011).

#### Laser Intensity Features

A set of statistical features was calculated from the intensity data from each point cloud representing a seedling at each wavelength. Intensity features were calculated from the calibrated intensity data of each point cloud. These features were the mean, minimum, maximum, standard deviation and percentiles 10, 20, 30, 40, 60, 70, 80, and 90 (p-value, e.g., p20, p60) of the intensity value distribution. Within this paper these statistics are referred to with a wavelength and a subscript describing the feature (e.g., 1,550mean, 690p30). The included features are shown in **Table 5**.

Based on these features, a set of spectral indices was calculated for each point cloud (**Table 6**). We aimed to observe combinations of the 690 and 1,550 nm and 905 and 1,550 nm wavelengths based on ratio and normalization operations by calculating a simple ratio (SR) index (Jordan, 1969) and a normalized difference index (NDI) (Hancock et al., 2017) as follows:

$$\text{SR\\_}i\_{\rho1,\rho2} = \frac{\rho \,\text{l}\_i}{\rho \,\text{2}\_i} \,, \tag{4}$$

#### TABLE 5 | Intensity features.


$$\text{NDI}\_{-}i\_{\rho1,\rho2} = \frac{\rho 1\_i - \rho 2\_i}{\rho 1\_i + \rho 2\_i} \,, \tag{5}$$

where ρ1<sup>i</sup> and ρ2<sup>i</sup> are the i feature of wavelength ρ1 and ρ2.

The indices are referred to with the abbreviation, feature name, and a subscript describing the wavelengths used for the calculation (e.g., NDI\_p80690, 1,550). The total number of calculated laser intensity features was 48.

#### Statistical Analysis

The Student's t-test was used to determine whether there were statistically significant differences in the length of the seedlings in the different treatment groups. The relationship between EWT and each laser intensity feature was investigated using simple linear regression. The regression models were developed for all points, leaf points and stem points separately. Regression models were also developed for the randomly sampled point clouds to investigate how point density affects the estimation of EWT. Multiple regression was used to investigate whether the estimation of EWT could be improved by including length of the seedling as a structural variable in the regression model with the best explanatory laser intensity feature. We used the coefficient of determination (R 2 ) and RMSE with cross-validation to assess the goodness of fit between the dependent and the predictor using the following equations:

$$RMSE = \sqrt{\frac{\sum\_{i=1}^{n} \left(\mathbf{y}\_{i} - \hat{\mathbf{y}}\_{i}\right)^{2}}{n}},\tag{6}$$

$$\mathcal{R}^2 = 1 - \frac{\sum\_{\mathbf{i}} \left(\boldsymbol{\wp\_{\mathbf{i}}} - \boldsymbol{\wp\_{\mathbf{i}}}\right)^2}{\sum\_{\mathbf{i}} \left(\boldsymbol{\wp\_{\mathbf{i}}} - \boldsymbol{\wp\_{\mathbf{i}}}\right)^2},\tag{7}$$

where n is the number of observations, y<sup>i</sup> is the observed value for the measurement i, yˆ<sup>i</sup> is the predicted value for the measurement i, and y¯ is the mean of the observed data. All of the statistical analyses were performed using the open source software package R ver. 3.2.3 (R Core Team, 2013). Due to the large number of explanatory laser intensity features, only the 10 best explaining variables with the highest R <sup>2</sup> were reported.

#### RESULTS

#### Effects of Drought on Growth and Equivalent Water Thickness

The drought treatments resulted in varying lengths of the seedlings in the treatment groups (**Figure 2**). The D75 group had a mean length of 29.5 cm, while the D50 and D25 groups had significantly lower mean lengths of 26.0 and 23.7 cm, respectively. The drought treatments significantly affected the growth of the seedlings.

The seedlings showed resistance to drought. Despite the different amounts of water that the seedlings in the drought treatment groups received, EWT was shown to stay at relatively similar levels between treatment groups until the irrigation amount was further reduced after 35 days from starting the

FIGURE 2 | Length of the seedlings in the drought treatment groups (number of samples: D75: 21, D50: 17, D25: 13). Asterisk (\*) on top of the bar denotes statistically significant difference between the groups. Black line denotes the median value, upper and lower edge of the box show 75 and 25% percentiles, respectively. The whiskers of the boxes extend to the extreme values of the data sets.

TABLE 6 | Calibrated intensity values for the mean of each wavelength and each spectral index for all points, leaf points, and stem points.


NDI, normalized difference index; SR, simple ratio (index).

experiment (**Figure 3**). The seedlings still showed few visual symptoms due to drought, and after further reductions of irrigation the EWT decreased significantly. The lowest EWT levels in the experiment were found at the end of the experiment in the D\_total group.

# Effects of *E. polonica* Treatment on Equivalent Water Thickness

The fungal pathogen treatment of the seedlings resulted in decreased EWT (**Figure 4**). The affected seedlings showed a rapid decrease in EWT because the effects were apparent only 14 days after the inoculation. The infected seedlings were shown to respond to the treatment during the first 35 days of the experiment. After that period, no signs of infection in the remaining seedlings were observed.

#### Does Equivalent Water Thickness Affect the Spectral Response of Multispectral Terrestrial Laser Scanning?

The calibrated intensity at 1550 nm wavelength from the FARO X330 showed significantly increasing values with decreasing EWT (**Figure 5**). The linear regression model of EWT and the best explanatory variable at 1,550 nm wavelength, 1,550p80, had a coefficient of determination of 0.7. For comparison, 1,550mean was able to explain only 47% of the variation in EWT. Spectral indices calculated from both combinations −905 and 1,550 nm and 690 and 1,550 nm – were able to predict EWT with improved accuracy. The combination of 905 and 1,550 nm showed the best prediction accuracy of EWT with R 2

of 0.89 and 0.87 for NDI\_p70905, 1,550 and NDI\_p60905, 1,550, respectively (**Table 7**). Similar prediction accuracy, with an R <sup>2</sup> of 0.87, was also observed for SR\_p70905, 1,550 and NDI\_p80905, 1,550. The best explanatory features for the combination of 690 and 1,550 nm showed prediction accuracies of R <sup>2</sup> of 0.82 and 0.80 for NDI\_p70690, 1,550 and SR\_p70690, 1,550, respectively.

The best explanatory laser intensity features varied slightly for leaf points. The 1,550std feature could explain 79%, and 1,550p90 73%, of the variation in EWT as the best predicting features for single wavelength data. The spectral indices showed



similar explanatory power compared with using all points, but the best features for estimating EWT were higher intensity percentiles than for all points. The NDI\_p80905, 1,550 and the NDI\_p70905,1550 features explained 89% of the variation in EWT, while SR\_p70905, 1,550 and SR\_p80905, 1,550 explained 87% of the variation in EWT. The NDI\_p90690, 1,550 and the SR\_p90690, 1,550 features were the best predictors for the combination of 690 and 1,550 nm wavelengths, with R <sup>2</sup> of 0.81 and 0.79, respectively.

The stem points were also shown to have statistically significant power in explaining the differences in EWT but with lower percentiles of the intensity value distribution and lower coefficients of determination of 0.49, 0.66, and 0.75 for 1550 nm only, 1,550 and 905 nm, and 1,550 and 690 nm, respectively. Single wavelength data explained 49% of the variation in EWT with the 1,550mean feature. The combination of 690 and 1,550 nm wavelengths showed the highest prediction accuracy, with the SR\_mean690, 1,550 and SR\_p40690, 1,550 features resulting in an R <sup>2</sup> of 0.75 and 0.74, respectively. The NDI\_p40905, 1,550 and the NDI\_p10905, 1,550 features were able to predict EWT with R <sup>2</sup> of 0.66 and 0.65, respectively.

The regression models explained 89% of the variation in EWT when 2,000 points from each seedling were randomly sampled. Thus, there was only a 0.0001 g/cm<sup>2</sup> increase in RMSE for both spectral indices compared using all points (**Table 8**). A slight decrease of 0.02 in R 2 and an increase of 0.0002 g/cm<sup>2</sup> was present for NDI\_p70905, 1,550 after random sampling of 1,000 points, whereas NDI\_p70690, 1,550 showed no changes compared with 2,000 points. Reducing the number of points to 500 resulted in no changes in the estimation accuracy of EWT for NDI\_p70905, 1,550 but R <sup>2</sup> decreased to 0.79 and RMSE increased to 0.0048 g/cm<sup>2</sup> for NDI\_p70690, 1,550. When the number of points was reduced to 250, R <sup>2</sup> decreased to 0.79 and 0.78 and RMSE increased to 0.0047 g/cm<sup>2</sup> and 0.0049 g/cm<sup>2</sup> for NDI\_p70905, 1,550 and NDI\_p70690, 1,550, respectively.

Multiple regression was able to improve the prediction of EWT. According to the multiple regression analysis, the length was statistically significant at p < 0.001. The estimation accuracy of EWT was improved compared with using simple TABLE 8 | R <sup>2</sup> and RMSE values for the linear regression models of the spectral indices and equivalent water thickness after random sampling of the point clouds.


regression models, with R <sup>2</sup> = 0.91 and RMSE = 0.003 g/cm<sup>2</sup> (**Figure 6**).

#### DISCUSSION

The seedlings showed surprisingly high resistance to drought despite the low amounts of irrigation for each seedling. The drought led to a significant reduction in growth, which correlated with the intensity of the drought treatment. The seedlings seemed to cope with drought by reducing growth, which led to reduced demand for water because there was less foliage to support and less transpiring leaf area. However, significant reduction in EWT was observed after further reduction in irrigation.

The results showed that the investigated laser intensity features were able to explain most of the variation in leaf EWT and provide a relatively good prediction accuracy. Single wavelength intensity features at 1,550 nm were able to explain 70– 79% of the differences in EWT using all points and leaf points only. Similar results have been obtained before for broadleaf species after applying an incidence angle correction to the intensity data at 1,550 nm (Zhu et al., 2015; Junttila et al., 2016). The correlations obtained in this study are relatively high for single wavelength data considering that no radiometric

corrections have been applied to the data. Due to the shape and size of the spruce needles, an incidence angle correction would be impossible to apply because the laser footprint illuminates several needles at a time.

The spectral indices calculated from the 690, 905, and 1,550 nm wavelengths showed higher correlations with leaf EWT than the single wavelength data, which is in accordance with previous research (Junttila et al., 2016). The regression model of EWT and NDI\_p70690, 1,550 showed a coefficient of determination of 0.82 for all points, while NDI\_p70905, 1,550 was able to explain 89% of the changes in EWT. The better correlation of the combination of 1,550 and 905 nm with EWT can be explained by the sensitivity of the 905 nm wavelength to internal structure and dry matter content, a property needed in the estimation of EWT according to the literature (Ceccato et al., 2001). Additionally, the scanners providing measurements at 1,550 and 905 nm wavelengths were both developed by FARO and seem to employ similar algorithms for producing the discrete point clouds. It can be seen from the relationships between raw intensity and reflectance (**Figure 1**) that FARO scanners are more similar compared with the Leica, which could have affected our results.

Leaf points were used to develop linear regression models of the relationship between EWT and laser intensity features. The models showed that the most accurate prediction of EWT was achieved with the NDI\_p80905, 1,550 feature with an R <sup>2</sup> of 0.89. The accuracy of the prediction of EWT with segmented leaf points was the same as with all points but with a different feature. The best explanatory feature using all points was NDI\_p70690, 1,550. This suggests that there are a lot of "noise" points still present in the point clouds considering the estimation of EWT in the lowest intensity percentiles because a higher percentile of the intensity distribution best explained EWT using leaf points. An intensity threshold could be applied to reduce the amount of noise points from the needles, but applying such a threshold greatly affects the distribution of the intensity values. Thus, a robust method would be needed to apply a threshold to different scans without losing valuable information regarding the intensity distribution.

The point density in each point cloud representing a seedling was reduced using random sampling to investigate the effect of point density on the estimation of EWT. The results showed that a lower scanning resolution could be used because the spectral indices calculated from the sparser point clouds still explained EWT almost as well as with maximum point density. After reducing the number of points down to 500 per seedling, the amount of variation explained was only slightly reduced. A more severe reduction in the amount of variance explained was present after the number of points was reduced to 250 per seedling. The NDI\_p70905, 1550 index seemed to be more affected than NDI\_p70690, 1550 by the reduction in point density.

A multiple regression model was developed with a laser intensity feature and length as predictors to estimate EWT. The inclusion of the length of the seedling in the regression model increased the accuracy of the estimation of EWT, although it should be pointed out that the length of the seedling varied only from 16.5 to 42 cm. Presumably, the inclusion of the length of the seedling is able to explain a part of the variation in EWT due to physiological alterations that occur during the life span of the needles. During the growing season, the specific leaf area (i.e., dry matter content per leaf area) increases due to accumulation of dry matter resulting in a lower EWT (Jach and Ceulemans, 2000).

The measurements in this study were conducted in a greenhouse, where environmental conditions were similar between measurements. Temperature and ambient light conditions did change a little between measurements depending on whether it was a sunny or cloudy day, but otherwise no alterations in range or background of the seedlings were present. The change in temperature and light conditions is a limitation that needs to be taken into account considering the applicability of the method for the forest environment. Here, we also found that stem points responded to changes in EWT, which is likely due to the thin bark of the seedlings. Mature trees have thicker bark; thus, the estimation of EWT with multispectral TLS likely requires segmentation or classification of laser returns to leaf and stem points.

The terrestrial laser scanners had slightly different technical parameters: The FARO scanners had similar specifications except for the laser beam diameter at output, which was 2.25 and 3 mm for the X330 and S120, respectively. The different specifications resulted in a different laser footprint at target but it seems that having the same laser footprint size at target is not crucial for calculating spectral indices because the correlations were high despite this factor. However, an airborne laser scanning measurement would result in much larger laser footprints, ∼5–30 cm in diameter depending on the flying altitude, and a significantly lower point density. At these resolutions, the separation of woody and leaf material is practically impossible. The results obtained in this study show that the separation is not always necessary, but mature trees with large branches are likely to show a different response in terms of laser intensity.

The effect of incidence angle on spectral indices developed from multispectral laser scanning has been recently studied (Kaasalainen et al., 2016; Hancock et al., 2017). An NDI utilizing 1,063 and 1,545 nm wavelengths was concluded to not be sensitive to incidence angle effects when retrieving leaf moisture content. The combination of 905 and 1,550 nm wavelengths could be assumed to have similar properties due to the physical similarity of the wavelengths. Here, the needles had varying incidence angles and also varying receiving area because needles are not evenly spaced along the branches. Using an NDI calculated from the 905 and 1,550 nm wavelengths seems to reduce the effect of such confounding factors because using single wavelength data at 1,550 nm did not provide estimates of EWT as accurate as the spectral index did.

A commercially available multispectral laser scanner has been developed for airborne measurements, called Optech Titan (Teledyne Optech, Concord, Canada), which employs three wavelengths at 532, 1,064, and 1,550 nm. The combination of 1,064 and 1,550 nm wavelengths is interesting from the forest health mapping aspect, because these bands have been used before for EWT estimation (Gaulton et al., 2013). However, the different wavelengths of the Optech Titan have different footprints in terms of spatial distribution due to the instrument design (Morsy et al., 2016). The 1,064 nm channel is looking at nadir while the 1,550 nm channel is looking forward at 3.5◦ angle. This feature of the instrument presumably complicates the use of NDIs with the data, but the matter requires further investigation on the extent of the effect of varying spatial footprints at the canopy scale. A multispectral laser scanner for airborne measurements with overlapping footprints would be of great benefit in exploiting all the gains of the multispectral domain.

Based on the results, multisensor MS-TLS is capable of explaining most of the variation in leaf EWT of trees (R <sup>2</sup> = 0.91) in a controlled environment. The combination of 905 and 1,550 nm wavelengths explained variation in EWT better than the combination of 690 and 1,550 nm, which is also supported by previous research (Ceccato et al., 2001). The high coefficients of determination in the estimation of EWT with spectral indices did not seem to require a high point density like we had in the original scans, but the scanning could be conducted at a

#### REFERENCES


lower resolution. Single trees measured from the sample plots provide a base for mapping and monitoring forests. Considering the small scale of our study and the larger scale of information that is required for monitoring declining forests and trees, we suggest that in the next step, tree-level EWT estimation in forest environment should be investigated using multispectral laser scanning at the 905 and 1,550 nm wavelengths and by classifying leaf and stem points.

## AUTHOR CONTRIBUTIONS

SJ designed the experiment, conducted data processing and statistical analysis, and wrote the manuscript. JS contributed to the laboratory analysis and writing of the manuscript. MV assisted in designing the experiment, statistical analysis, and the writing of the manuscript. RL contributed to the design of the experiment, data collection, and commented on the manuscript. HK contributed to the design of the experiment, gave technical advice, and commented on the manuscript. AK assisted with the measurement set-up and provided technical support. MH, HH, and JH arranged all the required resources, supervised the experiment, and participated in the writing of the manuscript. All of the authors have contributed to the finalizing of the manuscript.

# FUNDING

This work was supported by the Academy of Finland project Centre of Excellence in Laser Scanning Research (CoE-LaSR) under Grant 272195, Unmanned Airborne Vehicle–based 4D Remote Sensing for Mapping Rain Forest Biodiversity and Its Change in Brazil (Decision number 273806), the Finnish Cultural Foundation under Grant 2CE87B6E, the Finnish Society of Forest Science, and the Niemi Foundation.

## ACKNOWLEDGMENTS

We would like to thank the Natural Resources Institute Finland (Luke) for providing greenhouse and laboratory facilities, and for assistance in maintaining the seedlings.


different scales. For. Ecol. Manage. 308, 76–89. doi: 10.1016/j.foreco.2013. 07.043


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Junttila, Sugano, Vastaranta, Linnakoski, Kaartinen, Kukko, Holopainen, Hyyppä and Hyyppä. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Quantitative Analysis of Cotton Canopy Size in Field Conditions Using a Consumer-Grade RGB-D Camera

Yu Jiang<sup>1</sup> , Changying Li <sup>1</sup> \*, Andrew H. Paterson2,3, Shangpeng Sun<sup>1</sup> , Rui Xu<sup>1</sup> and Jon Robertson<sup>2</sup>

<sup>1</sup> Bio-sensing and Instrumentation Laboratory, School of Electrical and Computer Engineering, College of Engineering, University of Georgia, Athens, GA, United States, <sup>2</sup> Plant Genome Mapping Laboratory, College of Agricultural and Environmental Sciences, University of Georgia, Athens, GA, United States, <sup>3</sup> Department of Genetics, Franklin College of Arts and Sciences, University of Georgia, Athens, GA, United States

Edited by:

Norbert Pfeifer, Technische Universität Wien, Austria

#### Reviewed by:

Maria Balota, Virginia Tech, United States Marta Silva Lopes, International Maize and Wheat Improvement Center, Mexico

> \*Correspondence: Changying Li cyli@uga.edu

#### Specialty section:

This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science

> Received: 29 September 2017 Accepted: 19 December 2017 Published: 30 January 2018

#### Citation:

Jiang Y, Li C, Paterson AH, Sun S, Xu R and Robertson J (2018) Quantitative Analysis of Cotton Canopy Size in Field Conditions Using a Consumer-Grade RGB-D Camera. Front. Plant Sci. 8:2233. doi: 10.3389/fpls.2017.02233 Plant canopy structure can strongly affect crop functions such as yield and stress tolerance, and canopy size is an important aspect of canopy structure. Manual assessment of canopy size is laborious and imprecise, and cannot measure multi-dimensional traits such as projected leaf area and canopy volume. Field-based high throughput phenotyping systems with imaging capabilities can rapidly acquire data about plants in field conditions, making it possible to quantify and monitor plant canopy development. The goal of this study was to develop a 3D imaging approach to quantitatively analyze cotton canopy development in field conditions. A cotton field was planted with 128 plots, including four genotypes of 32 plots each. The field was scanned by GPhenoVision (a customized field-based high throughput phenotyping system) to acquire color and depth images with GPS information in 2016 covering two growth stages: canopy development, and flowering and boll development. A data processing pipeline was developed, consisting of three steps: plot point cloud reconstruction, plant canopy segmentation, and trait extraction. Plot point clouds were reconstructed using color and depth images with GPS information. In colorized point clouds, vegetation was segmented from the background using an excess-green (ExG) color filter, and cotton canopies were further separated from weeds based on height, size, and position information. Static morphological traits were extracted on each day, including univariate traits (maximum and mean canopy height and width, projected canopy area, and concave and convex volumes) and a multivariate trait (cumulative height profile). Growth rates were calculated for univariate static traits, quantifying canopy growth and development. Linear regressions were performed between the traits and fiber yield to identify the best traits and measurement time for yield prediction. The results showed that fiber yield was correlated with static traits after the canopy development stage (R <sup>2</sup> = 0.35–0.71) and growth rates in early canopy development stages (R <sup>2</sup> = 0.29–0.52). Multi-dimensional traits (e.g., projected canopy area and volume) outperformed one-dimensional traits, and the multivariate trait (cumulative height profile) outperformed univariate traits. The proposed approach would be useful for identification of quantitative trait loci (QTLs) controlling canopy size in genetics/genomics studies or for fiber yield prediction in breeding programs and production environments.

Keywords: cotton, high-throughput, phenotyping, field, RGB-D, morphological

# 1. INTRODUCTION

Cotton (Gossypium) is one of the most important textile fibers in the world, accounting for about 25% of total world textile fiber use (USDA-ERS, 2017). Thus, improvement of cotton production is vital to fulfilling the fiber requirements of over nine billion people by 2050 (Reynolds and Langridge, 2016). Plant canopy structure is an important trait, affecting crop functions such as light-energy production and utilization (Norman and Campbell, 1989). Optimal canopy structure can improve plant photosynthesis and thus crop yield potential (Reta-Sánchez and Fowler, 2002; Stewart et al., 2003; Giunta et al., 2008). One key to increasing yield is to figure out the optimal canopy structure for maximizing plant photosynthesis (Murchie et al., 2009; Zhu et al., 2010). Canopy size is an important aspect of canopy structure and critical to plant photosynthesis, fruiting, and biomass accumulation. However, assessment of canopy size becomes a bottleneck, which limits breeding programs and genetics studies (White et al., 2012; Cobb et al., 2013; Araus and Cairns, 2014; Barabaschi et al., 2016), especially for large crop populations and high-dimensional traits (e.g., canopy volume). Accurate and high throughput techniques for quantifying canopy size would facilitate cotton (and other) breeding programs and genetics studies (Araus and Cairns, 2014; Pauli et al., 2016; Reynolds and Langridge, 2016).

Canopy size is spatially and temporally variable, and morphological traits describing canopy size can be grouped based on different criteria (Norman and Campbell, 1989). From the spatial perspective, component traits of canopy size can be categorized as one-dimensional and multi-dimensional traits. One-dimensional (1D) traits quantify canopy size in a single dimension (e.g., canopy height and width), whereas multi-dimensional traits quantify canopy size by considering multiple dimensions (e.g., canopy area in two-dimensional (2D) space and volume in three-dimensional (3D) space). From the temporal perspective, the traits can be separated into static and dynamic categories. Static traits are directly measured for canopies at a certain time, whereas dynamic traits (e.g., growth rates) are the change of static traits over time. In addition to spatial and temporal criteria, data dimensionality is an important consideration of morphological traits for canopy size quantification. Based on the number of variables, morphological traits can be classified as univariate (e.g., maximum and average canopy height) or multivariate (e.g., cumulative height profile describing heights at different percentiles).

Univariate traits are widely used due to their simplicity (easy to define) and availability (most measurement instruments provide a single reading). One-dimensional traits can be measured manually (or automatically) using distance meters. These 1D traits represent canopy size in only one dimension, creating a potential bias in structure assessment and comparison (Norman and Campbell, 1989). Leaf area as a 2D trait has been widely used for canopy size studies, and its derivative indicator (leaf area index, LAI) has demonstrated successes in estimating crop photosynthetic activities, biomass, yield, and biotic/abiotic stress tolerance (Sinclair and Horie, 1989; Kross et al., 2015; Feng et al., 2017). Leaf area can be directly measured using destructive methods, such as using a leaf area meter or image scanner to obtain the area of leaf samples that have been cut off the plant. Such destructive methods are usually arduous and hard to apply in large-scale experiments. To address those issues, LAI can be indirectly estimated using instruments such as the LI-COR LAI-2200 and Decagon LP-80 (Bréda, 2003; Weiss et al., 2004). Conventional 2D imaging methods can also be applied to calculate projected leaf area or canopy coverage ratio, estimating the true leaf area or LAI (Jonckheere et al., 2004; Zheng and Moskal, 2009). Nonetheless, all aforementioned methods require laborious data collection and do not provide 3D information, and therefore are inadequate to rapidly and comprehensively quantify canopy size and development.

Advanced 3D imaging approaches provide new opportunities to accurately quantify canopy size in multiple dimensions such as canopy height and width (1D), leaf area (2D), and volume (3D). The approaches can be categorized into passive and active 3D imaging (Li et al., 2014).

Passive 3D imaging methods reconstruct 3D structures of objects by expanding conventional 2D imaging methods. Stereo vision and the structure-from-motion (SfM) technique are two representative passive imaging methods. Small unmanned aerial systems (UASs) provide a means to quickly collect images for 3D reconstruction and trait extraction (e.g., crop height) using digital surface models and photogrammetry. However, images collected by UASs usually have lower quality (e.g., image resolution and sharpness) than images from ground systems, which could significantly affect the reconstruction accuracy (Shi et al., 2016) or even cause failures of 3D structure reconstruction (Jay et al., 2015). In addition, most of the passive techniques are computationally expensive (taking several hours for reconstruction of one plot) (Jay et al., 2015; Muller-Linow et al., 2015; Nguyen et al., 2016; Dong et al., 2017). Speeding up the process would require either using highperformance computing (HPC) resources, which would impose a considerable cost for large-scale breeding programs, or limiting the experimental scale and therefore breeding efficiency.

Active 3D imaging methods directly acquire 3D information by operating external light sources. Commonly used sensors include light detection and ranging (LiDAR), triangulation laser scanners, and time-of-flight (TOF) and structured light cameras. With suitable camera setup and illumination, most active methods can accurately obtain 3D point clouds of plants in field conditions for canopy size analysis, but the instruments are usually expensive (\$2,000–100k or more) compared with cameras used in passive methods (Li et al., 2014). Recent advances in consumer-grade RGB-D cameras (e.g., Microsoft Kinect and ASUS Xtion) provide an inexpensive solution for 3D scanning (Nock et al., 2013; Paulus et al., 2014; Andujar et al., 2016). In particular, the Microsoft Kinect v2 camera uses the TOF principle with upgraded color and depth resolution, creating the possibility for inexpensive and high-resolution 3D sensing in field conditions. Several previous studies explored the use of Kinect v2 camera in measuring canopy size (height and volume) in field conditions, finding that the Kinect v2 was a promising tool for field-based phenotyping (Andújar et al., 2016; Jiang et al., 2016; Andujar et al., 2017). However, two aspects need to be further improved: (1) data processing should be fully automated to improve throughput, and (2) more spatially (single and multidimensional traits) and temporally (static and dynamic traits) morphological traits need to be measured and studied.

The overall goal of the study was to develop a 3D imaging approach to automatically and quantitatively analyze canopy size of cotton plants in field conditions. Specific objectives were to (1) develop algorithms to reconstruct colorized point clouds of individual plots using depth and color images collected by Kinect v2 camera, (2) develop algorithms to segment canopy point clouds from the ground and weeds, (3) extract canopy size traits (canopy height, width, projected area, and volume) and their dynamic changes over time, and (4) explore the potential of using the extracted traits for fiber yield prediction.

# 2. MATERIALS AND METHODS

# 2.1. Experimental Design and Field Data Collection

A field (33.727239 N, 83.299097 W) was planted with cotton seed at the Iron Horse Farm of the University of Georgia in Watkinsville, Georgia, USA on 13 June 2016. The field contained 128 plots (16 rows with 8 plots per row) of length 3.05 m, with 1.83 m alleys between consecutive plots in a row, and row-spacing of 1.52 m (see Figure S1 in Supplementary Material). Three experimental genotypes (GA2011158, GA2009037, and GA2010074) and one commercial variety (Americot conventional) were used, each having 32 plot replicates. A completely randomized design was used to assign genotypes to individual plots. In each plot, 15 cotton seeds were manually planted at a spacing of 0.15 m. The first seed was planted 0.15 m away from the plot starting point, and hence a total of 15 seeds occupied 2.4 m out of 3.05 m in each plot. Cotton fiber was handpicked and weighed for 96 randomly selected plots (24 per genotype) on 4 November 2016 (which was 144 days after planting, DAP 144).

A field-based phenotyping system, GPhenoVision, was developed (Jiang et al., 2017) and used to scan the experimental field during midday (around 1,200–1,330 h) on 8 days in 2016 including 28 July (DAP 45), 4 August (DAP 52), 19 August (DAP 67), 26 August (DAP 74), 9 September (DAP 88), 16 September (DAP 95), 23 September (DAP 102), and 30 September (DAP 109). The scanning period covered two growth stages: canopy development (DAP 45–74), and flowering and boll development (DAP 74 to DAP 109). During the scanning period, the Kinect v2 camera of the GPhenoVision system was consistently used at 2.4 m above ground level (**Figure 1A**). The system ran in a continuous scanning mode at a constant speed of 1 m/s, but the operator manually controlled the data acquisition software to start/stop saving images at the beginning/end of each row to save storage space (**Figure 1B**).

# 2.2. Image Processing Pipeline

# 2.2.1. Reconstruction of Colorized Point Cloud

The entire image processing pipeline included three sections: point cloud reconstruction, cotton canopy segmentation, and trait extraction. The aim of the point cloud reconstruction section was to reconstruct colorized point clouds for individual plots using collected depth and color images with their corresponding GPS information (**Figure 2**). This section contained four steps:

#### *2.2.1.1. Step 1.1: data grouping*

All acquired GPS records were firstly converted from the geographic coordinate system to the universal transverse mercator (UTM) coordinate system. Altitude measures were used as the height of image acquisition. Collected depth and color images were segregated into individual plots based on their GPS coordinates, and the following processes were executed within each plot.

#### *2.2.1.2. Step 1.2: camera position adjustment*

Each plot used both global and local coordinate systems. The global (UTM and altitude) system indicated exact positions of image acquisition (also image center), whereas the local system represented image pixel coordinates in which the yaxis was aligned with the vehicle moving (row) direction and the x-axis was aligned with the direction that is perpendicular to the vehicle moving (across-row) direction. Global (UTM and altitude) coordinates were converted to local coordinates, so image acquisition positions were aligned with image pixel coordinates. First, global coordinates were rotated to be aligned with y-axis in the local system using Equation (1).

$$
\begin{bmatrix}
\boldsymbol{\mathcal{P}}\_{\boldsymbol{x}}^{i} \\
\boldsymbol{\mathcal{P}}\_{\boldsymbol{y}}^{i} \\
\boldsymbol{\mathcal{P}}\_{\boldsymbol{z}}^{i}
\end{bmatrix} = \begin{bmatrix}
\cos\theta\_{\boldsymbol{G}-L} & \sin\theta\_{\boldsymbol{G}-L} & \boldsymbol{0} \\
\boldsymbol{0} & \boldsymbol{0} & 1
\end{bmatrix} \begin{bmatrix}
\boldsymbol{\mathcal{P}}\_{\boldsymbol{e}ating}^{i} - \boldsymbol{\mathcal{P}}\_{\boldsymbol{e}ating}^{1} \\
\boldsymbol{\mathcal{P}}\_{\boldsymbol{n}orthing}^{i} - \boldsymbol{\mathcal{P}}\_{\boldsymbol{n}orthing}^{1} \\
\boldsymbol{\mathcal{P}}\_{\boldsymbol{al}itude}^{i} - \boldsymbol{\mathcal{P}}\_{\boldsymbol{al}itude}^{1}
\end{bmatrix} \tag{1}
$$

where θG−<sup>L</sup> was the rotation angle (the angle between the fitting curve and north axis) in radians, p i x , p i y , p i <sup>z</sup>were the x, y, z coordinates of the acquisition position of the ith frame in the local system, and p i easting , p i northing , and p i altitude were the UTM coordinates and altitude values, respectively, of the acquisition position of the ith frame in the global system. p <sup>1</sup> was the starting (first) frame acquired in each plot.

After the coordinate system conversion, image acquisition positions were aligned with image pixel coordinates and the

starting point of each plot became the origin point in the local system. Individual depth and color images were reconstructed to colorized point clouds using functions provided by the manufacturer's software development kit (SDK).

#### *2.2.1.3. Step 1.3: camera orientation adjustment*

The Kinect v2 camera might be slightly tilted during data collection due to uneven terrain, and thus it was necessary to estimate camera orientation in each frame and to correct point cloud offset due to camera orientation changes. As individual frames would be eventually stitched, only a part of the frame, the region of interest (ROI), needed to be processed. To save processing time, camera orientation estimation and adjustment were performed on the ROI of each point cloud. ROIs were selected based on image acquisition positions using Equation (2).

$$
\begin{bmatrix}
ROI\_X^{Lower} \ ROI\_X^{Upper} \\
ROI\_Y^{Lower} \ ROI\_Y^{Upper} \\
ROI\_Z^{Lower} \ ROI\_Z^{Upper}
\end{bmatrix} = \begin{bmatrix}
0 & H\_{camera}
\end{bmatrix} \tag{2}
$$

where ROILower · and ROIUpper · were the lower and upper limits on a specific axis of an ROI in the point cloud of a single frame, Wenclosure was the width of the enclosure (1.52 m in the present study), Dist(i, j) was the absolute distance difference in acquisition position between the ith and jth frames, and Hcamera was the height of the camera above the ground level (2.4 m in the present study).

In each selected ROI, ground surface was detected using maximum likelihood estimation sample consensus (MLESAC) (Torr and Zisserman, 2000), and the normal of the detected ground surface was calculated. Angle differences in the normal between the detected and ideal (X-Y plane) ground surfaces were calculated, and points in the ROI were rotated based on the angle differences with respect to the X-Z and Y-Z planes using Equation (3).

$$
\begin{bmatrix} x\_{adj} \\ y\_{adj} \\ z\_{adj} \end{bmatrix} = \begin{bmatrix} \cos(\theta\_{xx}) & 0 & \sin(\theta\_{xx}) \\ 0 & 1 & 0 \\ -\sin(\theta\_{xx}) & 0 & \cos(\theta\_{xx}) \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & \cos(\theta\_{yx}) & \sin(\theta\_{yz}) \\ 0 & -\sin(\theta\_{yz}) & \cos(\theta\_{yz}) \end{bmatrix} \begin{bmatrix} x\_o \\ y\_o \\ z\_o \end{bmatrix} \tag{3}
$$

where θxz and θyz were the angles between the detected and ideal ground surfaces with respect to the X-Z and Y-Z planes, and xadj, yadj, and zadj were the adjusted coordinates of the original points (xo, yo, and zo) in the ROI.

If vegetation covered the entire ground surface and resulted in a failure of ground detection, the ground normal of the ROIs was assumed to be the same as the X-Y plane normal (nEX−<sup>Y</sup> = [0, 0, 1]). As a consequence, there was no adjustment to account for effects caused by camera orientation changes.

#### *2.2.1.4. Step 1.4: stitching of selected ROIs*

The final step in the reconstruction section was to stitch all of the selected ROIs into an integrated colorized point cloud for individual plots. Based on camera positions, the transformation of points in ROIs to the local coordinate system would naturally result in stitching of ROIs and produce the colorized point cloud for a plot. The coordinate transformation was conducted using Equation (4).

$$
\begin{bmatrix} \mathbf{x} \\ \mathbf{y} \\ \mathbf{z} \end{bmatrix} = \begin{bmatrix} \mathbf{x}\_{adj} \\ \mathbf{y}\_{adj} \\ \mathbf{z}\_{adj} \end{bmatrix} + \begin{bmatrix} p\_x^i - p\_x^1 \\ p\_y^i - p\_y^1 \\ p\_z^i - p\_z^1 \end{bmatrix} \tag{4}
$$

where x, y, and z were the coordinates of points in the final colorized point cloud of a plot, and p i x , p i y , and p i <sup>z</sup> were the coordinates of the acquisition position of the ith frame in the local coordinate system.

#### 2.2.2. Segmentation of Cotton Plant Canopies

Point clouds of cotton plots could contain irrelevant objects such as system frames, ground surface, and weeds. Therefore,

it was necessary to segment the cotton plant canopy from irrelevant objects (Cotton canopy segmentation in **Figure 3**). The segmentation of cotton canopies involved two steps:

#### *2.2.2.1. Step 2.1: vegetation segmentation*

Unlike point cloud data collected by conventional instruments such as LiDAR, point cloud data in the present study also contained color information (RGB values) for each point. A color filter was used to segment vegetation from background objects. According to previous studies (Hamuda et al., 2016), excess green (ExG) index was an effective indicator to obtain vegetation objects in an image, and thus ExG values were calculated for each point of a plot. A preliminary test was performed on a small subset of the collected images, showing that a threshold of 0.15 provided adequate separation between vegetation and background.

#### *2.2.2.2. Step 2.2: cotton canopy segmentation*

Color filtering was able to remove most irrelevant objects but not weeds, because plants and weeds are both vegetation and thus hard to differentiate based on color information. In this step, point clouds were rasterized to depth images, so pixels of cotton canopies were differentiated from those of weeds in the depth images by using 2D computer vision algorithms with spatial information including height (depth), area, and position. Identified canopy pixels were back-projected to 3D space to select points representing cotton canopies in point clouds. Point cloud data (x, y, and z coordinates) were rasterized to depth images using Equation (5).

$$\begin{aligned} I\_D(i, j) &= \max(z\_k), \\ \forall k &\in \{k | \mathbf{x}\_{Lower} + \mathbf{S}\_{\mathcal{S}} \times (i - 1) \le \mathbf{x}\_k \le \mathbf{x}\_{Lower} + \mathbf{S}\_{\mathcal{S}} \times i\} \\ \cap \{k | \mathbf{y}\_{Lower} + \mathbf{S}\_{\mathcal{S}} \times (j - 1) \le \mathbf{y}\_k \le \mathbf{y}\_{Lower} + \mathbf{S}\_{\mathcal{S}} \times j\} \\ i &= 1, 2, \dots, \lceil \frac{Range(\mathbf{x})}{\mathbf{S}\_{\mathcal{S}}} \rceil \\ j &= 1, 2, \dots, \lceil \frac{Range(\mathbf{y})}{\mathbf{S}\_{\mathcal{S}}} \rceil \end{aligned} \tag{5}$$

where I<sup>D</sup> represented a rasterized depth image, and i and j were pixel indices in ID. xLower and yLower were the lower limits of vegetation point clouds, and S<sup>g</sup> was the grid size (0.05 m in the present study). x, y, and z were coordinates of vegetation points, and k was the index of a point in vegetation point clouds. Range(·) was a function to calculate the maximum length along a particular axis in vegetation point clouds, and ⌈ Range(·) Sg ⌉ was the total number of grid cells along a particular axis.

Most in-row weeds were short and small, and removed using height and area information. Depth images were binarized by the 30th percentile of depth value of all pixels in order to exclude those representing short weeds. Connected components (CCs) were labeled in binary depth images, and small CCs representing weeds were removed using Equation (6). Thresholds of depth and area were empirical values based on observations of height and area of weeds in the present study.

$$CC\_i^{incluled} = \begin{cases} 1, & \text{Area(CC}\_i) > = Th\_{Area} \\ 0, & \text{otherwise} \end{cases}, \ i = 1, 2, \dots, n \quad (6)$$

where CCincluded is a flag for a connected component, and "1" or "0" indicated inclusion/exclusion of the connected component in the binary depth image. Area(·) was a function to count the number of pixels in a connected component, and ThArea was the threshold (15 pixels in the present study) for including a CC.

Weeds between plots were tall and large, and removed using position information. The CC with the largest area (pixel counts) was selected as the main canopy in a plot, and its bounding box was calculated. Based on their positions relative to the main canopy, remaining CCs were classified as weed or cotton canopy using Equation (7).

$$\text{CC}\_{i}^{\text{class}} = \begin{cases} 1, & \text{Dist(CCi, CCm)} \leqslant W\_{\text{CC}\_{m}^{b}} \text{ }, \,\forall i \in \{i|\text{CC}\_{i}^{\text{includcd}} = 1\} \\ 0, & \text{otherwise} \end{cases} \tag{7}$$

where CCclass is the class marker for a CC with "1" for cotton canopy and "0" for weeds. Dist was a function to calculate the distance between the center position of the ith CC and main canopy CC (CCm), and WCC<sup>b</sup> m was the width of the bounding box for main canopy CC.

According to the rasterization process, a pixel in depth images represented a grid cell in vegetation point clouds. Based on Equation (5), each pixel in the identified canopy CCs was backprojected to a grid cell, and vegetation points in that cell needed to be included in canopy point clouds. Vegetation points were selected using Equation (8) to form canopy point clouds that were used for trait extraction in each plot.

$$PtClould\_{canopy} = \{ \forall pt | pt \in CC\_i \}, \ i = \{ j | CC\_j^{class} = 1 \} \tag{8}$$

where PtCloudcanopy represented point clouds of cotton canopy, pt was a point in vegetation point clouds, CC<sup>i</sup> was the ith identified canopy CC, and CCclass <sup>j</sup> was the class marker of the jth CC in a depth image.

#### 2.2.3. Extraction of Morphological Traits

Morphological traits were extracted from point clouds of cotton plant canopies (Trait extraction in **Figure 3**). Trait extraction included two parts: extraction of static traits from multiple dimensions (one- and multi-dimensional traits) and calculation of dynamic traits (growth rates).

#### *2.2.3.1. Step 3.1: extraction of one-dimensional traits*

One-dimensional traits contained canopy height (maximum and mean), cumulative height profile, and width (maximum and mean) at the plot level. The maximum and mean canopy heights were defined as the tallest and average height values of all canopy

points. Cumulative height profile was the combination of canopy height from the 5th percentile to 95th percentile with an interval of 5 %. For canopy width, the point cloud of a cotton canopy was segregated into ten segments along the row direction, and the maximum length across the row direction was calculated in each segment. Maximum and mean widths were the maximum and average values of widths in the ten segments.

#### *2.2.3.2. Step 3.2: extraction of multi-dimensional traits*

Multi-dimensional traits contained projected canopy area and canopy volume. All canopy points were projected onto the X-Y plane, and the boundary of the projected shape was identified to calculate the projected canopy area. Convex and concave hulls were detected on canopy point clouds, and canopy volume was estimated using the detected convex and concave hulls.

#### *2.2.3.3. Step 3.3: calculation of growth rate*

In addition to static traits on a specific date, growth rates (dynamic changes) could provide information about growth behavior for cotton plants. Growth rate was defined and calculated between every two consecutive data collection dates using Equation (9).

$$G\_{T,P} = \frac{T\_{d\_{last}} - T\_{d\_{f\text{fstr}}}}{d\_{last} - d\_{f\text{fstr}}} \tag{9}$$

where G was the growth rate of a trait (T) during a period (P). dfirst and dlast were the first and last days after planting in the period (P).

#### 2.3. Performance Evaluation

#### 2.3.1. Accuracy of Canopy Segmentation

Cotton canopy segmentation strongly affected the accuracy of trait extraction. In particular, weed removal would significantly influence values of extracted morphological traits. Ground truth segmentation results of depth images were manually generated, and accuracy, false positive rate (FPR; weed pixels identified as canopy), and false negative rate (FNR; canopy pixels identified as weeds) were calculated for each plot. Based on these three values, segmentation results were evaluated and classified into three categories: clean canopy (accuracy > 95%), under-removal of weed (FPR > 10%), and over-removal of canopy (FNR > 10%). The percentages of the three categories were used as indicators to evaluate the performance of canopy segmentation.

#### 2.3.2. Accuracy of Sensor Measurements

It was important to evaluate the accuracy of depth measurement because depth information was not only used for height measurement but also calculation of x and y coordinates of individual points, thereby affecting the overall accuracy of obtained point clouds. As height was directly derived from depth measurement, maximum canopy height was measured for 32 field plots that were randomly selected on each of four scanning dates (DAP 45, 52, 74, and 88). Therefore, a total of 128 data points were obtained for evaluating the accuracy of height measurement.

In addition, due to difficulties in field measurement, potted and artificial plants were used to assess accuracies of measuring other traits. A total of 8 potted plants were used for validating measurements of width, length, and volume, and an artificial plant was used for projected leaf/convex area (see Figure S2). For the potted plants, canopy width (cross-movement direction) and length (movement direction) were measured using a ruler, whereas volume was measured using a protocol for tree volume estimation (Coder, 2000) (see Figure S3). Following the protocol, a plant was virtually segregated into layers with a height interval of 5 cm. Diameters were measured for the middle of individual layers, from which volume could be estimated using a cylinder model, and the plant volume was the summation of all layer volumes. For the artificial plant, a total of 8 layouts were configured to form different plant leaf/convex areas (see Figure S4). In each layout, all leaves were laid on the base surface and imaged with a size marker by a digital single-lens reflex (DSLR) camera (A10, Fujifilm Holdings Corporation, Tokyo, Japan) so that projected leaf/convex area could be accurately measured using image processing. After taking color images, leaves were vertically lifted at various heights to form a 3D layout simulating real plants. It was noteworthy that real plants had many more leaves than the artificial plant, resulting in a denser canopy, and thus projected convex area would be closer to projected canopy area in real situations. In total, 8 data points were used for validating sensor measurements of canopy width, length, projected area, and volume.

Simple linear regression analyses were performed between sensor and manual measurements for all traits. Adjusted coefficient of determination (Adjusted R 2 ) and root mean squared error (RMSE) were used as indicators for performance assessment.

#### 2.3.3. Efficiency of Image Processing

In addition to accuracy performance, the efficiency of the proposed algorithm was tested on a workstation computer that used an Intel i7-4770 CPU with 16 GB of RAM on a Windows 10 operating system. Processing time of point cloud reconstruction, cotton canopy segmentation, and trait extraction was recorded during the processing of all plot data collected on all eight data collection dates. For point cloud reconstruction, simple linear regression analysis was conducted between the number of frames in each plot and the reconstruction time. For cotton canopy segmentation and trait extraction, the percentage of processing time for each key step was calculated.

#### 2.4. Statistical Analyses between Fiber Yield and Extracted Traits

Extracted traits were grouped into two categories: univariate and multivariate traits. For univariate traits, simple linear regression analyses were conducted between the traits and fiber yield, whereas for multivariate traits, multiple linear regression analyses were conducted between the traits and fiber yield. The adjusted R 2 and RMSE were used to assess the potential of the usefulness of extracted traits for prediction of cotton fiber yield. As regression models established using the maximum and mean canopy height were nested to those using cumulative height profile, F-tests were



conducted to rigorously test the statistical significance of model differences. All regression and F-test analyses were performed in R software (R Core Team, 2016).

# 3. RESULTS

# 3.1. Performance of Segmentation of Cotton Canopy

Overall, the cotton canopy segmentation achieved an accuracy of 98.6% on all collected data, which indicates that the scanning system provided accurate canopy point clouds for trait extraction throughout the growing season (**Table 1**). The proposed algorithms precisely segmented cotton canopy under various plot conditions (**Figures 4A–D**). In-row weeds were usually shorter and smaller than cotton plants, so they were mostly removed by height and area filters. In contrast, weeds between rows were generally large and tall, but they could be removed using position information because they were located between plots and away from the main canopy in each plot. Therefore, the method could provide accurate segmentation of cotton plant canopy. However, the weed under-removal rate increased noticeably on DAP 67 and DAP 74. This was because no weeding activity was arranged during that period, and many weeds grew substantially and became comparable with cotton plants in height and size; as a consequence, cotton plants became hard to differentiate from weeds (**Figure 4E**). In late growth stages (e.g., flowering and boll development stage), cotton plant leaves started to shrink and fall down, resulting in a reduction of canopy size and overlap. Large canopies were segregated into small leaf areas, especially along the outer part of canopies. These small leaf areas were largely filtered out by the area filter, leading to over-removal of canopy (canopy pixels identified as weeds) (**Figure 4F**).

# 3.2. Representative Colorized Point Clouds

Representative colorized point clouds of cotton canopies were generated and demonstrated at the plot level over the growing season (**Figure 5**). Poorly germinated plots contained some empty areas (**Figure 5A**), which tended to be at least partly filled by neighboring plants after the canopy development stage (DAP 74). In well germinated plots, seeds generally sprouted around the same time and seedlings grew at comparable speeds, maintaining similar canopy height along a plot. Even in cases with some variation of germination date (thus development speed), canopy height tended to reach a similar level in a plot, avoiding the occurrence of extremely tall or short sections (**Figure 5B**). Well germinated plots showed canopy overlap at an earlier date (approximately DAP 52 or earlier) than poorly germinated ones and were relatively flat (or slightly arched) along the row direction, whereas in poorly germinated plots the canopy close to empty areas was short. This slightly arched shape of the canopy is due to "edge effect," and plants can be either bigger or smaller at the plot edge than those in the center.

# 3.3. Accuracy of Sensor Measurement

Sensor measurements were strongly correlated (adjusted R 2 > 0.87) with manual measurements for all extracted traits (**Figure 6**). In particular, height measurements were obtained from multiple days and the RMSE was 0.04 m, suggesting a high accuracy and repeatability of depth measurement (and therefore point clouds) in field conditions. Consequently, point clouds acquired by the Kinect v2 sensor could be used to accurately measure traits such as width, length, projected leaf/convex area. It should be noted that although correlation was strong (adjusted R <sup>2</sup> > 0.87) between sensor and manual measurements for volume, the absolute values were significantly different (see the slope in regression equations for convex and concave hull volumes). Convex and concave hull volumes were smaller than reference measurements. In the present study, the Kinect v2 sensor acquired point clouds from the top view, and thus only canopy surface could be imaged, resulting in a lack of information from two sides (especially sections under the canopy). As a consequence, the volume of a plant (or a plot) estimated by convex and concave hulls was a portion of the ground truth value. Since the canopy of most cotton plants was roughly a cone, the sensed portion could generally represent the entire plant. This was also supported by the high correlation between sensor and reference measurements. In addition, convex hull volume showed better correlation with reference measurement than concave hull volume, because both convex hull and reference measurements included volume of empty space among branches of a plant. Nonetheless, as both convex and concave hull demonstrated strong correlation, they could be used as indicators for growth pattern analyses and/or yield prediction.

### 3.4. Efficiency of the Proposed Approach

The proposed approach used on average 215 s for processing a plot, including 184 s for point cloud reconstruction and 31 s for canopy segmentation and trait extraction (**Figure 7**). Variations in reconstruction time primarily came from the different number of frames acquired for a plot (**Figure 7A**). Although the GPhenoVision system was set at a constant speed, the actual system speed would vary due to different terrain conditions (dry vs. wet) and scanning status slower at the two ends of a row than in the middle). Consequently, the number of frames could be different in various plots on different dates. Generally, the reconstruction time increased linearly with the number of frames acquired in a plot (**Figure 7B**). However, for plots with equal number of frames,

FIGURE 4 | Representative results of cotton canopy segmentation: successful segmentation under (A) poorly germinated plot with weeds, (B) well germinated plot with weeds, (C) plot with a segregated plant, and (D) well germinated plot with connected weeds; (E) failure of weed removal; and (F) over-removal of cotton plant canopy. In classified connected component images, blue, green, and red colors indicated main canopy, cotton canopy, and weeds, respectively. Ground truth images were manually generated for including cotton canopy.

variations in processing time were mainly due to different number of points in frames. For instance, point clouds of poorly germinated plots contained more points of ground than those of well germinated plots, resulting in variation in ground surface detection and thus in reconstruction time.

For canopy segmentation and trait extraction, efficiency variations occurred due to different numbers of points in point

cloud data (**Figure 7C**). The number of points in a point cloud was determined by the canopy size, which was affected by both germination condition and growth stage. Plant canopies were larger in well germinated plots or plots in the canopy development stage than those in poorly germinated plots or plots in the flowering and boll development stage. Thus, the number of canopy points varied from plot to plot and stage to stage, resulting in processing time variations. Among the three operations, canopy segmentation and trait extraction were more time consuming than vegetation segmentation (**Figure 7D**). Vegetation segmentation was based on color filtering, in which all points were processed simultaneously. In contrast, canopy segmentation and trait extraction processed information point by point, thus using a significantly longer time than vegetation segmentation.

#### 3.5. Growth Trend of Cotton Plants 3.5.1. Static Traits

Overall, plants elongated substantially from DAP 45 to DAP 74 (canopy development stage) and expanded considerably from

DAP 45 to DAP 88 (canopy development stage and early flowering stage) (**Figure 8**). The values of canopy morphological traits gradually decreased during the rest of the growing season (flowering and boll development stages). Among the genotypes, the Americot conventional showed the shortest and smallest canopy, compared with the other three experimental genotypes. Genotype GA2009037 was the tallest genotype but with the least width, indicating a tall and narrow canopy (statistical comparison results are provided in Tables S1–S7).

Cumulative height profiles of all four genotypes showed logarithmic growth during the canopy development stage (DAP 45 to DAP 74) and linear growth during the flowering and boll development stage (DAP 74 to DAP 109) (**Figure 9**). In the canopy development stage, plant leaves increase in transpiration capability and maintain a solid shape to receive more sunlight for photosynthesis. Therefore, at the canopy development stage, upper leaves occluded lower ones, resulting in less variation in canopy height after the 25th percentile. The large difference between the 5th and 25th percentiles was due to un-occluded leaves on expanded branches at lower positions along the canopy border. When transitioning into flowering and boll development stage, plant leaves receive fewer nutrients (due to nutrient demands of maturing fruit) and start to shrink. Previously occluded leaves could be captured, and thus canopy height was

represented by leaves at a wider range of positions, resulting in an increase of height difference at various percentiles. This information might be useful for predicting the change of growth stages for cotton plants.

#### 3.5.2. Dynamic Traits

The four genotypes showed different patterns of canopy development in different growth stages (**Figure 10**). Generally, growth rates of all four genotypes approached zero in P5 (DAP 74 to DAP 88, which was the first data collection period in the flower and boll development stage) for maximum and mean canopy height, and in P6 (DAP 88 to DAP 95, which was the second data collection period in the flower and boll development stage) for the other traits, because the four genotypes were short season cottons and had similar growth stage transitions. In addition, all genotypes stopped canopy elongation approximately a week earlier than canopy expansion. Although the overall growth trend was similar, there were differences in growth behavior among the four genotypes (statistical comparison results are provided in Tables S8–S14). The Americot conventional was generally the slowest in developing its canopy, whereas the three experimental genotypes grew rapidly in different dimensions. Genotype GA2011158 showed the fastest growth in all dimensions, resulting in large canopy height, width, projected area, and volume. Genotype GA2009037 primarily elongated rather than expanded, resulting in tall canopies with the least projection coverage, whereas Genotype GA2010074 mainly expanded rather than elongated, resulting in short

canopies with large coverage and volume. Detection of such variations might permit identifying genes controlling cotton plant growth patterns and/or selecting genotypes suitable for different production or harvesting strategies.

#### 3.6. Performance of Yield Prediction

In general, static traits had some value (R <sup>2</sup> = 0.12–0.71) for predicting cotton fiber yield (**Figure 11A**). Among univariate traits, multi-dimensional traits (e.g., projected canopy area and volume) considerably outperformed one-dimensional traits such as canopy height and width, presumably because multidimensional traits could depict canopy size more completely. For instance, genotypes GA2011158 and GA2009037 had similar canopy height but different fiber yield, resulting in a low correlation between canopy height and fiber yield. However, the two genotypes had obvious differences in projected canopy area and canopy volume, with improved correlation between the morphological traits and fiber yield. This indicated the usefulness of extracting multi-dimensional traits using advanced 3D imaging techniques. Compared with univariate traits (the maximum and mean canopy height), multivariate traits (cumulative height profile) showed a significant improvement in yield prediction on most days (see detailed F-test results in Table S15). This was because cumulative height profiles intrinsically incorporated spatial distribution: high percentile ranks represented height information in the middle of the canopy, whereas low percentile ranks represented the borders. The capability of extracting multivariate traits was an advantage of using 3D imaging modalities. Most static traits achieved the highest correlation with fiber yield after the canopy development stage (DAP 67), and then the correlation decreased slightly during the rest of the growing season. In the growing season that was studied, the best period to use morphological traits for yield prediction was the transition period between canopy

development and flowering and boll development stages. Canopy size during the transition period reached the maximum size and reasonably represented the potential of flower and boll development. However, mean canopy width and projected canopy area achieved the highest correlation at later dates (DAP 95 and DAP 102) because all genotypes kept expanding until DAP 95 and most of the expansion was due to the development of sympodial (fruiting) branches.

Compared with static traits, dynamic traits (growth rates) demonstrated less capability for yield prediction, but multidimensional traits that could comprehensively quantify canopy size still outperformed one-dimensional traits (**Figure 11B**). This further supported the advantage of using 3D imaging for measuring morphological traits. In contrast to static traits, growth rates were more informative for yield prediction in early stages (early periods in canopy development stage) rather than late stages. This may have been because all four genotypes were short season cotton and had similar overall growth trends. In early stages, growth rates were positive, reflecting canopy development, and were well correlated with the final vegetative structure and productive capacity. However, in late stages, growth rates were negative, reflecting leaf and

FIGURE 11 | Coefficients of determination (R 2 ) of linear regressions between extracted traits and cotton fiber yield: (A) Static traits and fiber yield and (B) dynamic traits and fiber yield. Black, white, and red colors indicated statistical significance of "<0.001," "<0.01," and "<0.05," respectively. Insignificant R 2 values were not shown in blocks. P1 was the period from the day of planting to DAP 45, and P2–P8 were the periods between two consecutive data collection dates. Superscripts of R 2 values for height profile denoted the F-test results between regression models: a number sign (#) indicated a significant model improvement by using height profile rather than maximum height, and an asterisk (\*) indicated a significant model improvement by using height profile rather than both maximum and mean heights (see Table S15 for detailed F-test results). In addition, Pearson's correlation coefficients were calculated for all univariate traits (see Figure S5).

plant senescence, and were not well correlated with fiber yield.

# 4. DISCUSSION

Consumer-grade RGB-D image-based measurement of plant canopies in field conditions could lower the cost and increase the use of imaging techniques in phenotyping, benefitting the plant science community. The entire system used in the present study cost \$70,500, including an RGB-D camera (\$200), high-clearance tractor platform (\$60,000), RTK-GPS (\$8,000), power unit (\$300), and rugged laptop (\$2,000). This is still a considerable investment, but can be reduced. The tractor platform can be replaced with a low-cost pushcart (usually less than \$2,000) if experiments are less than a hectare (Bai et al., 2016). In addition, a pushcart system allows for "stop-measure-go" mode, in which image acquisition locations can be predefined and manually controlled, and thus RTK-GPS is not required. However, these low-cost solutions decrease the efficiency and

throughput of data collection. The consumer-grade RGB-D camera is an inexpensive 3D sensing solution for plant phenotyping, but building a phenotyping system requires comprehensive consideration of balance between research budget, experimental scale, and data collection and processing throughput.

The processing algorithms reported here can accurately extract morphological traits of plant canopies at the plot level, providing useful information for genomic studies and/or breeding programs. However, we note two limitations of the proposed approach. First, the Kinect v2 camera cannot be directly used under strong illumination (e.g., during midday in the field), so a mechanical structure is required to provide shade. Second, the reconstruction step currently relies on functions provided by Kinect v2 SDK, and must be performed on a computer running on an operating system of Windows 8 (or later) with connection to the Kinect v2 camera. This may significantly decrease the post-processing throughput, because cloud computing services usually run on remote Unix/Linux systems and users cannot connect hardware components to them while processing. This limitation could be addressed with third party libraries or userperformed registration between depth and color images (Kim et al., 2015). We acknowledge that the developed method was tested in a 1-year experiment, and altered conditions may change results. For example, predictors of yield were most effective near the end of canopy development in this experiment—however, dramatic differences in growing conditions (such as drought or cold) could change that in other years. Additional experiments might also consider different degrees of replication—here, a total of 32 replicates per genotype were used, which is considerably more than the number that is generally used in genetics/genomics studies. Variability among these samples provides a basis for estimating the minimal number of samples that might be used to discern differences between treatments of pre-determined magnitude, which is important in experiments involving large numbers of genotypes such as breeding programs. Based on theoretical calculation (Cochran and Cox, 1992), most static traits required three replicates, whereas most dynamic traits needed more than ten replicates (see Figure S6). This is because the four genotypes used in this study are very similar in growth patterns, requiring a high number of replicates to increase the statistical power for genotype differentiation. However, the minimal number of replicates could be reduced for growth rates in experiments involving genotypes with distinctive growth patterns (e.g., wild and elite cotton germplasm lines).

Two important findings were observed for the extracted traits. Firstly, static and dynamic traits showed the highest correlation with fiber yield in different periods; dynamic traits were informative in early canopy development stages, whereas most static traits were useful in the transition period between canopy development and flower and boll development. Canopy size (canopy height, width, projected area, and volume) remained relatively constant near its peak for a period after canopy development, indicating to some extent the capability to develop flowers and bolls. However, growth rates were negative in late stages, reflecting plant senescence as resources were redirected to the maturing bolls. Secondly, multivariate traits consistently showed better yield prediction than univariate traits, presumably because the multivariate traits intrinsically incorporated information such as the spatial variation of canopy height. Several previous studies reported methods to find the best percentile of canopy height to use to increase the accuracy of yield prediction (Friedli et al., 2016; Weiss and Baret, 2017). However, according to our results, it may be more efficient to explore the use of multivariate traits to predict yield. It is noteworthy that environment is an important factor, and analysis of these parameters in different growing seasons with various environments may yield different results. These differences may help to better explain interactions between genotype and environment (G×E), and make breeding programs more effective.

# 5. CONCLUSIONS

The 3D imaging system and data processing algorithms described here provided an inexpensive solution to accurately quantify cotton canopy size in field conditions. The extracted morphological traits showed potential for yield prediction. Multidimensional traits (e.g., projected canopy area and volume) and multivariate traits (e.g., cumulative height profile) were better yield predictors than traditional univariate traits, confirming the advantage of using 3D imaging modalities. Early canopy development stages were the best period in which to use dynamic traits for yield prediction, whereas late canopy development stages were the best period in which to use static traits. Future studies will be focused on improving the data processing throughput (via methods such as parallel computing) and extracting new multivariate traits for canopy size quantification.

# AUTHOR CONTRIBUTIONS

YJ, CL, and AP conceived the main concept of the study; YJ, JR, SS, RX, AP, and CL designed the experiment and performed field data collection; YJ developed the data processing pipeline; YJ, CL, AP, SS, and RX analyzed the data; AP and CL provided plant materials, instruments, and computing resources used in the study; YJ, CL, and AP contributed to writing the manuscript. All authors reviewed the manuscript.

# FUNDING

This study was funded jointly by the Agricultural Sensing and Robotics Initiative of the College of Engineering, and the College of Agricultural and Environmental Sciences of the University of Georgia. The project was also partially supported by the National Robotics Initiative (NIFA grant No: 2017-67021-25928) and Georgia Cotton Commission.

# ACKNOWLEDGMENTS

The authors gratefully thank Dr. Peng Chee (University of Georgia) for providing plant materials and insights to the present study; and Mr. Zikai Wei, Mr. Jamal Hunter, Mr. Aaron Patrick, Mr. Jeevan Adhikari, Ms. Mengyun Zhang, Mr. Shuxiang Fan, and Dr. Tariq Shehzad for their assistance in planting, field management, data collection, and harvesting.

#### REFERENCES


#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpls.2017. 02233/full#supplementary-material

Meeting (American Society of Agricultural and Biological Engineers), 1 (Spokane, WA).


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Jiang, Li, Paterson, Sun, Xu and Robertson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Leaf Movements of Indoor Plants Monitored by Terrestrial LiDAR

Mónica Herrero-Huerta1,2,3 \*, Roderik Lindenbergh<sup>1</sup> and Wolfgang Gard<sup>4</sup>

<sup>1</sup> Department of Geoscience and Remote Sensing, Delft University of Technology, Delft, Netherlands, <sup>2</sup> TIDOP Research Group, Higher Polytechnic School of Avila, University of Salamanca, Avila, Spain, <sup>3</sup> Agronomy Department, Purdue University, West Lafayette, IN, United States, <sup>4</sup> Department of Structural and Building Engineering, Delft University of Technology, Delft, Netherlands

Plant leaf movement is induced by some combination of different external and internal stimuli. Detailed geometric characterization of such movement is expected to improve understanding of these mechanisms. A metric high-quality, non-invasive and innovative sensor system to analyze plant movement is Terrestrial LiDAR (TLiDAR). This technique has an active sensor and is, therefore, independent of light conditions, able to obtain accurate high spatial and temporal resolution point clouds. In this study, a movement parameterization approach of leaf plants based on TLiDAR is introduced. For this purpose, two Calathea roseopicta plants were scanned in an indoor environment during 2 full-days, 1 day in natural light conditions and the other in darkness. The methodology to estimate leaf movement is based on segmenting individual leaves using an octree-based 3D-grid and monitoring the changes in their orientation by Principal Component Analysis. Additionally, canopy variations of the plant as a whole were characterized by a convex-hull approach. As a result, 9 leaves in plant 1 and 11 leaves in plant 2 were automatically detected with a global accuracy of 93.57 and 87.34%, respectively, compared to a manual detection. Regarding plant 1, in natural light conditions, the displacement average of the leaves between 7.00 a.m. and 12.30 p.m. was 3.67 cm as estimated using so-called deviation maps. The maximum displacement was 7.92 cm. In addition, the orientation changes of each leaf within a day were analyzed. The maximum variation in the vertical angle was 69.6◦ from 12.30 to 6.00 p.m. In darkness, the displacements were smaller and showed a different orientation pattern. The canopy volume of plant 1 changed more in the morning (4.42 dm<sup>3</sup> ) than in the afternoon (2.57 dm<sup>3</sup> ). The results of plant 2 largely confirmed the results of the first plant and were added to check the robustness of the methodology. The results show how to quantify leaf orientation variation and leaf movements along a day at mm accuracy in different light conditions. This confirms the feasibility of the proposed methodology to robustly analyse leaf movements.

Keywords: leaf movements, plants, terrestrial LiDAR, indoor, temporal series

# INTRODUCTION

Plant canopy structure properties and their spatial changes, are linked to different vegetation processes, such as radiation absorption, plant water balance, precipitation interception and photosynthetic activity (Harley and Baldocchi, 1995). Canopy structural and biochemical variables are frequently used as constraints to model interactions between the land surface and the

#### Edited by:

Eetu Puttonen, National Land Survey of Finland, Finland

#### Reviewed by:

Jan Behmann, University of Bonn, Germany Gerben van Ooijen, University of Edinburgh, United Kingdom

\*Correspondence:

Mónica Herrero-Huerta monicaherrero@usal.es

#### Specialty section:

This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science

> Received: 11 October 2017 Accepted: 31 January 2018 Published: 16 February 2018

#### Citation:

Herrero-Huerta M, Lindenbergh R and Gard W (2018) Leaf Movements of Indoor Plants Monitored by Terrestrial LiDAR. Front. Plant Sci. 9:189. doi: 10.3389/fpls.2018.00189

**120**

atmosphere (Sellers et al., 1997). Moreover, many studies that investigated the implications of interception and light transmission for species competition, biodiversity, ecosystem and agro-ecosystem dynamics, as well as wood production, depend on the spatial distribution of leaves and branches (Pretzsch, 2009).

In ecology and plant physiology, circadian rhythms are activities that occur on a near-24-h cycle due to ecologically useful adaptions, regarding plant's physiology and its environment (Sadava et al., 2009). In this context, the growth patterns of roots and leaves are determined by the circadian clock and leaf starch metabolism (Ruts et al., 2012). Therefore, diel leaf movements are a well-typified symbol of the circadian clock (Farré, 2012). These movements have been studied intensively for a long time (Barak et al., 2000). Two reason are notably identified by scientific researches as operators of these fluctuations, plant water balance (Chapin et al., 2011), and photoperiodism (Sysoeva et al., 2010), related to the fluency rate of photosynthetically active radiation in the plant. To optimize the interception of incoming light and to avoid temperature related stress, the leaf angle is adjusting (Medina et al., 1978; Ehleringer, 1998). Plant imaging aims to analyse and quantify the development, growth, physiological and other phenotypic plant properties by different accurate and automated processes. A complete review focuses on the latest advances in high-throughput image-based plant phenotyping (Fahlgren et al., 2015). Although many successes are reported, image based techniques may require the use of a flash during low-light conditions, while some effort is needed in setup and/or processing to manually or automatically obtain 3D results.

TLiDAR has newly arisen as a promising tool to fast measure 3D vegetation structure at plot level with high spatial resolution and millimeter accuracy (Dassot et al., 2010). Light Detection And Ranging (LiDAR) is an active remote sensing technique that accurately measures distances by transmitting laser energy and detecting the time of arrival of the return energy. As an active technique, it is almost insensitive to varying external lighting conditions and is able to capture data even in absence of light. TLiDAR data are extensively used in engineering applications to monitor morphological terrain changes (Herrero-Huerta et al., 2016). The interest on TLiDAR for vegetal plot measurements started in the past decade in forestry (van Leeuwen and Disney, 2018) and (Thies and Spiecker, 2004). Recently, a fully-automatic approach for tree structural modeling at plot level has been proposed in Raumonen et al. (2013). Circadian rhythms in tree geometry can be accurately monitored at millimeter scale and sub-hour temporal resolution by TLiDAR in outdoor conditions (Puttonen et al., 2016). Dornbusch et al. analyse circadian movements and rhythmic leaf growth of Arabidopsis rosettes using a near-infrared laser scanner (Dornbusch et al., 2014). Low cost 3D imaging tools were used for this purpose in Paulus et al. (2014) and evaluated with respect to the metric quality of estimated geometric plant features. To summarize, TLiDAR has potential to monitor vegetation geometry at different scale levels. In addition, scanning can be done outdoors, is insensitive to vary lighting conditions and is able to operate at up to temporal resolutions of minutes. The scanning mechanism is based upon a fast rotating multi-facet polygonal mirror, so little effect is transmitted to plants. As a disadvantage, point clouds cannot provide direct biochemical parameters from plants, as hyperspectral and thermal chlorophyll fluorescence imaging do (Fiorani and Schurr, 2013). On the other hand, image based-3D modeling needs a significant amount of post-processing, while TLiDAR requires a higher cost.

Goal of this research is to show how plant moves can efficiently be captured and parameterized using TLIDAR. In general, data capturing using TLIDAR is considered relatively easy. However, there is still a lack of methodology and corresponding software tool to automatically extract geometric parameters from point cloud data sampling moving plants. Thus, the goal of this study is to propose a phenotyping pipeline to automatically determine movement from leaves in indoor plants through TLiDAR.

The paper is organized as follows: after this brief introduction, the proposed methodology is described in section Methodology. Next, the results and the discussion are presented in section Experimental Results and Discussion, respectively. Finally, conclusions are drawn.

# METHODOLOGY

# Experimental Setup

An experiment was carried out with the understory species Calathea roseopicta (Linden) Regel, a native species from Tropical America, with basal leaves growing in a rosette shape. Because of the photo- and thermotropism of most tropical plants, Calathea unfurl its leaves and moves the whole system including the stem toward the light to optimize the surface for sunlight absorption. During the dark period (night) the leaves coil up and minimize the surface to prevent the plant from coolingoff. Therefore, this species is suitable to monitor the complex, spatial movement of the leaf-stem system. It needs a minimum temperature of 16◦C so in temperate systems, it is commonly kept in indoor conditions. Its non-lignified tissues allow the presence of soft tissues susceptible to soil water and light conditions (Chao-ying, 2009). Two plants were tested, planted in an 850 cm<sup>3</sup> PVC container filled with a substrate composed of sand and a high content of organic material. Studied plant #1 had a mean height of ∼45 cm and a canopy diameter of ∼31 cm during the measurements, while plant #2 has ∼44 cm and ∼29 cm, respectively, as **Figure 1** illustrates.

The experiment was performed in December 2016. Air temperature (◦C) and relative humidity (%) were continuously monitored during the sampling period with a HOBO U30- NRC weather station, with sampling intervals of 5 min. Light conditions were measured during the experiment by the "LuxMeterPro Advanced App" from AM PowerSoftware, with 12 lux-accuracy for a distance <5 m. A Leica Scan-Station C10 scanner, a high speed TLiDAR implementing the so-called timeof-flight principle, is used to collect data (Vosselman and Maas, 2010) 1 . Obstructions in the visibility w.r.t. the position of the

<sup>1</sup>Available online at: https://hds.leica-geosystems.com (Accessed January 11, 2018).

laser scanner is hindering complete sampling of the plant surface. Indeed, leaves in the front, as seen from the scanner position, may occlude leaves in the back. Thereby, the relative horizontal position of the TLIDAR w.r.t. the scanned plant should be carefully chosen to have best coverage of leaves within a single scan. Using multiple scans could mitigate this occlusion effect but would (i) include larger time differences within one acquisition and (ii), would require an additional matching or registration step to align the scans as obtained from different positions into a common coordinate system.

Both plants were located in a room, 1.6–1.8 m from the single window oriented in SW direction. Scans were performed at 2 different days. The first sampling day (test A) was done under natural light conditions. Previously to these TLiDAR measurements, the plants were kept in full light conditions. After a 2 day-period in which the plants were kept in complete darkness, the second sampling (test B) was carried out in the dark. An overview of the acquired data is shown in **Table 1**.

#### Method of Leaf Movement Estimation Outlier Removal

The point cloud sample of the plants may contain outliers and noisy points caused by various reasons such as backscatter from interference effects (Thies and Spiecker, 2004). Such points are not regarded as samples of the actual plant and the first step is to filter them from the point cloud. To remove separate isolated points or few-point clusters, a statistical analysis of the distances is performed as follow. First, all points are visited. For each query point, all neighboring points within a pre-set distance are determined and all distances from the query point to these points are determined. After visiting all points, the mean and standard deviation of these distances (d and σd, respectively) is stored.

Next, a threshold (Ld) is defined as:

$$L\_d = \overline{d} + \mathfrak{p} \cdot \sigma\_d \tag{1}$$

where p is the confidence level, expressed as the critical value associated in the standard normal density curve (Palnick, 2014).

Again, all points are visited. For a given point, when its average distance to its k neighbors is above Ld, it is marked as an outlier and removed (Rusu et al., 2008). This method is reasonably fast once the points are organized in a data structure like an octree.

#### Point Cloud Registration and Deviation Map Calculation

Scans are taken without moving the TLiDAR during the sampling periods. Still, it is not guaranteed that the scans obtained at different moments are fully aligned. Therefore, the registration or alignment of the 3D point clouds is checked using 7 external spherical targets of 10 cm-diameter from Leica©. This registration step estimates a rigid body transformation (only translations and rotations can be applied to the point clouds). The Iterative Closest Point algorithm by Besl and McKay (1992) is used, reaching a negligible mean absolute error among targets from multi-temporal datasets.

After that, all the point clouds are cropped using a common bounding box. At this point, the recognition of correspondences among point clouds is essential to compute plant movements of the plant as displacements between multi-temporal datasets. Therefore, first cloud-to-cloud differences between sequential TLiDAR datasets are computed using a chamfer distance approach that exploits an octree organization (Akmal Butt and Maragos, 1998). Second, the distance statistics resulting from this first step are used to refine the displacements. A more precise cloud-to-cloud distance between two multi-temporal point clouds is extracted by applying a local approximation model to the reference cloud by a quadric surface. As a result, a deviation map of displacements between point clouds is obtained which precisely outlines the coordinates and enables the quantification of plant movements.

#### Segmentation in Individual Leaves

The changes in the orientation of the leaves along time are estimated from the point cloud data. For that purpose, individual plant leaves are extracted from the point clouds using a segmentation method based on an organization of the point cloud in an octree structure, as proposed in Woo et al. (2002).


Initially, the point normals are estimated by Delaunay triangulation (Golias and Dutton, 1997). Next, the 3D points are organized in an octree structure, where the local point normals are used as a criterion for splitting an octree cell: if points at a certain octree depth all have similar normals, subdivision terminates; otherwise, the octree cell is divided. This process is repeated until normals stabilize at a certain depth or the maximal octree depth is reached. After splitting is finished, adjacent cells having similar orientation are merged in a bottom-up procedure. First, cells with a large variation in normals are simply removed. Such behavior might, for example, occur at the edges of leaves. Next, an optimally flat cell is selected as a seed cell and adjacent cell are added as long as a homogeneity criterion is met. When such segment is finished, a new seed is selected and the procedure is repeated using the remaining cells. The standard deviation of the normal vectors in a cell is taken as homogeneity criterion. After termination of this procedure, the point cloud is automatically separated into certain regions by considering the obtained segments. Using a suitable criterion value, it is possible to tune the method such that the resulting segments correspond to individual leaves.

The last step is to match individual leaves, which is difficult as the individual leaves are moving and changing the shape between the different data acquisitions. Automatic co-registration of each leaf point cloud segment is done by least squares matching of overlapping surfaces (Least Squares 3D Surface Matching). The transformation parameters of the 3D compared surface (leaf segment after movement) to a 3D reference surface (leaf segment before movement) is approximated by a so-called Generalized Gauss–Markov model, based on minimizing the squared sum of Euclidean distances between surfaces (Gruen and Akca, 2008).

#### Leaf Normal Calculation

Once the leaves are segmented, leaf-wise orientation is approximated by the leaf normal. For this purpose, Principal Component Analysis (PCA) is used (Weinmann et al., 2014). This statistical analysis uses the first and second moments of the points and results in three orthogonal vectors centered on the center of gravity of the point cloud. The PCA synthesizes the distribution of points along the three dimensions and therefore models the principal directions and magnitudes of variation of the point distribution around the center of gravity.

The coordinates x<sup>i</sup> , y<sup>i</sup> , and z<sup>i</sup> for each point i=1,. . . k from the point cloud of each leaf, is considered. The covariance matrix (Σ) (2) of each leaf point cloud (X) is defined by:

$$\sum = \frac{\left(X - \overline{X}\right)^{T} (X - \overline{X})}{\mathbf{k}} = \begin{pmatrix} \sigma\_{\mathbf{x}}^{2} \,\sigma\_{\mathbf{x}\mathbf{y}} \,\sigma\_{\mathbf{x}\mathbf{z}}\\ \sigma\_{\mathbf{x}\mathbf{y}} \,\sigma\_{\mathbf{y}}^{2} \,\sigma\_{\mathbf{y}\mathbf{z}}\\ \sigma\_{\mathbf{x}\mathbf{z}} \,\sigma\_{\mathbf{y}\mathbf{z}} \,\sigma\_{\mathbf{z}}^{2} \end{pmatrix} \tag{2}$$

where (σ 2 x , σ 2 y , σ 2 z ) are the variances of the coordinate directions and the elements outside the main diagonal of Σ are the covariances. X contains k copies of the mean of the three coordinates of the leaf point cloud (X). The principal leaf normal is the third eigenvector of Σ (Jolliffe, 2002).

#### Computation of Orientation Variation

The orientation variation of each leaf is characterized in a spherical coordinate system by determining the change in both spherical azimuth (θ) (aspect) and elevation angle (8) (hyponasty). For this reason, firstly, a subtraction of the normals from each leaf at different times is done, always in temporal order with respect to the first epoch. These angles obtained from the difference-in-normal vector (x, y, z) are calculated as follows:

$$\theta = \arctan\frac{\mathbf{y}}{\mathbf{x}}\tag{3}$$

$$\mathcal{Q} = \arctan{\frac{\mathbf{z}}{\sqrt{\mathbf{x}^2 + \mathbf{y}^2}}}\tag{4}$$

**Figure 2** explains how the computation of the orientation variation is carried out.

#### Volume Variations by Canopy Mesh

The foliage shape of the full plant may help to understand the radiation regime and canopy structure. However, the single scan acquisitions only give a partial point cloud due to the one-sided field of view of the TLiDAR. This paragraph explains how to quantify the canopy volume variation from the point cloud of the plant.

The approach is supported by a 3D modeling of the plant using a 3D Delaunay triangulation (Golias and Dutton, 1997), as used before for the leaf segmentation (section Segmentation in Individual Leaves). The result is a 3D mesh. An additional filter phase is required to obtain a plant canopy model close to reality. The approximation shown in Attene (2008) is used in this step which incorporates several smoothing operators to repair issues in the mesh like holes and different types of noise (Fan et al., 2008).

Once the canopy mesh is fixed, its shape is approximated by a 3D convex hull (Barber et al., 1996). The convex-hull is chosen due to its capacity of enveloping the effective area which potentially can be related to the radiance absorption. As an alternative in future, an alpha-shape approximation of the hull could be used, which is able to more flexible estimate the shape of the plant canopy, by varying the alpha coefficient.

As single scans only provide partial point clouds, the recovery of the less well-visible backside is achieved through symmetry. Therefore, the symmetry axis is estimated by a 180◦ field of view from the dataset. Finally, the canopy shape is obtained which allows the direct volume estimation and changes along time.

#### EXPERIMENTAL RESULTS AND DISCUSSION

Sunrise and sunset times were similar during the experiment. Therefore, the sampling periods are comparable. The plant substrate was maintained well-watered to reduce the effect of root water shortage. The room temperature remained constant during the sampling period; however the relative humidity changed between test A and B (**Table 2**). This variation in humidity could have led to stress or more water movement within the plants in test B.

The scan pre-processing settings to remove noise (section Outlier Removal) from the data were fixed to a neighborhood size of a 1.6 cm sphere diameter. The leaf movements of C. roseopicta plant 1 are illustrated in **Figure 3**, by superimposing the scan obtained at 7.00 a.m. to the one obtained at 12.30 p.m. during test A (**Figure 3A**) and by calculating the deviation map from the compared point cloud (at 12.30 p.m.) to the reference point cloud TABLE 2 | Room conditions during test A and B [sunrise and sunset times provided in GMT(+1)].


(at 7.00 a.m.) (**Figure 3B**), as section Point Cloud Registration and Deviation Map Calculation explains. This time interval was chosen as during this time the biggest movement in terms of leaf displacement occurred. Further estimations can be obtained from this product. For instance, the maximum displacement of 7.92 cm between the two scans occurs at leaf #7. The average distance movement of this leaf is 3.67 cm.

In order to apply the segmentation method to extract individual leaves, some parameters have to be fixed; notably, the minimum voxel size and the number of iterations. These parameters were set as 1.8 cm and 5 iterations, respectively. Furthermore, the standard deviation of point normals within a cell as a tolerance for subsequent subdivisions, was set at 6 cm. To perform the Least Square Matching for individually matching the leaves among multi-temporal datasets, the iteration criteria threshold was set as 3.8 cm. As a result, 9 leaves of plant 1 and 11 leaves of plant 2 were recognized from all the scans. **Figure 4** shows the individual leaves extracted from both plants at 7.00 a.m. (test A) in different colors, where normals in red are obtained by PCA. Outliers are colored in dark blue.

The global accuracy reached was analyzed by comparison to a manual segmentation of the leaves. To do so, each leaf is considered a target class and for each point. It can be determined if it is classified in the correct class. Results are collected in a confusion matrix (not shown), (Sasidharan et al., 2010). The resulting global accuracy was found to be 93.57% for plant 1 and 87.34% for plant 2. The plant species morphology gives the opportunity to use the proposed automatic segmentation technique due to the fact that the leaves and stems have dissimilar orientations. Once the individual leaves are obtained, the variability of each leaf orientation along the sampling tests is estimated. **Figure 5** shows the variation in orientation from the leaves of plant 1 during test A (**Figure 5A**) and during test B (**Figure 5B**). These variations were first derived between 7.00 a.m. and 12.30 p.m. (beginning of the arrow) and next between 12.30 and 6.00 p.m. (end of the arrow). These intervals were chosen because during these intervals, movement was bigger. The azimuth changes are represented on the x-axis while the vertical angle change is on the y-axis (both in sexagesimal

degrees), taking into account the normal direction of each leaf as obtained by PCA. The color of each trajectory corresponds to the leave color in **Figure 1**. Different light conditions clearly correspond to different movement directions. The orientation change in darkness (**Figure 5B**) is smaller than during natural light conditions (**Figure 5A**).

The results of the orientation variations from plant 2 are displayed in **Table 3**. A similar magnitude order in the orientation variation is reached regarding plant 1 during both tests.

From the scan analysis results, we conclude that C. roseopicta displayed significant movement in the leaves during light and dark periods, with different patterns for both tests and plants.

FIGURE 4 | Individual leaves of plant 1 (A) and plant 2 (B) extracted using an octree-based segmentation (leaves in different colors) and consecutive orientation estimation by PCA (normal vector in red of each leaf).

Herrero-Huerta et al. Leaf Movements by Terrestrial LiDAR

This species shows conspicuous movement between the tested light conditions, however this leaf movement can be influenced by growing patterns of young leaves (Puttonen et al., 2016).

Leaf #1 from plant 1, for example, moves in the same direction during both light conditions, but its movement is larger in natural light conditions. In both situations, the starting point is similar: it means that the angle variation between 7.00 a.m. and 12.30 p.m. is comparable. Conversely, the end point is really different (from 12.30 to 6.00 p.m.). A possible explanation is that the movement of this leaf during the afternoon is related to the photosynthesis process, so the leaf movement has less influence for the sun-light in the morning. It can probably be related


to the different transpiration rates produced by leaf #1 within a day.

Canopy variations along time can actually be quantified. As section Volume Variations by Canopy Mesh describes, the convex-hull of the entire plant is generated from the point cloud. This process is illustrated in **Figure 6**, where the shape variation of the plant 2 during test A at 12.30 p.m. (**Figure 6A**) and 6.00 p.m. (**Figure 6B**) can be clearly defined as 'closing the leaves' (**Figure 6C**). The convex-hull volume was analyzed to extract the variations at different times. **Table 4** characterizes these values during the morning and afternoon in test A and B from the two studied plants, proving that in natural light conditions the variation is always bigger.

Thus, our data shows that light conditions influence diel leaf movement, characterized by movement patterns and leaf movement rate. As a general idea, we saw that leaves were opening during the day and closing at night (changing the elevation angle along sampling periods), generating an up and down movement as previous studies already indicated (Manel et al., 2001).

Notice that leaves can generate shadows within plants, occluding certain parts, therefore the obtained 3D point cloud of the plant surface, usually is incomplete. To overcome this challenge, the individual leaf architecture could be reconstructed via a model, already proposed for cereal leaves (Dornbusch et al., 2012).

The biological significance of diel leaf movements is related to favorable conditions of energetic requirements, water availability and auxin responsiveness (Stitt and Zeeman, 2012). This proposition agrees with the maximal peak of movement and


orientation variation during the early morning. Moreover, the leaf vertical angle tracks the usual daily temperature oscillations with a top in the late-afternoon. Bridge et al. (Bridge et al., 2013) determined that moving up the leaves is favorable to cool them during the warm daily hours and reduces the radiation contents when it exceeds the photosynthetic capacity.

#### CONCLUSIONS

This paper shows how TLiDAR can be used to efficiently and non-invasively parameterize plant movement. It presents a phenotyping method to monitor leaf movements from plants by TLiDAR, even in absence of light. We show that this approach is robust and computationally easy to determine, providing a tool to accurately measure complex leaf-stem system movements from plants with high temporal and spatial resolution.

Calathea roseopicta species was studied, showing different leaf movements in presence of natural daylight conditions and in darkness. Moreover, canopy variations in volume were analyzed by a convex hull approach. In darkness, the displacements are smaller and with a dissimilar orientation pattern than in natural light conditions. The natural movements are expressed by opening of the leaves during the day and closing them at night, pointing to the source of natural light (the window, in our study case). In absence of light, the plant still showed displacements, but with different pattern and magnitude.

The low relative humidity conditions during both sampling periods may have resulted in an increment in transpiration rates, leading to possible leaf movement during the photosynthesis. As

#### REFERENCES


further consideration, water availability should be controlled and sunlight received by the plant should be equal in all directions to rigorously compare multi-temporal datasets.

Using TLiDAR technology to derive vegetation movements open up fresh challenges which merit more methodological advance. The proposed methodology could help to support biological investigations of exchange of water, sunlight absorption and other bio-geochemical between plants and the atmosphere. Furthermore, these achievements would break through the barrier of knowledge in objectively determining the behavior of plants and even its structure and function in ecosystem, nowadays remaining poorly understood (Whippo and Hangarter, 2009). Furthermore, the behavior of interior plants is linked to indoor air quality and this connection will be explored as further studies.

#### AUTHOR CONTRIBUTIONS

MH-H designed the experiment, created the software, ran the data analysis and analyzed the data; MH-H and RL wrote the paper; WG edited the paper. All authors read and approved the final manuscript.

#### ACKNOWLEDGMENTS

Authors would like to thank Cesar Jimenez-Rodriguez from the Department of Water Management and Jacintha Tjia, bachelor' student, both from Delft University of Technology, for their collaboration during the experimental phase of this research.


**127**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Herrero-Huerta, Lindenbergh and Gard. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# A Low-Cost Imaging Method for the Temporal and Spatial Colorimetric Detection of Free Amines on Maize Root Surfaces

Truc H. Doan<sup>1</sup> , Tu A. Doan<sup>1</sup> , Michael J. Kangas<sup>2</sup> , AdreAnna E. Ernest<sup>1</sup> , Danny Tran<sup>3</sup> , Christina L. Wilson<sup>2</sup> , Andrea E. Holmes<sup>2</sup> , Erin L. Doyle<sup>1</sup> and Tessa L. Durham Brooks<sup>1</sup> \*

<sup>1</sup> Department of Biology, Doane University, Crete, NE, United States, <sup>2</sup> Department of Chemistry, Doane University, Crete, NE, United States, <sup>3</sup> Crete High School, Crete, NE, United States

#### Edited by:

Alexander Bucksch, University of Georgia, United States

#### Reviewed by:

Hannes Schmidt, University of Vienna, Austria Diana Heuermann, Leibniz-Institut für Pflanzengenetik und Kulturpflanzenforschung (IPK), Germany

\*Correspondence:

Tessa L. Durham Brooks tessa.durhambrooks@doane.edu

#### Specialty section:

This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science

> Received: 30 May 2017 Accepted: 16 August 2017 Published: 30 August 2017

#### Citation:

Doan TH, Doan TA, Kangas MJ, Ernest AE, Tran D, Wilson CL, Holmes AE, Doyle EL and Durham Brooks TL (2017) A Low-Cost Imaging Method for the Temporal and Spatial Colorimetric Detection of Free Amines on Maize Root Surfaces. Front. Plant Sci. 8:1513. doi: 10.3389/fpls.2017.01513 Plant root exudates are important mediators in the interactions that occur between plants and microorganisms in the soil, yet much remains to be learned about spatial and temporal variation in their production. This work outlines a method utilizing a novel colorimetric paper to detect spatial and temporal changes in the production of nitrogen-containing compounds on the root surface. While existing methods have made it possible to conduct detailed analysis of root exudate composition, relatively less is known about where in the root system exudates are produced and how this localization changes as the root grows. Furthermore, there is much to learn about how exudate localization and composition varies in response to stress. Root exudates are chemically diverse secretions composed of organic acids, amino acids, proteins, sugars, and other metabolites. The sensor utilized for the method, ninhydrin, is a colorless substance in solution that reacts with free amino groups to form a purple dye. A detection paper was developed by formulating ninhydrin into a print solution that was uniformly deposited onto paper with a commercial ink jet printer. This "ninhydrin paper" was used to analyze the chemical makeup of root surfaces from maize seedlings grown vertically on germination paper. Through contact between the ninhydrin paper and seedling root surfaces, combined with images of both the seedlings and dried ninhydrin papers captured using a standard flatbed scanner, nitrogen-containing substances on the root surface can be localized and concentration of signal estimated for over 2 weeks of development. The method was found to be non-inhibiting to plant growth over the analysis period although damage to root hairs was observed. The method is sensitive in the detection of free amines at concentrations as little as 140 µM. Furthermore, ninhydrin paper is stable, showing consistent color changes up to 2 weeks after printing. This relatively simple, low-cost method could contribute to a better understanding of root exudates and mechanisms used by plants to interact with the complex soil environment during growth and development.

Keywords: rhizosphere, root exudates, maize seedling, colorimetric detection, ninhydrin, printing sensors, root imaging

# INTRODUCTION

fpls-08-01513 August 29, 2017 Time: 11:56 # 2

The rhizosphere is a complex and dynamic microecosystem formed between plant roots and their associated microbes, single-celled eukaryotes, mycorrhizal fungi, and small soildwelling multicellular organisms (Badri and Vivanco, 2009). At its core, it is a partnership between roots, which exude a variety of metabolites, ions, and cofactors, and microbes that respond to, utilize, and process them (Badri and Vivanco, 2009; Dennis et al., 2010; Kumar et al., 2012). Interactions of exudates with microbes in the rhizosphere can have significant benefits to the plant including mobilization of soil nutrients, prevention of and enhanced response to infection, and promotion of plant growth and development (Badri and Vivanco, 2009; Kumar et al., 2012). Advances in DNA sequencing technology have shown that microbial composition and abundance of the rhizosphere differs from that of the bulk soil, as evidenced by a decline in diversity and an increase in specialization near the root surface (Micallef et al., 2009; Bulgarelli et al., 2013; Peiffer et al., 2013). Root exudates are thought to be a primary driver of microbial community makeup (Micallef et al., 2009; Dennis et al., 2010; Bulgarelli et al., 2013). Additionally, root exudates play a critical role in plant–plant communication and mediate a variety of positive and negative interactions between nearby plants such as the induction of defense responses (Walker et al., 2003). Therefore, understanding exudate composition and determining genetic and environmental factors that influence its production is a primary focus of crop improvement efforts (Bulgarelli et al., 2013; Chaparro et al., 2013).

Root exudates represent a significant portion (20–50%) of the total metabolism of the plant (Aguiar et al., 2014). They are produced within the root and transported into the nearby soil environment by root epidermal cells and root hairs, and by the production of mucilage and the sloughing off of cells at the root cap (Walker et al., 2003; Dennis et al., 2010). Root exudates are composed of a mix of primary metabolites including sugars, amino acids and organic acids. In smaller abundance are fatty acids, sterols, cofactors, and regulatory molecules (Aguiar et al., 2014). Their composition changes with the plant's genetic makeup, development of the plant and in response to stress. Both biotic and abiotic stressors are able to mediate both positive and negative interactions (via exudate components) between rhizosphere microbes and surrounding plants (Czarnota et al., 2001; Walker et al., 2003; Warembourg et al., 2003; Micallef et al., 2009; Dennis et al., 2010). Amino acids represent the second-most abundant component of root exudates (Chaparro et al., 2013; Moe, 2013). They contribute to nitrogen cycling in the soil and are important in attracting beneficial root microbes (Jones et al., 2002; Oku et al., 2012; Moe, 2013). The amino acid exudate gamma-aminobutyric acid can act as a signal to prime root defenses (Jakab et al., 2001; Moe, 2013). Finally, amino acid exudates, particularly tryptophan, can be important signals and metabolic intermediates affecting root growth and development through the production of phytoactive hormones by plant growth promoting bacteria (PGPB) (Sukumar et al., 2013; Koul et al., 2015).

Maize exudates contain a variety of free-amine containing compounds, primarily in the form of amino acids (Kraffczyk et al., 1984; Baudoin et al., 2003; Fischer et al., 2010). Glutamate is thought to be a primary component of this portion of the exudate, though serine, alanine and glycine have also been observed in relatively high proportions (Baudoin et al., 2003; Fischer et al., 2010) and its concentration increases with the presence of growth-promoting rhizobacteria and certain nutrient deficiencies (Kraffczyk et al., 1984; Carvalhais et al., 2011). Functionally, maize root exudates play roles similar to what they do in other species including nutrient mobilization, enhanced defense response, and attraction of growth-promoting bacteria (Mench and Martin, 1991; Baetz and Martinoia, 2014; Zhang et al., 2015).

Only recently have metabolomic studies been undertaken to more systematically describe the chemical makeup of exudates (Chaparro et al., 2013). While it is known that apical structures of the root are generally more active in producing exudates, the spatial distribution of exudates and how they change with time remain largely unexplored (Walker et al., 2003; Bulgarelli et al., 2013). Understanding the complex relationships between plant roots, root exudates, and rhizosphere microbes requires detailed measurements of exudate composition and their spatial/temporal distribution. A number of approaches have been published, all of which have advantages and costs. Root exudates may be collected from roots grown in soil, hydroponically, or on other solid support systems such as glass beads, nutrient agar or a capillary mat (Czarnota et al., 2001; Ernst et al., 2010; Dundek et al., 2011; Chaparro et al., 2013; Aguiar et al., 2014; Slota et al., 2016). For some exudate collection methods, roots are washed to remove residual soil, nutrients, and dead cells and are then incubated in a bath solution from which the exudates are recovered (Dundek et al., 2011). This approach, while providing a more natural growing environment, has the disadvantage of being destructive, thereby only allowing a single time point to be collected for each plant. Some experimental setups allow for exudates to be non-destructively sampled as hydroponic media is continuously perfused through growth system and collected, but localization of exudate production in these systems is difficult (Oburger et al., 2013; Slota et al., 2016). Another way to track exudates is to introduce radioactive isotopes in combination with imaging techniques (Oburger and Schmidt, 2016), but introduction of radioisotopes into the lab is not feasible in some research settings.

Water-soluble exudates are typically filtered and concentrated prior to analysis (Dundek et al., 2011). The recovered exudates may be fractionated using chromatography techniques and sample fractions are characterized using UV-vis, mass spectrometry, or NMR (Micallef et al., 2009; Kim et al., 2010; Dundek et al., 2011; Aguiar et al., 2014). In a targeted analysis approach known exudate compounds are detected in the sample. However, methods for non-targeted profiling of exudate composition are rapidly growing and can lead to discovery of new compounds in the exudate and determinations of how exudate composition changes with a given condition (Marti et al., 2013; Oburger et al., 2013). Any metabolomics approach necessarily requires use of highly specialized equipment. A much more accessible approach has been to use colorimetric indicators and assays to more generally assess total carbohydrate, sugar, or

protein content (Van Egeraat, 1975; Dundek et al., 2011). Intact roots can also be exposed to a variety of indicators to detect the location of production and release of specific molecules. For example, blotting exposed plant roots with agar sheets containing the indicators bromocresol purple or BPDS were used to localize production of protons or iron-reducing compounds, respectively (Marschner et al., 1982; Römheld et al., 1984; Neumann et al., 2009). However, the agar sheets must be carefully prepared and stored, which adds difficulty and tedium to the procedure. In summary, there have been a variety of methods developed for collecting, detecting and analyze root exudate, but a need remains for development of simple (and higher throughput) approaches to enable spatial and temporal descriptions of exudate production.

We have developed a simple, minimally disruptive method for detecting and measuring root exudate production on the surface of a growing root (**Figure 1**). The colorimetric indicator ninhydrin has been used to detect free amine groups in exudates on the root surface. Ninhydrin is a useful compound for this purpose due to the easily visualized purple-colored product that is formed in the presence of analyte. Ninhydrin is also sensitive to free amines at physiological concentrations (Van Egeraat, 1975; Dundek et al., 2011; Chutipongtanate et al., 2012). A novel process for printing the indicator on commercially available tissue paper was developed and roots grown vertically on germination paper were blotted with this paper to capture a snapshot of free-amines on the root surface. Images of blotted ninhydrin papers and seedlings were captured and image processing tools developed to visualize free amine localization and relative concentration. Because our method does not disturb the plant as is grows and the indicator formulation is nontoxic to the seedlings, it can be used to collect time course data for individual plants. Additionally, we have determined that the ninhydrin-printed paper is shelf-stable and reliably detects root exudates at biologically relevant concentrations. Here, we demonstrate the application of our method to maize roots, but it could readily be expanded to other plant systems and indicators.

# MATERIALS AND METHODS

#### Preparation of Ninhydrin Paper

All reagents and chemical products were purchased from Sigma– Aldrich (St. Louis, MO, United States) at technical grade or better and were used without further purification. The print solution was composed of 71.2% (v/v) sodium acetate buffer (0.1 mM, pH 5), 14.8% glycerol, 8.2% diethylene glycol, and 5.8 % of triethylene glycol monobutyl ether.

Ninhydrin ink (3% w/v) was prepared using the print solution as the solvent and sonicated (Bransonic ultrasonic bath, CPX2800H) for 3 h at room temperature. The solution was stored in an amber bottle in a dark cabinet at room temperature.

The 3% ninhydrin ink was added to an empty cartridge (InkOwl, Champlain, NY, United States) using a syringe (with 0.8 µm filter) and placed into the printer (Canon MG5520). The high quality setting on the printer was used. Commercially available white tissue paper (Hallmark, Kansas City, MO, United States) cut to size (8.5 in × 12 in) was taped onto regular copy paper using all-purpose labels (Avery, 6737) to facilitate feeding of the tissue paper through the printer. The template provided in **Supplementary Presentation S1** was used to uniformly print ninhydrin on the paper, except for two 3 × 3 grids used for paper calibration and quantification of exudates. After printing, the ninhydrin paper was placed in an opaque envelope and stored at ambient temperature and humidity until used.

## Preparation of Mock Exudates

A mock exudate solution was prepared according to Baudoin et al. (2003) for use in determining ninhydrin paper sensitivity and longevity. The 70X mock exudate consisted of 18.4 mM glucose, 18.4 mM fructose, 9.2 mM sucrose, 9.2 mM citric acid, 18.4 mM lactic acid, 13.8 mM succinic acid, 9.2 mM alanine, 9.2 mM serine, and 5.5 mM glutamic acid. The total free amine concentration of 70X mock exudate was 23.9 mM. When diluted to 1X with distilled water, carbon content of mock exudate was at a concentration similar to

previously published values (Iijima et al., 2000) and the C/N ratio was similar to previously reported values (Kraffczyk et al., 1984).

#### Germination and Planting

fpls-08-01513 August 29, 2017 Time: 11:56 # 4

Seeds of maize inbred genotype B73 were sterilized by washing with 70% ethanol for 3 min and 2% (w/v) bleach for 5 min (Naveed et al., 2014). The seeds were then washed three times with autoclaved distilled water, stirring for 1 min for each wash. The seeds were rolled in paper towels wetted with sterile distilled water. The paper rolls were then covered by aluminum foil and imbibed at 4◦C for 2 days (due to improved germination of B73 during cold imbibition) then moved to a growth chamber (Percival, Geneva Scientific) set at 24◦C with continuous light (to avoid circadian effects) and with a 75% relative humidity (van de Venter and Grobbelaar, 1985; Naveed et al., 2014).

Three-day old seedlings were planted using a pouch method (Hund et al., 2009). Blue germination paper (Anchor Paper Co.) was trimmed to 8.5 in × 12 in, and a small triangle was removed from the center of the top edge. Seedlings were positioned at the apex of the triangle. Polyethylene sheets were cut approximately to the size and shape of the germination paper and positioned over it. Seedlings were held to the germination paper by placing paper clips on each side of the triangle apex. Plastic sheets and blotter germination paper were clipped to a wooden dowel using two binder clips and the dowel was used to hang the entire seedling apparatus using either end of the rim of 8-gallon opaque plastic containers (**Supplementary Figure S1**). Each 8-gallon containers held up to ten seedlings and were filled with enough sterilized distilled water to reach the bottom of the germination paper. Lightweight black cloth was draped over the either side of the seedling to keep the roots relatively dark. Seedlings were grown at 24◦C with continuous light and 75% relative humidity.

# Blotting and Scanning

Pouches were removed from the growth chamber every 2 days for scanning and blotting. The polyethylene sheet covering the surface of the root was pulled back to expose the root surface for blotting. To minimize water contacting and wrinkling the ninhydrin paper during blotting, laminated paper was used to cover portions of the germination paper where the root had not grown. The ninhydrin paper was gently pressed to the root for approximately 10 s (**Supplementary Figure S2**). After blotting, the ninhydrin paper was dried for approximately 24 h at room temperature with compression in a press (Carolina, Cat #: 663050) to eliminate wrinkles in the paper during scanning. The blotted ninhydrin papers were placed on the flatbed surface and color images were acquired using an EPSON V700 scanner and VueScan x64 software with settings shown in Supplementary Table S1.

An EPSON V700 scanner using VueScan x64 software with settings shown in Supplementary Table S2 was used for collecting RGB images of the roots. To minimize bubbles on the blue blotter germination paper and to ensure a consistent background, a clear plastic sheet was placed on the surface of the scanner, and the surface of the scanner was flooded with a small volume of water that was contained within the beveled edge of the scanner housing. The plastic sheet covering the root was lifted off the root surface and the seedling pouch was carefully placed onto the flooded scanner surface with the seedling facing down. Pouches were oriented in the same manner as the ninhydrin papers.

# Image Analysis

ImageJ software, which is freely available, open-source, and able to run on nearly any operating system, was used for analysis of spot intensities and root lengths (Schneider et al., 2008). Primary root length was measured using the segmented line tool. For determining paper sensitivity and stability, average pixel intensity of a spot created by depositing 1 µL of mock exudate on ninhydrin papers was determined by visually selecting areas containing signal with the magic wand tool. The average pixel intensity was quantified using the RGB measure function. Average pixel intensity was calculated as the weighted sum of the red, green and blue channels (0.30 R + 0.59 G + 0.11 B). This method can be set as an option in ImageJ (Edit/Options/Conversions menu).

The program GIMP2 (GNU Image Manipulation Program)<sup>1</sup> was used to produce overlay images between signals in a ninhydrin paper and seedling roots growing on germination paper. After opening the seedling root image it was converted to grayscale using the Colors/Desaturate command. The ninhydrin paper image was imported by dragging it onto the seedling image or by using File/'Open as Layers' from the seedling image. The ninhydrin paper layer was overlaid using a multiply operation. The 'Mulitply' option was selected from the 'Mode' pull-down menu in the 'Layers' toolbox to produce the overlay. The orientation of the ninhydrin paper was adjusted using the Move Tool as needed. The image overlay was exported in the desired format using the File/Export command.

## Determination of Seedling Growth Rate

Seedling growth rate (R) for a given day (D) was determined using the following formula where (L) is length in cm:

$$R\_D = \frac{L\_{D+2} - L\_D}{2} \tag{1}$$

#### Paper Calibration

To calibrate ninhydrin papers for root exudates quantification, three replicate 0.1 µL aliquots of mock exudate containing free amine concentrations of 1.71, 3.41, or 4.71 µM were deposited on one of the two printed grids on the paper immediately prior to time of blotting. After a 24 h incubation period in a press and imaging the paper as described above, the magic wand tool (ImageJ) was used to manually capture the developed areas for each aliquot. The average pixel intensity of each calibration spot was calculated in ImageJ as described above. For each of the three mock exudate concentrations, the lower and upper quartiles (Q3 and Q1) and the interquartile range (IQR, calculated as Q3-Q1) for average pixel intensity were

<sup>1</sup>http://www.gimp.org

calculated using the calibration spots of that concentration from the blotted ninhydrin papers for 3-, 5-, and 7-day old seedlings. An individual calibration spot was considered to be an outlier if its average pixel intensity was less than Q1 – 0.5<sup>∗</sup> IQR or greater than Q3 + 0.5<sup>∗</sup> IQR for the concentration. Papers with two or more outlier calibration spots for a single mock exudate concentration were determined to be outlier papers and were removed from the dataset.

For the retained papers, a linear regression was used to model the relationship between the known concentration of mock exudate applied and the measured pixel intensity of each spot. The regression equation is shown below, where y is the average pixel intensity and x is the free amine concentration in the mock exudates (R <sup>2</sup> = 0.72, p = 0.0):

$$y = -20.98x + 241.71\tag{2}$$

A 95% confidence interval (CI) was calculated for the regression line to determine the range of concentrations expected based on given average pixel intensity values.

In order to visualize exudate concentrations on the root surface, the blotted and scanned ninhydrin paper images were read using the Python OpenCV library and converted to grayscale. Using the regression equation and computed CI, pixel intensities across the entire image were converted to estimated free amine concentrations. A heat map function was applied to re-draw the image with approximate exudate concentrations indicated by a color gradient. A custom Python script for executing this visualization is provided as **Supplementary Data Sheet S1**.

# RESULTS

#### Method Overview

The method workflow is illustrated in **Figure 1**. Seedlings are grown using a modified pouch method, which allows easy access and visualization of the root surface (Hund et al., 2009). The roots are gently pressed against paper printed with a ninhydrin solution (**Figures 1A,B**). The blotted ninhydrin paper and the seedling roots are scanned and the resulting digital images overlaid to identify regions of the root that are active in nitrogenous exudate production (**Figures 1D,E**). After calculating the average pixel intensities of spots containing known free amine concentrations, outlier papers can be identified and free-amine containing exudate concentration in the remaining papers can be estimated (not shown).

## Ninhydrin Paper Sensitivity to Free Amines

The sensitivity of ninhydrin paper to the detection of free amines was determined by measuring the average pixel intensity of selected regions of 3% ninhydrin paper spotted with 1 µL of mock exudate solutions containing known free amine concentrations (**Figure 2**). Free amine concentrations between 0 and 3.41 mM were tested. A grid of nine 1 µL volumes for each mock exudate concentration was deposited onto on four separate ninhydrin papers. The aliquoted volumes were allowed to spread, creating uniform spots on the paper. Papers were stored in an opaque envelope in the dark overnight in order to develop the indicator. The colorimetric transition from white/pale yellow

∗∗ indicates a statistically significant difference (p < 0.01) between each sampling concentration to the control (water, 0 µM free amine). Statistical significance was determined using a two-way ANOVA with exudate free-amine concentration and paper (n = 4) as the factors and average intensity as the dependent variable. Pairwise comparisons were made from a post hoc Tukey's HSD. All points more concentrated than the free amine concentration indicated by the double asterisks were significantly different from the control (p < 0.01).

to purple was apparent within this time frame. Papers were scanned and average pixel intensity determined using ImageJ. Free amine concentrations as little as 140 µM were detected based on statistically significant differences compared to the control (p < 0.01, two-way ANOVA, Tukey post hoc). All concentrations tested were within the linear range of the indicator based on the linear trend of indicator response with concentration (**Figure 2**) and within the physiological range of the amine content expected in the exudate (Baudoin et al., 2003). Therefore, ninhydrin paper is sensitive to exudates within the physiological range.

# Ninhydrin Paper Stability in Atmospheric Conditions

The optimal use period, or time from printing until which a stable signal could be achieved, was determined for conditions typical to the laboratory (room temperature and ambient humidity). Ninhydrin paper was treated with mock exudate containing 3.41 mM free amines every day for 21 days after printing with 3% ninhydrin print solution (**Figure 3**). The ninhydrin paper was stored in an opaque envelope to protect it from light. The average pixel intensity was found to change most dramatically the first day after printing as observed by the increase from 134.77 ± 2.63 to 147.29 ± 3.09. This is hypothesized to be the result of the ninhydrin ink drying which at atmospheric humidity and temperature takes at least 2 days to complete. On day three to fourteen the RGB intensity stabilized to an average value of 150 with insignificant fluctuation in signal during this period (p > 0.05, two-way ANOVA, Tukey post hoc). After 21 days in storage a significant drop in ninhydrin paper sensitivity was observed which is hypothesized to stem from interaction of the paper with humid air or light, which could likely be mitigated by storage of the paper under more controlled conditions. Based on these data, ninhydrin paper was conservatively deemed optimal for blotting 4–13 days after printing.

#### Seedling Sensitivity to Ninhydrin Paper

As root growth is sensitive to environmental factors, such as chemical and mechanical stress, the impact of blotting with ninhydrin paper on root growth was determined. Insignificant differences were observed in growth length (**Figure 4**) and growth rate (**Figure 5**) between roots that were not blotted (Control), roots blotted with ninhydrin-free paper, and roots blotted with 3% ninhydrin paper. Seedlings that were blotted with ninhydrin paper were slightly, but significantly longer than roots used for the other two groups at the start of the experiment (**Figure 4**, p < 0.01, Student's T-test). Ninhydrin-free paper was used to determine the impact of mechanical stress of blotting while use of 3% ninhydrin paper indicates the potential impact of mechanical stress as well as chemical stress should ninhydrin be transferred from the paper to the root during contact. Roots achieved linear growth rates from day three through nine and plateaued in growth at around 15–18 cm after experimental day thirteen (**Figure 4**). The trend in growth rate was similar in all treatment groups (**Figure 5**) with roots growing at a steady rate of 4.5 cm/day from days three through eight of blotting followed by incremental decreases to 3.3, 1.2, 0.5, and 0.25 cm per day days 9, 11, 13, and 15 respectively. Overall, growth length and rate remained unaffected by mechanical stress and roots did not show any indication of toxicity to ninhydrin exposure.

### Root Hair Growth and Exudate Localization

In order to assess the impact of the method on root hair development, digital images taken before and after scanning were

FIGURE 4 | Impact of ninhydrin paper blotting on root length. Growth of B73 seedling roots along the germination paper surface without blotting (Control), while blotting seedlings with plain paper every 2 days (Ninhydrin-free paper), or while blotting with 3% ninhydrin paper every 2 days (Ninhydrin paper). Germination occurred at day 0. The data are the mean of 10 seeds for control, 9 seeds for the ninhydrin-free paper control, and 10 seeds for ninhydrin paper. Error bars are the standard error of the mean. <sup>∗</sup> indicates a statistically significant difference (p < 0.05) and ∗∗ indicates a statistically significant different (p < 0.01) between the ninhydrin paper condition and the control as determined by a Student's t-test. Note that seedlings blotted with ninhydrin paper were slightly, but significantly longer than roots used for the other two groups at the start of the experiment.

inspected. The blotting method damaged root hairs. Regions of the root containing root hairs before blotting (**Figure 6A**) no longer contain them after blotting (**Figure 6C**). However, root hairs were again found on regions of new growth within 2 days after blotting (**Figure 6D** and **Supplementary Figure S3**). While damage to root hairs contributes to some of the signal observed on the ninhydrin paper, signal was also observed in regions of the root that do not contain root hairs. For example, exudate production in the 3-day-old seedling in **Figure 6** is largely localized to the root tip and elongation

zone, rather than the region in which root hairs were present (**Figure 6B**).

In order to gain an initial characterization of root exudate production in young seedlings, roots growing in distilled water were blotted every other day over the course of several weeks with 3% ninhydrin paper. Grayscale images of scanned seedlings were overlaid with the corresponding developed ninhydrin paper. The first week of growth is shown for a representative B73 seedling in **Figure 7**. As expected, exudate production declined with age and was primarily localized to the apical regions of the root. By days five and seven, signal could also be observed at the base of the root and at the tip of seminal roots. Even though seedlings were grown in distilled water, signal could still be detected in some 17 and 19-day-old plants (**Figure 8**).

#### Quantification of Detected Free Amines

In order to facilitate relative quantification of signal within and across papers, two 3 × 3 grids of ninhydrin were printed on the bottom corner of each ninhydrin paper (**Supplementary Presentation S1**). Directly before blotting, mock exudates of known concentrations (three replicates of three standard concentrations) were applied to one of the grids (see Materials and Methods). After scanning, the developed spots on each grid were analyzed as described in the Materials and Methods to calculate mean pixel intensity of each spot. Outlier papers were detected and removed from the dataset using a Tukey fence as described in the Section "Materials and Methods." Overall, 20 of 30 papers collected from seedlings 3–7 days old were retained for analysis.

After outlier papers were removed, a linear regression of all spots for the remaining 20 papers was calculated to predict free amine concentration based on average RGB intensity. The regression was significant (p < 0) with an R <sup>2</sup> of 0.72. The upper and lower 95% CIs were used to determine the range in signal intensities that could be expected at each concentration of standard. These values ranged from 19 intensity units at the lowest standard (1.71 mM) to 30 intensity units at the highest standard (4.71 mM). The slope of the regression was used to convert these ranges into concentrations, which ranged from 1.0 mM at the lowest standard to 1.6 mM at the highest standard. The average range in concentration across the three standards was 1.3 mM. This value estimates the approximate resolution in quantifying the concentration of signal on the papers. In order to visualize and compare free amine signal intensities across papers, a heatmap was applied to the grayscale images of developed ninhydrin paper using a custom Python script (**Supplementary Data Sheet S1**). The resulting visualization enabled estimation of exudate production levels within and between ninhydrin papers. The standardized color scale that was determined from the linear fit of the retained data is shown in **Figure 9A**.

Closer inspection of grids from papers retained after removal of outliers and application of the heatmap revealed that the standards for approximately 65% of papers were colored to indicate the expected concentrations on the color scale. An example of two grids displaying this pattern are shown in **Figure 9B**. The grids of standards and the corresponding root signals shown in **Figure 9B** come from blots of the same seedling at 3 and 5 days after germination. For papers in this category,

FIGURE 7 | Overlay images of a seedling during the first week of growth. A single seedling during the first week of development shows production of free amines in the root tip and elongation zone (Day 3), the root base (Day 5) and at the tips of seminal roots (Day 7). Scale bar is 2 cm.

regions of interest can be referenced to the standard color scale shown in **Figure 9A** for approximation of signal concentration of signal within and between papers. The data shown for the seedling in **Figure 9B** indicates that free amine production did not exceed 4.2 mM at day 3 or 7. Free amines were concentrated at a relatively apical region of the root on day 3. Signal was still observed in apical regions of the root at day 7, but the most concentrated region of signal was found more basally than was observed at day 3.

A smaller proportion of retained papers contained grids with standards that were overexposed or underexposed compared to the expected values on the color scale. An example grid and corresponding root signal from a 5-day-old seeding are shown in **Figure 9C**. The data indicates that free amine production anywhere in this root was less than 4.71 mM and that much of the production was less than 1.71 mM. The signal concentration for papers in this category can be estimated by comparing regions of interest to the grid standards found on the same paper, but these estimations should not be made by comparing signals to the standardized color scale.

# DISCUSSION

This paper describes a cost-effective method for localization and low-resolution quantification of root exudate production during development. Roots are sensitive to chemical and physical stress, and exudate production or growth may be altered as a result of this stress, so any method developed for the purpose described here would aim to minimize such disruption (Badri and Vivanco, 2009; Baetz and Martinoia, 2014). Root hairs are recognized as important cells in production of root exudate and are estimated to make up as much as 77% of the total root surface area (Badri and Vivanco, 2009). While damage to root hairs was observed during our blotting procedure, regrowth of these cells was observed on new root tissue within 2 days (**Figure 6**). This observation and the

fact that the overall root growth rate was unaffected by blotting contribute to its utility in quickly gathering information about exudate production in a living plant. The indicator ninhydrin, which reacts with free amine groups, was chosen in part because amino acids represent a major component of root exudates and also because the indicator creates a color change that is easily detected using standard light imaging (Chutipongtanate et al., 2012; Chaparro et al., 2013; Moe, 2013). These features make it possible to explore adaptation of the method to other plant species and experimental conditions. Similarly, there are many more indicators that could be suitable for use in this method, such as pH indicators (Neumann et al., 2009). A future direction will be to identify additional chemical sensors for detection of other exudate components, including sugars.

Methods have been previously developed for visualization of root exudates within the soil environment and include use of rhizotrons or rhizobox systems for in situ sampling or visualization (Haase et al., 2007; Shi et al., 2011; Oburger et al., 2013; Oburger and Schmidt, 2016). Exudates can be sampled by exposing the root/soil interface with agar containing pH indicators or complexation and chelation agents (Neumann et al., 2009). Previous studies have also described use of colorimetric indicators for visualization of exudate production (Van Egeraat, 1975; Römheld et al., 1984; Dundek et al., 2011). However, while these methods have the advantage of allowing for more precise quantification of exudates within a more natural growth environment, the method described here is generally cheaper and simpler, while minimizing exposure of the root to the visualization chemicals making it possible to collect multiple samples from one plant and to potentially collect data at higher throughput.

Total free amine concentrations have been estimated at approximately 100–300 µM in soil extracts (Jones et al., 2002; Baudoin et al., 2003). However, growing conditions, particularly hydroponic growth systems, have previously been reported to limit free amine content of maize exudates (Oburger et al., 2013). Nitrogen limitation is an additional factor limiting free amine abundance in exudate (Carvalhais et al., 2011). The described method introduced both of these limiting factors. However, our ninhydrin paper, which was capable of detecting free amines in solution with a lower concentration limit of 140 µM (**Figure 2**), enabled visualization of free-amine containing compounds on the root surface despite these potential inhibiting effects. Therefore, we expect that our method could be useful for exudates detection across a range of plant species and experimental conditions. The root tip and apical meristem of the growing root, as well as root hairs, are known to be active regions of exudate production (Jones et al., 2002; Dennis et al., 2010). This agrees with the signal patterns observed in our data both for early seedling and later seedling growth (**Figures 7–9**). However, the method developed could be used for studies to detect exudate production patterns that may differ from this established scheme.

The method we have developed describes a semi-quantitative approach for visualization of root exudate production on the root surface. Next steps will include improving quantitative measurements of exudate production and automation of the image processing tasks. Future work will also include exploring nitrogen sources that can be added to the growth media to encourage normal growth and enhance exudate production and root growth while minimizing the amount of background signal. Other colorimetric sensors will be explored to improve sensitivity and to allow for detection of other types of exudates.

# CONCLUSION

The described method is simple and easy to use, consumables are inexpensive, and materials are commercially available. A flatbed scanner and open source software (Image J, GIMP2, Python/OpenCV) was used for overlaying and analyzing root images. The intermittent application of the ninhydrin paper did damage root hairs, but did not prevent normal root growth.

The rapid and low-cost detection of root exudates containing free amines was achieved by utilizing a novel colorimetric paper to detect spatial and temporal changes in exudate production on the root surface through the first 2 weeks of development.

# AUTHOR CONTRIBUTIONS

THD and TAD made substantial contributions to the study concept and design, ninhydrin paper preparation, seedling growth, image data collection and analysis, prepared figures and references, and provided relative interpretation. MK made substantial contributions to the study concept and design, ninhydrin paper preparation, image data collection process, and handled manuscript preparation including preparation of figures and references, gave intellectual contributions, improvements and revisions. AE and DT wrote Python scripts for visualizing ninhydrin signal concentrations and helped prepare the related figure. CW, AH, and ED drafted and revised the article including preparation of figures, gave intellectual contributions, improvements, and revisions. TDB conceived and supervised the research in all aspects and made significant intellectual contributions in outlining the article, article content, and critically revising the article for final submission.

# FUNDING

This work was funded by the National Science Foundation (NSF) EPSCoR Research Infrastructure Improvement (RII) Track-1 (1557417), National Institute for General Medical Science (NIGMS) (5P20GM103427), a component of the National Institutes of Health (NIH), SBIR Phase II - Chemical Biological Radiological Nuclear and Explosives (CBRNE) Reconnaissance Sampling Kit (W911SR-16-C-0051), and the Camille and Henry Dreyfus Foundation.

# ACKNOWLEDGMENTS

We would like to acknowledge Rachel Lukowicz, and Armando Pliego for their assistance in training and help with formulation of indicators. We would like to acknowledge Keeliann Mark and Michael Tross for assistance with printing and image analysis.

#### SUPPLEMENTARY MATERIAL

fpls-08-01513 August 29, 2017 Time: 11:56 # 12

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpls.2017.01513/ full#supplementary-material

FIGURE S1 | Seedling growth pouch setup. A seedling growth apparatus was constructed according to Hund et al. (2009). Seedlings were grown vertically on germination paper and a polyethylene sheet was used to maintain moisture on the sheet and to help affix the seedling to the paper surface. Seedlings were grown in distilled water. Up to 10 plants were grown together in one container. Seedling

## REFERENCES


roots were shaded from light in the growth chamber by draping the containers with sheets of lightweight black fabric.

FIGURE S2 | Illustration of the blotting method. Shown is a sheet of ninhydrin paper that has been laid on top of a piece of a seedling grown as described in Section "Materials and Methods." Regions of the germination paper that do not contain seedling have been shielded using laminated paper. The tissue paper is lightly blotted to ensure contact of all root surfaces with the ninhydrin paper.

FIGURE S3 | Detailed view of root hair regrowth after blotting. A detailed view of regions of the roots from Figures 6A,D are shown in (A,B), respectively. (A) Well-developed root hairs of a 3-day-old seedling root that has never been blotted. (B) A section of the same root is shown 48 h later. The initiation of root hair development within this timeframe can be observed.

PRESENTATION S1 | Printing template for ninhydrin papers.

DATA SHEET S1 | Custom script for applying a heatmap to calibrated ninhydrin papers.


Micallef, S. A., Shiaris, M. P., and Colón-Carmona, A. (2009). Influence of Arabidopsis thaliana accessions on rhizobacterial communities and natural variation in root exudates. J. Exp. Bot. 60, 1729–1742. doi: 10.1093/jxb/erp053

Moe, L. A. (2013). Amino acids in the rhizosphere: from plants to microbes. Am. J. Bot. 100, 1692–1705. doi: 10.3732/ajb.1300033


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Doan, Doan, Kangas, Ernest, Tran, Wilson, Holmes, Doyle and Durham Brooks. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# In-field High Throughput Phenotyping and Cotton Plant Growth Analysis Using LiDAR

Shangpeng Sun<sup>1</sup> , Changying Li <sup>1</sup> \*, Andrew H. Paterson2,3, Yu Jiang<sup>1</sup> , Rui Xu<sup>1</sup> , Jon S. Robertson<sup>2</sup> , John L. Snider <sup>2</sup> and Peng W. Chee<sup>2</sup>

*<sup>1</sup> School of Electrical and Computer Engineering, College of Engineering, University of Georgia, Athens, GA, United States, <sup>2</sup> Department of Crop and Soil Sciences, College of Agricultural and Environmental Sciences, University of Georgia, Athens, GA, United States, <sup>3</sup> Department of Genetics, Franklin College of Arts and Sciences, University of Georgia, Athens, GA, United States*

Plant breeding programs and a wide range of plant science applications would greatly benefit from the development of in-field high throughput phenotyping technologies. In this study, a terrestrial LiDAR-based high throughput phenotyping system was developed. A 2D LiDAR was applied to scan plants from overhead in the field, and an RTK-GPS was used to provide spatial coordinates. Precise 3D models of scanned plants were reconstructed based on the LiDAR and RTK-GPS data. The ground plane of the 3D model was separated by RANSAC algorithm and a Euclidean clustering algorithm was applied to remove noise generated by weeds. After that, clean 3D surface models of cotton plants were obtained, from which three plot-level morphologic traits including canopy height, projected canopy area, and plant volume were derived. Canopy height ranging from 85th percentile to the maximum height were computed based on the histogram of the z coordinate for all measured points; projected canopy area was derived by projecting all points on a ground plane; and a Trapezoidal rule based algorithm was proposed to estimate plant volume. Results of validation experiments showed good agreement between LiDAR measurements and manual measurements for maximum canopy height, projected canopy area, and plant volume, with *R* 2 -values of 0.97, 0.97, and 0.98, respectively. The developed system was used to scan the whole field repeatedly over the period from 43 to 109 days after planting. Growth trends and growth rate curves for all three derived morphologic traits were established over the monitoring period for each cultivar. Overall, four different cultivars showed similar growth trends and growth rate patterns. Each cultivar continued to grow until ∼88 days after planting, and from then on varied little. However, the actual values were cultivar specific. Correlation analysis between morphologic traits and final yield was conducted over the monitoring period. When considering each cultivar individually, the three traits showed the best correlations with final yield during the period between around 67 and 109 days after planting, with maximum *R* 2 -values of up to 0.84, 0.88, and 0.85, respectively. The developed system demonstrated relatively high throughput data collection and analysis.

Keywords: field-based high throughput phenotyping, 3D point cloud, morphologic traits, plant growth analysis, LiDAR

#### Edited by:

*Eetu Puttonen, National Land Survey of Finland, Finland*

#### Reviewed by:

*Wei Guo, The University of Tokyo, Japan Marco Maccaferri, University of Bologna, Italy*

> \*Correspondence: *Changying Li cyli@uga.edu*

#### Specialty section:

*This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science*

> Received: *30 September 2017* Accepted: *04 January 2018* Published: *22 January 2018*

#### Citation:

*Sun S, Li C, Paterson AH, Jiang Y, Xu R, Robertson JS, Snider JL and Chee PW (2018) In-field High Throughput Phenotyping and Cotton Plant Growth Analysis Using LiDAR. Front. Plant Sci. 9:16. doi: 10.3389/fpls.2018.00016*

# INTRODUCTION

The global population is estimated to approach nine billion by 2050, and demand for food and fiber crops is expected to increase by 60% (Tilman et al., 2011; Gerland et al., 2014). New plant breeding approaches need to be developed to overcome these tremendous challenges. An important step in this direction is to gain a better understanding of the relationship between genotype and phenotype (Goggin et al., 2015; Großkinsky et al., 2015; Rahaman et al., 2015). However, in-field high throughput phenotyping technologies, which can facilitate automatic measurement of phenotypic traits over the entire growing season, are still considered to be a major bottleneck limiting crop improvement (Furbank and Tester, 2011; Cobb et al., 2013).

Plant morphologic traits can often be used for evaluating plant growth (Hosoi and Omasa, 2012; Taheriazad et al., 2016), which determines plant performance in terms of final crop biomass and yield (Dhondt et al., 2013). Several studies showed that morphologic traits such as canopy height and leaf area index (LAI) were strongly related to plant species, type of cultivation, plant growth rate, and final yield (Gebbers et al., 2011; Sharma and Ritchie, 2015; Friedli et al., 2016; Sun et al., 2017). Importantly, plant growth and yield is dependent upon leaf area development, the average photosynthetic efficiency of all leaves in the plant canopy (Gardner, 1985; Krieg and Sung, 1986), and partitioning of dry matter to the harvested portion of the crop (Earl and Davis, 2003). Thus, plant canopy development should provide some indication of the crop's capacity for growth and yield.

The traditional manual measurement of plant morphologic traits is time consuming, labor intensive, and sometimes destructive. Novel technologies for plant phenotyping in a noninvasive and high throughput manner with high spatial and temporal resolution offer improved efficiency (Furbank and Tester, 2011; Dhondt et al., 2013; Großkinsky et al., 2015). Over the past decade, several non-invasive approaches using sensing technologies were developed for in-field phenotyping (Lin, 2015; Simko et al., 2016). Computer vision was one commonly-used technology. Usually, plant traits were extracted from color (RGB) images. Li et al. (2016) introduced a method for in-field cotton boll detection based on color and texture features using 2D color images. Si et al. (2015) developed a machine vision system to automatically recognize and locate apples; over 89.5% accuracy was achieved. Such 2D image based methods provide potential to conduct phenotypic measurements with a high spatial resolution, but are limited by plant occlusion. In addition, one major challenge with 2D digital image methods is that image quality is significantly affected by highly variable illumination conditions in the field, which limits automatic data processing (Li et al., 2014).

The use of 3D model-based methods for plant phenotyping are receiving increasing attention, as they permit multiple morphologic traits such as canopy height, plant volume, and LAI to be simultaneously extracted (Bietresato et al., 2016; Vazquez-Arellano et al., 2016; Gibbs et al., 2017) while mitigating plant occlusion. Moreover, 3D models have the potential to assist growers to continuously monitor and quantify plant growth and development, as well as plant responses to environmental stresses. A stereo-imaging based 3D reconstruction system was established to capture rape seedling leaf area and plant height (Xiong et al., 2017); two identical RGB cameras were utilized as an imaging unit. The mean error for leaf area and plant height measurements was 3.68 and 6.18%, respectively. The system was put in a well-designed box, in which homogenous illumination was provided. Thuy Tuong et al. (2015) developed a 3D reconstruction system based on 10 digital color cameras that were mounted on a custom structure, and an illumination system was used to enhance the visual texture of plants from all camera viewpoints. The system produced very high quality, dense, and complete point clouds. However, as both systems were designed for indoor use, they would need to be modified for field applications under natural illumination. 3D models can also be produced by time of flight (TOF) cameras; however, similarly to RGB image based methods, data quality would be significantly affected by sunlight under field conditions, which limits in-field applications. In Busemeyer et al. (2013) and Jiang et al. (2016) the TOF cameras were mounted inside an enclosure in order to mitigate the influence of sunlight.

Light detection and ranging (LiDAR) technology provides an alternative approach for 3D plant model reconstruction. LiDAR is a remote sensing technology to measure the distance between the sensor and an object of interest by illuminating the object with a laser and analyzing the TOF. LiDAR may be the best known and most widely used sensor for 3D canopy reconstruction (Deery et al., 2014; Gibbs et al., 2017). A 2D LiDAR collects two dimensional scans in a measured plane, and a 3D model can be obtained by moving the sensor along the perpendicular direction to the scanning plane. Although the spatial resolution of the 3D model produced by LiDAR is not as dense as those obtained by camera-based methods, it is sufficient for the extraction of most plant morphologic traits (Rosell-Polo et al., 2009; Bietresato et al., 2016; Sun et al., 2017). In Deery et al. (2014), a LiDAR (LMS400, SICK AG, Waldkirch, Germany) with a monochromatic red laser light source was used to generate intensity images of several crops including rice, wheat, and maize. It was concluded that LiDAR is a potential alternative to image-based methods for phenotyping morphologic traits at the plot or plant level under field conditions. Moreover, in contrast to image based methods, the LiDAR based method uses its own light source, mitigating problems with highly variable illumination conditions in the field. In addition, LiDAR can be used with a high scanning frequency and a large scanning range (Lin, 2015). Therefore, LiDAR has excellent potential for in-field plant phenotyping. 3D point clouds can also be generated by some other sensors such as triangulation line scanner (Paulus et al., 2014b) and ultrasonic sensing (Llorens et al., 2011). A very dense 3D model can be reconstructed using a triangulation line scanner and morphologic traits at the organ level could be extracted, however the relatively short working distance limits its application for large plants such as cotton.

**Abbreviations:** CH, canopy height; PCA, projected canopy area; PV, plant volume; 3PLM, three-parameter logistic model; CI, confidence interval.

Growth dynamics of plant morphologic traits provide important information toward determining plant productivity. Tessmer et al. (2013) described a high-throughput phenotyping platform for plant growth modeling and functional analysis (HPGA). Plant growth curves were generated by the platform, which were used to gain a deeper understanding of energy distribution. Friedli et al. (2016) introduced a terrestrial 3D laser scanner-based plant growth monitoring system which could be used for monitoring canopy height growth for different crops under field conditions. Paulus et al. (2014b) monitored the organspecific growth dynamics of cereal plants with a high precision triangulation line laser scanner by scanning every 2–3 days. Three morphologic traits including leaf area, stem height, and plant volume were measured, allowing quantification of the growth dynamics of the barley plant.

In the present study, three morphologic traits—canopy height, projected canopy area, and plant volume—of cotton plants were derived based on data collected by a 2D LiDAR. Morphologic trait data was collected and growth analysis was conducted. Several improvements were made over previously mentioned studies. A 1-cm accuracy level RTK-GPS was used in this study in order to provide accurate spatial coordinates for LiDAR scans so that a precise 3D surface model could be reconstructed, which is important because the model is the basic dataset for further analysis. In addition, analyses of growth dynamics and correlation of morphologic traits with final yield were conducted over the growing season. The system repeatedly scanned plants over the growing season, permitting analysis of the effects of different cultivars on the growth and final yield of cotton plants.

The overall goal of this work was to develop a high throughput phenotyping system for morphologic traits of cotton plants using LiDAR under field conditions. This LiDAR-based system is one component of our broader effort to develop field-based high throughput phenotyping (HTP) systems, and it complements other image-based sensors such as Kinect V2, thermal camera, and hyperspectral camera. This LiDAR-based system provides accurate morphologic traits in a robust and fast way, and saves storage space and computing resources compared to image-based sensors. Specific objectives were to: (1) develop algorithms to extract multiple morphologic traits—canopy height, projected canopy area, and plant volume—from a 3D point cloud obtained with a 2D LiDAR; (2) conduct 4D monitoring (3D plant reconstructions over time) of the derived plant morphologic traits to detect growth patterns of plants from different cultivars; and (3) explore relationships between derived morphologic traits and final yield.

### MATERIALS AND METHODS

#### Experimental Field

The study site was located at the Iron Horse Farm (IHF) in Greene County, GA, USA. The entire study included 128 plots arranged in 16 rows and 8 columns (**Figure 1A**), using a randomized complete block design with four cultivars of cotton and 32 replicate plots per cultivar. Four plots of each cultivar were planted in each column. The distribution of cultivars in each column was randomly assigned (**Figure 1B**). Plots were 3.05 m wide. A total of 15 seeds were sowed in each plot at spacing of 0.15 m. Inter-row spacing was 1.52 m, and inter-column spacing was 1.83 m. Cotton seeds were sowed on June 13, 2016.

The four cotton cultivars were GA2011158 (cultivar 1), GA2009037 (cultivar 2), GA2010074 (cultivar 3), and UA48 (cultivar 4) which is commercialized by the private seed company Americot. All four cultivars were developed by conventional breeding possessing no transgenic insects or herbicides tolerant traits. However, they have different fiber quality, growth habits, and plant architecture due to adaptation to different production regions. Cultivars 1, 2, and 3 are elite breeding lines developed by the University of Georgia cotton breeding program for adaptation to the southeastern cotton production region, bred to have indeterminate growth habit to take advantage of the long growing season in the southern US cotton belt. Plants from these cultivars will continue adding vegetative growth at the same time as the reproductive development, therefore they can grow excessively tall and rank in high nitrogen environment or if there are severe insects damage causing excessive square loss. Cultivar 4, on the other hand, was released by the University of Arkansas cotton breeding program, bred to have an early maturity growth habit for adaptation to the northern region of the US cotton belt. It has a determinate growth habit which resulting in shorter statured, extended sympodial branches, and shorter flowering date.

### Data Acquisition

The data collection system mainly consisted of a LiDAR (LMS 511 PRO SR, SICK AG, Waldkrich, Germany) (**Figure 2A**), an RTK-GPS (Cruizer II, Raven Industries Inc., Sioux Falls, SD, USA) (**Figure 2B**), and a rugged laptop as a DAQ (data acquisition) and storage device. The LiDAR was mounted on a tractor (Spider DL, LeeAgra, Inc., Lubbock, TX, USA) platform at a height of 2.4 m to scan cotton plots from directly above (**Figure 2C**). The RTK-GPS was mounted on the roof of the tractor, which provided the spatial coordinates during data collection. The LiDAR and RTK-GPS receiver were aligned to the center of the tractor (**Figure 2D**).

The LiDAR was developed for outdoor use, and measured in 2D radial coordinates from −5 to 185◦ with a maximum range of 80 m. Line scans could be acquired at a rate of 25–100 Hz with an angular resolution of 0.1667–1◦ . The built-in filters eliminated interference from particles of dust, raindrops, and snowflakes. An enclosure was used to provide a controlled environment for data acquisition. The RTK-GPS provided coordinates with 1 cm accuracy with an update rate up to 10 Hz.

Data collections were conducted in the field from July 26 to September 30, 2016, i.e., from 43 to 109 days after planting (DAP) (**Table 1**). When scanning plants, the angular resolution of the LiDAR was configured to be 0.33◦ with a sampling frequency of 50 Hz, and the echo filter and particle filter were enabled. The update frequency of the RTK-GPS was 5 Hz. The tractor scanned the field row by row. For each row, the tractor traveled from column 1 to column 8 with an average speed of about 0.5 m/s. Since the first two sample dates—July 26 and 28—were close to each other, the data for July 26 were not presented except for growth rate analysis.

#### Data Processing Algorithms

After raw data was collected in the field, further processing and analysis were performed in the lab. The data processing and analysis program was developed and implemented in MATLAB 2016b (The Math Works Inc., Natic, MA, USA) on a desktop equipped with an intel I7-6700 CPU 3.40 GHz with 16 GB RAM, running on a Windows 10 Enterprise operating system.

Two steps were executed to derive plant features (**Figure 3**): 3D plant surface model generation (section Generation of 3D Model) and morphologic plant parameter extraction (section Extraction of Morphologic Traits).

#### Generation of 3D Model

The 3D model for each row was reconstructed based on GPS and LiDAR data. The GPS and LiDAR dataset was depicted by Equation (1). The two kinds of data were synchronized using timestamps.

$$\begin{array}{lcl}P\_{GPS} &=& \left\{P\_{GPS0}, P\_{GPS1}, \dots, P\_{GPS(N-1)}\right\} \\ F\_{LiDAR} &=& \left\{F\_{LiDAR0, } F\_{LiDAR1, \dots, } F\_{LiDAR(M-1)}\right\} \end{array} \tag{1}$$

**P**GPS was the set of collected GPS data, and **F**LiDAR was the set of scanned frames of LiDAR. The number of GPS points was N, and the number of LiDAR frames was M.



The distance between two adjacent GPS points, denoted by 1**P**GPS, was computed by Equation (2). f LiDAR and f GPS were the data acquisition frequency of LiDAR and GPS, respectively. In this study, the data acquisition frequency of GPS was f GPS = 5 Hz, and LiDAR scanning frequency was f LiDAR = 50 Hz. Therefore, there were 10 scanned frames, each containing 571 points (the aperture angle was 190◦ with angular resolution 0.33◦ ) between every two adjacent GPS points (Equation 3). Assuming that the tractor was moving at a constant speed during the interval (200 ms) of two adjacent GPS points, the distance of the two adjacent frames within two adjacent GPS points was computed using Equation (4). Therefore, the position of each LiDAR scanned frame was obtained using Equation (5). **D**offset was the offset between LiDAR and GPS. In this study, **D**offset was fixed during data collection in the field, and the measured point at 0◦ scanning angle was used to depict the frame position.

$$\begin{cases} \Delta \vec{P}\_{\text{GPS}}(i) = P\_{\text{GPS}}(i) - P\_{\text{GPS}}(i-1), & i = 1, 2, \dots \\ \Delta \vec{P}\_{\text{GPS}}(0) = \vec{0}, & i = 0 \end{cases} \tag{2}$$

FIGURE 3 | Data processing pipeline. (A) Example of 3D point cloud of one scanned row; (B) segmented 3D point cloud of a plot indicated by cuboid with solid red lines in (A); (C) point cloud of ground plane; (D) point cloud of plants; (E) denoised point cloud of plants; (F) height profile of plants and maximum canopy height; (G) projected canopy area; (H) plant volume.

$$\alpha = \frac{f\_{\text{LiDAR}}}{f\_{\text{GPS}}} \tag{3}$$

$$\vec{d}\_{\rm i} = \begin{array}{c} \Delta \vec{P}\_{GPSi} \\ \alpha \end{array}, i = 0, 1, 2, \ldots \\ N - 1 \tag{4}$$

$$f\_{\text{LiDAR}}(k) = \begin{array}{c} f\_{\text{LiDAR}}(i\alpha + j) = p\_{\text{GPS}}(i) + j \times \vec{d}\_i + \vec{D}\_{\text{offset}} \\\ i = 0, 1, 2, ... \\\ \end{array} \tag{5}$$
 
$$i = \ 0, 1, 2, ... \\ N - 1, j = 0, 1, 2... \\\ \end{array} \\ \begin{array}{c} \text{(5)}$$

More details related to this processing can be found in Sun et al. (2017). **Figure 3A** shows an example of the reconstructed 3D model of one scanned row. A distance filter was used to remove points hitting the enclosure. A plot level 3D model was extracted from the 3D model of the row according to the proportions of the field layout (**Figure 3B**). The standard RANSAC algorithm was applied to cut-off points of the ground plane (**Figures 3C,D**). An Euclidean clustering algorithm (Rusu et al., 2008) was used to remove points generated by weeds that were not attached to cotton plants (**Figure 3E**).

#### Extraction of Morphologic Traits

The maximum canopy height (CH) was measured by calculating the distance from the ground plane to the apex of all measured points (**Figure 3F**). In addition, different percentiles canopy height—from 85th to maximum CH with steps of 3%—were calculated based on the histogram of the z coordinate for all measured points. The boundary points of the plant canopy were detected by projecting all points onto the ground plane, and the projected canopy area (PCA) and bounding box of canopy structure (representing maximum length and width occupied by the canopy) were extracted from boundary points (**Figure 3G**). A Trapezoidal rule based algorithm was used to calculate plant volume (PV) (**Figure 3H**) in order to provide an indication of the 3D space occupied by each plot.

**S** denoted the line scan set for a plot (Equation 6) which contained k line scans. For a scan s<sup>i</sup> which contained n measured points, the dashed line was the surface profile and the red spots were measured points by LiDAR (**Figure 4**). The area denoted as A<sup>i</sup> under the line scan s<sup>i</sup> could be estimated using the measured points (red spots in **Figure 4**) based on Trapezoidal rule (Equation 7), where (x<sup>i</sup> , yi) were the coordinates of the ith measured point.

$$\mathcal{S} = \{s\_1, s\_2, \dots, s\_k\} \tag{6}$$

$$A\_i = \sum\_{j=1}^{n-1} \left(\mathbf{x}\_{j+1} - \mathbf{x}\_j\right) \left(\frac{\mathbf{y}\_{j+1} + \mathbf{y}\_j}{2}\right) \tag{7}$$

Therefore, the area set denoted as **A** for the plot was obtained using Equation (8).

$$A = \{A\_1, A\_2, \dots, A\_k\} \tag{8}$$

Similar to the area calculation process, PV was obtained with Equation (9).

$$\text{PV} = \sum\_{i=1}^{k-1} \left( l\_{i+1} - l\_i \right) \left( \frac{A\_{i+1} + A\_i}{2} \right) \tag{9}$$

l<sup>i</sup> was the position along the tractor moving direction of the ith line scan.

#### Validation Experiments

To verify the accuracy of canopy height measurements using point cloud data, 96 samples of maximum CH—the perpendicular distance from the highest point to the ground plane—were manually measured using a tape measure during data collection in the field. The samples were measured on four different days: July 28, August 04, August 26, and September 09, 2016. On those days, the average wind speeds were around 2.7, 3.4, 2.3, and 2.5 m/s, respectively.

For PCA validation experiments, eight "dummy plants" were made with different canopy shapes using printed paper leaves and metal wires. The dummy leaves were in three different sizes–10, 41, and 92 cm2–and there were six of each size. The leaves of two dummy plants were arranged to be overlapped. Each dummy plant was scanned by LiDAR and imaged by a DSLR camera (FUJIFILM X-A10 mirrorless camera, FUJIFILM, Tokyo, Japan). A scale bar was used to calibrate the real size of the plant. The ground truth of PCA was computed by segmenting leaves based on color information.

For PV validation experiments, eight plants at different growth stages were used, among which five were real cotton plants, one was a shrub plant, and two were dummy plants. The six real plants were in the leaf and canopy development growth stage, whereas the two dummy plants were used to mimic flowering and boll development growth stages. Each plant canopy was divided into 50 mm cylindrical discs from the top to the bottom and the diameter of each disc was manually measured using a tape measure. The diameter and height of each cylindrical disc were used to estimate the volume of each disc, and then the volume of the whole plant was obtained by summing the volumes of each disc. Manual measures of the three traits were plotted vs. point cloud estimates, and regression analysis was used to compute the root mean square error (RMSE) and coefficient of determination (R 2 ). More information about PCA and PV validation experiments can be found in supplementary materials.

#### Plant Growth Analysis

Plant growth dynamics were obtained by monitoring plants over the growing season when plant morphologic traits were extracted. Growth trends and growth rates for each morphologic trait noted above (CH, PCA, PV) were computed and analyzed over the monitoring period. Growth trends were determined as the variation in measured traits over the monitoring period. The results were given as mean values and standard deviations of all morphologic parameters. A three-parameter logistic model (3PLM) was used to fit growth curves of the three detected traits for each cultivar (Tessmer et al., 2013). The model is a function of time t as shown in Equation (10).

$$\chi(t) = \frac{\varkappa\_0 \varkappa\_n}{\varkappa\_0 + (\varkappa\_n - \varkappa\_0)e^{-\tau(t-T)}} \tag{10}$$

where t was the time which was denoted by days after planting in this study, τ was a coefficient, y(t) was detected traits at time t, x0, x<sup>n</sup> were the initial value and the upper horizontal asymptote of the detected trait, respectively, and T was the first day of data collection which was also denoted by days after planting (in this study T = 45). x0, xn, and τ can be estimated using non-linear least squares based on the observations of the detected traits.

Growth rate was determined as the average change in measured traits over a time interval. Growth rate was calculated by Equation (11).

$$GR = \frac{P\_t - P\_{t-\Delta t}}{\Delta t} \tag{11}$$

where GR was the growth rate, 1t was the time internal, and P<sup>t</sup> was the measured plant trait at time t.

#### Correlation Analysis between Morphologic Traits and Yield

Seed cotton (mature fiber plus seeds to which it was attached) was harvested manually on November 4, 2016, and yield was expressed as g/plot. In order to explore the relationship between derived morphologic traits and final yields, linear regression analysis was conducted for each cultivar over the monitoring period. The coefficient of determination (R 2 ) was computed.

color images taken from above. The second row shows the reconstructed 3D point clouds of cotton plants. The third row shows height maps of cotton plants, obtained by projecting all points on the ground plane (color indicates height).

# RESULTS

# Morphologic Traits Extraction and Validation

Morphologic trait information of cotton plants was derived from the reconstructed 3D point clouds. **Figure 5** shows the 3D point clouds over time compared to the 2D color images of the same plot. The reconstructed 3D point clouds on August 19, August 26, and September 9, respectively, were obtained using 207, 256, and 263 scans and contained 11,403, 14,379, and 14,485 points. The cuboids indicated by solid red lines for graphs in row 2 of **Figure 5** are 3D bounding boxes and points within the cuboids belong to plants. The rectangles indicated by red solid lines for graphs in row 3 of **Figure 5** are 2D bounding boxes, and the dashed red lines are the detected boundaries from which PCA was derived. Morphologic traits including CH, PCA, and PV were extracted from the 3D point clouds.

CH-, PCA-, and PV-values derived from LiDAR data were highly correlated with manually measured values, with R 2 -values of 0.97, 0.97, and 0.98 and RMSE values of 0.03 m, 0.007 m<sup>2</sup> , and 0.011 m<sup>3</sup> , respectively (**Figure 6**).

**Figure 7A** shows the side view of the point cloud of the plot presented in **Figure 5**. The data was collected on August 19 and consisted of 207 line scans. The number of measured points for each line scan varied depending on the width of plants. A total of 66 points were measured for the 144th scan (**Figure 7B**). Points near the center were denser than points located at two terminal areas because the inter-distance between two adjacent points increased with increasing distance between the LiDAR and measured points. A minimum threshold of 0.05 m was applied to remove points. Based on the proposed PV calculation method, the area under the profile and PV were estimated: A<sup>144</sup> = 0.52 m2 , PV = 2.33 m<sup>3</sup> (**Figure 7C**).

#### Plant Growth Analysis

Overall, the three measured morphologic traits—CH, PCA and PV—showed similar growth trends over the monitoring period based on the measurements from the sensor, but the actual value was cultivar specific for each trait (**Figure 8**). For maximum CH, all cultivars reached the maximum height on around day 88. Cultivar 1 and 2 had similar average maximum heights of 1.08 m, while cultivars 3 and 4 reached the peak values of 0.96 and 0.88 m,

respectively. Cultivar 1 and 2 were around 22.7% higher than cultivar 4. PCA continued to increase until around 95 DAP, which was 7 days longer comparted to maximum CH. The maximum PCAs of cultivars 1, 2, 3, and 4 were 2.73, 2.23, 2.47, and 2.34 m2 , respectively. Cultivar 1 had much larger PCA-value than the other three cultivars which had similar PCA. It is around 22.4% larger than cultivar 2 which had the minimum area. Cultivar 2 showed the highest CH, but lowest PCA over the whole monitoring period due to less horizontal canopy expansion in the horizontal direction. For the growth curve of PV, the maximum points for all four cultivars were reached on day 88, they were 2.75, 2.17, 2.11, and 1.59 m<sup>3</sup> for cultivar 1, 2 3, and 4. Larger differences among cultivars were observed for PV than PCA. The maximum volume was around 73.0% larger than the minimum one. In summary, cultivar 1 had the largest canopy (highest PV, CH, and PCA) while cultivar 4 had the lowest maximum CH and PV. Although cultivar 2 had high maximum CH, its PCA was the lowest, making its PV in the middle range among the cultivars. The derived values and variation of morphologic traits over time were in good agreement with the developmental phases of cotton (Ritchie et al., 2007).

Growth curves generated for each cultivar of all three detected traits showed good correlation with the sensor measurements. The date when the traits reached their upper horizontal asymptotes and the values of their upper horizontal asymptotes were similar to the results obtained from measured data. The growth curve varied little after reaching the upper horizontal asymptotes, while the sensor measurements showed a decreasing trend which was mainly due to defoliation. Supplementary Table 1 presented estimated parameters of the 3 PLM and their 95% confidence interval (CI).

**Figure 9** shows the growth rates of maximum CH, PCA, and PV at different periods of time. For maximum CH, cultivars 1 and 2 had similar growth rates, and grew faster than cultivars 3 and 4 during the period from P1 to P5. The peak of GR of cultivar 1 and 2 was observed during P3 (between 52 and 67 DAP), which was around 0.018 m per day (m/d); However, it was during P2 for cultivar 3 and 4, which was around 0.015 m/d (**Figure 9A**). For PCA, all four cultivars had similar GR during P2 and P3; However, large differences emerged during the period from P3 to P6. The peak of GR for cultivar 1 and 3 was in P4, with the values of 0.065 m<sup>2</sup> /d; while it was in P2 for cultivar 2 and 4, with the same value of 0.054 m<sup>2</sup> /d. PCA increased until P6 (95 DAP) and then started to decrease (**Figure 9B**), 21 days later than the period when the maximum CH started to decrease. During the PCA increasing period, cultivar 1 grew faster than the other three.

PV had a faster GR than the other three cultivars during P1 to P4—it reached the peak in P4 with the value of around 0.1 m<sup>3</sup> /d. Cultivar 2, 3, and 4 reached the peak of GR in P4 (0.06 m<sup>3</sup> /d), P4 (0.07 m<sup>3</sup> /d), and P3 (0.05 m<sup>3</sup> /d), respectively. In summary, the fastest GRs for all three traits were during P2 to P4, that was from 45 to 74 DAP, which was also indicated by the growth curves in **Figure 8**.

ANOVA-tests showed that cultivars had a significant influence on the derived morphologic traits over the monitoring period (**Table 2**). This allowed us to utilize regression analysis to determine which morphologic traits and measurement times were most closely associated with final yield for each cultivar.

### Relationship between Morphologic Traits and Yield

A significant difference in the final yield was observed between the four cultivars (**Table 3**). Cultivar 2 produced significantly lower yields than cultivar 3 and 4.

The relationship between CH and final yield showed similar trends using data from various CH percentiles, especially between the 85th and 94th percentiles. The R 2 -values for cultivar 1, 3, and 4 increased over the monitoring period although there existed only slight variation, whereas cultivar 2 exhibited a decreasing trend (**Figure 10**). Among the cultivars studied, cultivar 4 had the highest R 2 -values (up to 0.84) from day 67 on, based on the 85th CH to maximum CH. The variation curve of cultivar 1 was analogous to the curve for cultivar 3, both reaching the highest R 2 -values between 88 and 95 DAP. In contrast, cultivar 2 demonstrated the highest correlation with yield at an early growth stage around 52 DAP. Based on the results from this study, percentiles from 85 to 94% of CH during the period from 67 to 109 DAP are recommended for yield estimation application due to not only high but also stable R 2 -values for all four cultivars.

Overall, the R 2 -values between PCA and final yield increased over the monitoring period for all four cultivars (**Figure 11A**). The R 2 -values for cultivars 2, 3, and 4 on each data collection day were similar, while cultivar 1 exhibited lower R 2 -values. Compared to CH parameters, a major difference was that cultivar 2 had an opposite variation trend. The highest R 2 -values for all four cultivars were reached between 88 and 109 DAP, and they were 0.65, 0.83, 0.87, and 0.88 for cultivars 1, 2, 3, and 4, respectively. This indicated that PCA was more closely related to final yield than CH. For both CH and PCA, cultivar 4 showed the strongest correlation with final yield among the four cultivars.

For PV, the difference in R 2 -values between the four cultivars became smaller over the monitoring period compared to CH and PCA, especially between 88 and 102 DAP (**Figure 11B**). The maximum R 2 -values were 0.77, 0.85, 0.84, and 0.83 on 95, 67, 74, and 74 DAP for cultivars 1, 2, 3, and 4, respectively, which were similar to the values for PCA, and better than CH. This indicated that PV was a more stable trait than CH and PCA and could feasibly be used to predict cotton final yield.

PCA and PV had higher R 2 -values than CH over the monitoring period (**Figure 12**) when combining all cultivars. The R 2 -values increased steadily to a final value of 0.72 for PCA. PV

TABLE 2 | Effects of cultivars on derived parameters over the monitoring period (Year: 2016).


\**Significant at the 0.05 probability level,* \*\**Significant at the 0.01 probability level.*

TABLE 3 | Differences in final yield between four cultivars.


*p* < *0.05. Different letters indicate significant differences between cultivars.*

reached the highest R 2 -value (0.56) on 67 DAP—although the PV R 2 -values were less than those of PCA from 67 to 109 DAP, they were more stable than those of PCA. The lowest correlation was found between CH and final yield. A decreasing trend was observed from 45 to 109 DAP which was mainly due to cultivar 2.

#### DISCUSSION

We have demonstrated that 3D point clouds reconstructed based on data collected by a 2D LiDAR and an RTK-GPS, and associated data processing methodology, accurately estimated specific morphologic traits from cotton plants under field conditions. The precise 3D point cloud, which was the basic dataset for analysis, was critical for the successful extraction of plant morphologic traits (Paproki et al., 2012; Paulus et al., 2014b; Duan et al., 2016). The reconstructed 3D model visually represented plant canopy structures (**Figure 5**). The CH, PCA, and PV derived by our system were highly correlated (the slopes were close to one) with those measured manually (**Figure 6**).

The system demonstrated great potential for high-throughput phenotypic analysis. The number of points for each plot varied from 2,000 to 20,000 over the monitoring period, depending on the plant size. The tractor was driven in the field at a speed of

around 0.5 m/s for data collection, taking about 6 s to scan one plot that contained 15 plants. Thus, one could anticipate scanning about 600 such plots per hour. The average time consumed for 3D reconstruction and parameter extraction was around 5.27 s per plot so about an hour for the aforementioned 600-plot field. Therefore, both data acquisition and data analysis speeds are suitable for application to field experiments of the sizes used in many breeding programs. The system also has great potential to implement online data analysis with faster computing power in the future, which is particularly useful for large field applications.

The high throughput phenotypic analysis by a LiDARbased method has several advantages compared to imagebased methods. Paproki et al. (2012) conducted cotton plant morphologic trait analysis using an image-based 3D reconstruction method, requiring about 7 min to collect images for each plant and an average processing time of 15 min for 3D reconstruction work of each plant. Additionally, image quality could be significantly affected by highly variable illumination conditions (Nuske et al., 2014), which limited its in-field applications. In contrast, LiDAR is more versatile in a wide range of light conditions since it is equipped with its own light source. For manual operation, ∼30 min were required for a typical analysis per plant depending on the size and complexity, and destructive harvests were often required (Paproki et al., 2012).

The high throughput of the proposed system and its noninvasive features permit data acquisition repeatedly over the growing season, opening the door to acquisition of plant growth rate variation, which is valuable for a wide range of applications such as building plant growth models (Tessmer et al., 2013; Weraduwage et al., 2015), exploring factors influencing growth processes (Rahaman et al., 2015; Awlia et al., 2016), genomicsassisted crop breeding (Watanabe et al., 2017), and QTL analysis

(Bac-Molenaar et al., 2015; Cui et al., 2017). In this study, all plants in the experimental field were scanned from 43 to 109 DAP. Plant growth curves were generated and correlation between derived parameters and the final yield were explored, which could be used for yield prediction (Sharma et al., 2015) and species identification (Remagnino et al., 2016). Thus, the system described herein not only provides an indication of which LiDAR-derived parameters are most closely associated with yield, but also allows us to define the period during the growing season in which a morphologic trait has the greatest correlation with final yield.

While CH in the current study was highly correlated with within-cultivar yield variation in some instances (i.e., cultivar 4 on later sample dates), it should be noted that correlations between CH and yield for all cultivars combined were substantially lower than those obtained for other derived parameters such as PCA and PV. This is not surprising since the cotton crop exhibits a quadratic response of yield to CH (Sui et al., 2012), where internode elongation must often be controlled using exogenously applied plant growth regulators (PGRs) in real-world production scenarios (Dodds et al., 2010). While CH may not be inherently predictive of genotypic differences in yield, the deployment of high-throughput methods for estimating maximum CH and rate of change in CH could strongly influence cultivar-specific PGR management strategies. Specifically, cultivars with rapid plant height development typically require more aggressive management strategies (Collins, 2013).

PCA and PV were more strongly associated with yield than CH when considered across all cultivars, suggesting these parameters might have greater utility as high-throughput phenotyping tools to identify potential differences in productivity. The impact of canopy development on yield is easily understood when yield is expressed as the product of total intercepted photosynthetically active radiation, radiation use efficiency, and harvest index (Monteith and Moss, 1977; Monteith, 1994; Earl and Davis, 2003; Stöckle and Kemanian, 2009). Within this conceptual framework, the amount of intercepted photosynthetically active radiation during a growing season will be strongly impacted by the length of the growing season and the leaf area available to intercept solar radiation. Thus, PCA may serve as a suitable proxy for leaf area magnitude and persistence throughout the growing season. PV incorporates both CH and PCA, which could potentially be related to both the amount of leaf area available to intercept incoming solar radiation and the three dimensional space available for developing fruit. Both of these considerations are important given the indeterminate nature of cotton. As fruit development progresses, vegetative growth slows, and Constable and Bange (2015) have clearly illustrated that the ability of the cotton crop to attain high yield potential will be influenced by total leaf area and number of fruiting sites available prior to the development of a large fruiting load on the plant. Thus, it is not surprising that PCA and PV were correlated with yield for much of the growing season. Future work should couple the aforementioned measures with high-throughput assessment of fruit development as fruiting dynamics can drastically impact final yield in cotton (Constable and Bange, 2015).

The top view scan setting of LiDAR provided precise 3D surface models of plants; however, organs under the canopy could not be reached due to plant occlusion. This does not reduce the accuracy of CH and PCA measurements, but hinders the estimation of PV by our system, especially when plant structure becomes complex. Multi-view scan is a commonly used method to reduce plant occlusion effects (Paulus et al., 2014b), and in our future studies two side view scans could be added. In addition, algorithms such as plant shape models (Pastrana and Rath, 2013) have been proposed to overcome plant occlusion effects. However, when plant structure complexity increases, especially at the mature stage with a greatly increased number and size of leaves and stems, occlusion effects would still be present. Plant occlusion remains a challenge for plant phenotyping, especially under field conditions with limited inter-row spacing (Paproki et al., 2012; Paulus et al., 2014a). Wind is another factor which could affect the accuracy of the derived traits since it might result in blurred point clouds. Therefore, calm weather conditions are best for data collection. Based on the CH validation results, our system has a certain robustness against wind influence; wind speed under 3 m/s was feasible. Another limitation was that the tractor speed was restricted by the scanning frequency of the LiDAR (Sun et al., 2017), limiting throughput. While the present system is already compatible with iterative study of thousands of plots, 3D LiDAR technology may greatly reduce this limitation (Weiss and Biber, 2011).

# CONCLUSION

Precise 3D surface models were reconstructed by the highthroughput phenotyping system developed in this study under field conditions. Multiple morphologic traits at the plot level including plant height, projected canopy area, and plant volume were extracted simultaneously. The system could be used to scan the field repeatedly due to its relatively high data collection and

# REFERENCES


processing capability, which was particularly useful for large field applications. The measured morphologic traits were most highly correlated with final yield during the period between around 67 and 109 DAP. Projected canopy area and plant volume were more closely correlated than plant height to final yield. Future work will focus on involving other sensor data to extract more phenotypic traits from the 3D point cloud. Although this system was solely tested on cotton plants, it is expected to be applicable for use with other crops such as wheat, rice, and soybeans.

# AUTHOR CONTRIBUTIONS

SS, CL, and AP: conceived the idea and designed the experiments; AP and PC: contributed to the preparation of materials and equipment; SS, YJ, RX, and JR: conducted the experiments; SS, YJ, and RX: analyzed the data; SS, CL, AP, JS, and PC: interpreted results and wrote the paper; All authors read and approved the final manuscript.

# FUNDING

This study was funded jointly by the Agricultural Sensing and Robotics Initiative of the College of Engineering, and the College of Agricultural and Environmental Sciences of the University of Georgia. The project was also partially supported by the National Robotics Initiative grant (NIFA grant No. 2017-67021-25928).

# ACKNOWLEDGMENTS

The authors gratefully thank Tariq Shehzad, Jeevan Adhikari, Jamal Hunter, Mengyun Zhang, Zikai Wei, and Shuxiang Fan for their assistance in conducting the experiments.

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpls.2018. 00016/full#supplementary-material


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Sun, Li, Paterson, Jiang, Xu, Robertson, Snider and Chee. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Considerations for Achieving Cross-Platform Point Cloud Data Fusion across Different Dryland Ecosystem Structural States

Tyson L. Swetnam<sup>1</sup> \*, Jeffrey K. Gillan<sup>2</sup> , Temuulen T. Sankey <sup>3</sup> , Mitchel P. McClaran<sup>2</sup> , Mary H. Nichols <sup>4</sup> , Philip Heilman<sup>4</sup> and Jason McVay <sup>3</sup>

*<sup>1</sup> BIO5 Institute, University of Arizona, Tucson, AZ, United States, <sup>2</sup> School of Natural Resource and Environment, University of Arizona, Tucson, AZ, United States, <sup>3</sup> Informatics and Computing Program, Northern Arizona University, Flagstaff, AZ, United States, <sup>4</sup> USDA Agricultural Research Service, Southwest Watershed Research Center, Tucson, AZ, United States*

#### Edited by:

*Alexander Bucksch, University of Georgia, United States*

#### Reviewed by:

*Jinhu Wang, Delft University of Technology, Netherlands Andrew M. Cunliffe, University of Edinburgh, United Kingdom*

\*Correspondence: *Tyson L. Swetnam tswetnam@email.arizona.edu*

#### Specialty section:

*This article was submitted to Technical Advances in Plant Science, a section of the journal Frontiers in Plant Science*

> Received: *09 September 2017* Accepted: *04 December 2017* Published: *10 January 2018*

#### Citation:

*Swetnam TL, Gillan JK, Sankey TT, McClaran MP, Nichols MH, Heilman P and McVay J (2018) Considerations for Achieving Cross-Platform Point Cloud Data Fusion across Different Dryland Ecosystem Structural States. Front. Plant Sci. 8:2144. doi: 10.3389/fpls.2017.02144* Remotely sensing recent growth, herbivory, or disturbance of herbaceous and woody vegetation in dryland ecosystems requires high spatial resolution and multi-temporal depth. Three dimensional (3D) remote sensing technologies like lidar, and techniques like structure from motion (SfM) photogrammetry, each have strengths and weaknesses at detecting vegetation volume and extent, given the instrument's ground sample distance and ease of acquisition. Yet, a combination of platforms and techniques might provide solutions that overcome the weakness of a single platform. To explore the potential for combining platforms, we compared detection bias amongst two 3D remote sensing techniques (lidar and SfM) using three different platforms [ground-based, small unmanned aerial systems (sUAS), and manned aircraft]. We found aerial lidar to be more accurate for characterizing the bare earth (ground) in dense herbaceous vegetation than either terrestrial lidar or aerial SfM photogrammetry. Conversely, the manned aerial lidar did not detect grass and fine woody vegetation while the terrestrial lidar and high resolution near-distance (ground and sUAS) SfM photogrammetry detected these and were accurate. UAS SfM photogrammetry at lower spatial resolution under-estimated maximum heights in grass and shrubs. UAS and handheld SfM photogrammetry in near-distance high resolution collections had similar accuracy to terrestrial lidar for vegetation, but difficulty at measuring bare earth elevation beneath dense herbaceous cover. Combining point cloud data and derivatives (i.e., meshes and rasters) from two or more platforms allowed for more accurate measurement of herbaceous and woody vegetation (height and canopy cover) than any single technique alone. Availability and costs of manned aircraft lidar collection preclude high frequency repeatability but this is less limiting for terrestrial lidar, sUAS and handheld SfM. The post-processing of SfM photogrammetry data became the limiting factor at larger spatial scale and temporal repetition. Despite the utility of sUAS and handheld SfM for monitoring vegetation phenology and structure, their spatial extents are small relative to manned aircraft.

Keywords: lidar, terrestrial laser scanning, structure from motion, sUAS

# INTRODUCTION

Measurement and monitoring of ecological processes are limited by what Levin (1992) termed the 'problem of pattern and scale' where linking observations across cells, leaves, plants, community, and ecosystem require exponential amounts of information be transferred between fine and broad spatial scale, short and long temporal scale. Conventional ecological research studies require tens to hundreds of small quadrats or plots between 1 and 100 meters square (m<sup>2</sup> ) for enough observations to ensure statistically significant p-values for testing hypotheses (Huenneke et al., 2001; Kachamba et al., 2017). This change in scale from individual plot to ecosystem inventory is perhaps the most important concept of remote sensing in the life sciences (Woodcock and Strahler, 1987; Turner, 1989; Turner et al., 1989).

Dryland ecosystems, characterized by sparse or patchy vegetation and long periods of senescence, are punctuated by short periods of rapid growth following seasonal precipitation events. Therefore, characterization of both spatial and temporal patterns of vegetation abundance requires both high spatial resolution and temporal repetition over short periods of time. For instance, landscape-scale models of primary production and carbon-uptake should be more accurate when the vegetation abundance is frequently represented at a fine, plant-scale spatial resolution. Recent advances in sensor and platform technology appear to bring these resolutions within reach, but better understanding of detection bias and accuracy are needed, as well as examining the potential for combining multiple sources in a data-fusion approach that builds on strengths of one system to avoid the weakness of the other. For example, measuring the height and volume of an individual tree or shrub requires accurate minimum elevation (bare ground), apical leader, and crown diameter measurements. Given these requirements, the data may become unusable if bare ground is obstructed or incompletely illuminated. In these situations, multiple observations from different platforms may be required to generate a better estimate.

Light detection and ranging (lidar) is the predominant technology for measuring vegetation and earth surface phenomena in three dimensions (3D) (Glennie et al., 2013; Harpold et al., 2015). Manned aircraft equipped with lidar now survey local areas to entire ecosystems or countries (Higgins et al., 2014; Stoker et al., 2016), while terrestrial lidar is used to collect data at sub-centimeter resolution over hectare size areas. Critically, the spatial scale needed for managing at landscape scale can be provided by manned aerial lidar but the scale needed for monitoring ecosystem process often requires resolution that can only be provided by terrestrial lidar.

Advancements in computational processing power and machine vision also allow 3D object reconstruction from nadir and oblique 2D imagery by so-called structure from motion (SfM) photogrammetry and multi-view stereo (MVS) methods (Westoby et al., 2012; Carrivick et al., 2016; Wallace et al., 2016; Mlambo et al., 2017), hereafter referred to simply as "SfM." Under the right conditions small unmanned aerial systems (sUAS) with SfM can generate sub-centimeter precision point clouds and create digital elevation models (DEM) and digital surface models (DSM) (Dandois and Ellis, 2010; Dandios and Ellis, 2013; Cruzan et al., 2016). Cruzan et al. (2016) give excellent examples of the capability of small and micro UAS in plant ecology.

In the present study, we compare detection biases amongst different 3D remote sensing techniques (lidar and SfM photogrammetry), from three platforms (ground, sUAS, and manned aircraft) in various dryland ecosystem structural states. We combine the new and less expensive technologies, specifically sUAS SfM, with manned aerial lidar to produce more representative measurements of vegetation height and volume. The objectives of the paper are (1) to identify detection biases amongst technologies for four general feature classes: bare earth (both in barren areas and beneath vegetation), herbaceous plants, low woody shrubs, tall woody shrubs and trees, and (2) explore the potential for combining or fusing observations from any two platforms to overcome the detection bias of another. For Objective 1, we use point cloud-to-cloud inter-comparison and raster-based differencing to establish the detection bias. For Objective 2, we use raster-based differencing and also meshto-point cloud differencing to establish improved accuracy by combining data sources.

Current trends in research are moving toward collaborative open data and code sharing (Boulton et al., 2011; Kitchin, 2014; Hampton et al., 2015). Version control systems, like GitHub, offer common repositories for software and code development. In this paper, we provide only abbreviated descriptions of the technical details involving the image processing and point cloud alignment workflows in our main text, and instead provide those details in a supplemental public GitHub repository: https://github.com/ tyson-swetnam/srer-wgew. We also provide a quick reference table for acronyms and abbreviations used in this text in the supplemental materials (Table S1).

### METHODS

## Study Areas and Species

The Walnut Gulch Experimental Watershed (WGEW) is a long-term research site surrounding the town of Tombstone, Arizona (30.74◦ N, −110.05◦ W) (**Figure 1**) administered by the United States Department of Agriculture, Agricultural Research Service Southwest Watershed Research Center. The soils in WGEW vary from high in carbonates on the western lower watershed to sandy gravel loams on the eastern upper watershed (Keefer et al., 2008; Osterkamp, 2008; Pelletier et al., 2016). Mean annual temperature in WGEW, measured in Tombstone at 1,384 m a.s.l., is 17.6◦C and mean annual precipitation is 300 mm year−<sup>1</sup> . The Santa Rita Experimental Range (SRER) near Tucson, Arizona (31.80◦ N, −110.84◦ W) (**Figure 1**) is administered by the University of Arizona (http:// cals.arizona.edu/srer) (McClaran et al., 2003). The SRER soils are characterized as clay loams, sandy loams, and limey upland soils (Brekenfeld and Robinett, 1997). Mean annual temperature and precipitation are 19◦C and 358 mm year−<sup>1</sup> , respectively (McClaran and Wei, 2014). Both study sites are typical of dryland ecosystems, characterized as regions where evaporation is 2–5 times greater than precipitation (Safriel et al., 2006).

FIGURE 1 | The Santa Rita Experimental Range and Walnut Gulch Experimental Watershed boundaries are shown in black. In the lower left panel the gray triangle is the UA-C pasture, in the lower right panel the two black squares are Lucky Hills Shrubland and Kendall Grassland sites.

At WGEW, we worked in two areas: Lucky Hills Shrubland and Kendall Grassland (**Figure 1**). Lucky Hills is characterized as Chihuahuan desert scrub with shrubs as the dominant life form (Scott et al., 2006; Scott, 2010b). Common species are Larrea tridentata (creosote), Vachellia vernicosa (whitethorn acacia), Flourensia cernua (tarbush), Parthenium incanum (mariola), Rhus microphylla (little-leaf desert sumac), Condalia warnockii (Warnock's Snakewood), Ephedra triferca (Mormon tea). The Kendall Grassland site is characterized as semi-arid desert grassland with a mixture of shrub, grass and cactus life forms (Scott et al., 2006; Scott, 2010b). Common species are Eragrostis lehmanniana (Lehmann's lovegrass) a non-native grass, and Prosopis velutina (velvet mesquite), native grasses include Hilaria belangeri (curly mesquite), Bouteloua eriopoda (black grama), B. hirsuta (hairy grama), and Aristida hamulosa (threeawn) (Skirvin et al., 2008), other species include Yucca baccata (Banana yucca), Y. elata (Soaptree yucca), and Agave palmeri (Palmer's agave).

At SRER, we worked in another semi-desert grassland area with mixture of shrubs, grass, and cactus (Pasture UA-C). Common species are P. velutina, Opuntia engelmannii (Engelmann's prickly pear cactus), Acacia greggii (catclaw acacia), Mimosa dysocarpa (velvet pod mimosa), and Calliandra eriophylla (false-mesquite), and many species of perennial grass (McClaran et al., 2010; Scott, 2010a).

#### Software, Code, and Computing

Links to the aerial lidar vendor reports, code, and analysis workflows are maintained in a public GitHub repository: https:// github.com/tyson-swetnam/srer-wgew. Watershed boundaries and major infrastructure features in WGEW are available from Heilman et al. (2008), Moran et al. (2008), and the SRER data portal: https://cals.arizona.edu/srer/.

For the SfM point cloud creation we used proprietary software packages including: Pix4D (SenseFly, Switzerland) and Photoscan (AgiSoft, 2017). Analysis of point clouds and derivative models were also done in open-source CloudCompare (Girardeau-Montaut, 2011), Point Data Abstraction Library (PDAL, Hobu Inc., 2017) with Docker Community Edition (Merkel, 2014; Boettiger, 2015; Docker Development Team, 2017), and QGIS (Quantum GIS Development Team, 2017). SfM generation of dense point clouds in Agisoft Photoscan took from 1 to 192 h, depending on the number of images used in the reconstruction. Point cloud analyses in CloudCompare were typically less than 5 min per procedure, with large multi-scale model to model cloud comparison (M3C2, Lague et al., 2013) analyses required up to 20 min.

NSF Jetstream cloud computing service (Stewart et al., 2015) was used to filter point clouds for outliers and to translate projections for the aerial lidar and SfM data to common datums using the Docker version of PDAL (docker://pdal/pdal:1.5). PDAL and LAStools (Isenburg, 2014) were also used to remove outliers from the original <sup>∗</sup> .las data files and to compress to <sup>∗</sup> .laz format for online data hosting.

#### Instruments and Field Methods

Characteristics of lidar sensors varied among terrestrial, manned aerial, and sUAS platforms (**Table 1**). At the Lucky Hills site, the terrestrial lidar survey (TLS) included nine 360 degree scans on October 1, 2015; and at the Kendall grassland site, eight scan and six scan collections were made on September 23, 2015 and October 8, 2015, respectively. Eight terrestrial lidar scans were made at the SRER UA-C site on August 25th, and September 30th, 2016. The individual terrestrial lidar scans from positions located around the AOI insured >50% overlap between scans. Individual scans were mosaicked together during post-processing in Riegl's RiScan Pro software to comprise a total scanned area of ∼10 ha with a focus area of interest of ∼2 ha (Riegl, 2017). The individual terrestrial lidar scans were aligned using 10 cm diameter reflective cylinder targets mounted on 2 m tall poles. Reflective targets for each scan were centered on rebar driven into the ground and their locations were measured with a survey-grade Trimble GPS system in UTM coordinates tied to existing Coast and Geodetic Survey benchmarks on the watershed (Trimble, 2013). The Trimble R10 (Trimble, 2012) had a measurement error atop ground control points (GCP) of ∼5–7 mm in the horizontal and ∼6–16 mm in the vertical dimension. At the SRER, stable rebar GCPs were established at 13 locations and 17 cm diameter round plastic lids painted black & white in a cross pattern were temporarily fixed to the rebar rods. We surveyed the GCPs with a Real-Time Kinematic GNSS on October 7, 2016.

Manned aerial lidar scanning (ALS) data was collected by Woolpert Inc. over all of WGEW (∼50 km<sup>2</sup> ) on September 16–18, 2015. The lidar was georeferenced and geometrically corrected using a Real-Time Kinematic GNSS survey as well as a Rapid-Static GPS survey (see Supplemental Materials). Manned aerial lidar were gathered at the SRER in April 2011 by Sanborn Inc.. The SRER data was georeferenced and geometrically corrected using GNSS Survey (see Supplemental Materials). The positional accuracy of the manned aerial lidar given by the vendors was 6.6–8.0 cm RMSE<sup>z</sup> for bare earth, and 36–50 cm RMSE<sup>z</sup> for vegetation (**Table 1**), meeting the USGS quality level 1 standard (Heidemann, 2012).

An octocopter sUAS (Service-Drone, Germany) equipped with Velodyne-32 lidar sensor was deployed at the WGEW in October, 2016 (Velodyne Acoustics, Inc., Morgan Hill, CA) (**Table 1**). The octocopter sUAS weighs 5.5 kg and was developed to carry an additional heavy payload of up to 6.5 kg, for a maximum takeoff weight of 12 kg (Sankey et al., 2017a,b). The lidar data were mosaicked together in ENVI v. 5.3 (Exelis Visual Information Solutions, Boulder CO, 2010).

Characteristics of the cameras used for SfM varied (**Table 2**). Collections from hand held photography focused on either specific individual plants (grasses, cacti, mesquite) or along 30 m transect lines within the SRER UA-C pasture plots. The DJI Osmo was held on a 1 m range pole overhead with the camera pointed approximately nadir and 20–40◦ oblique. Both still photos and UHD (4 k) video were recorded along transects. Individual frames were extracted from the videos using ffmpeg (FFmpeg Developers, 2016). The rate at which frames were extracted from video varied, given the forward speed of the collection (∼1–2 m/s). Typically, 3–6 frames per second were enough to ensure 80% overlap between frames (4 k videos were shot at 24 and 30 frames per second). Quadcopter sUAS were flown over the SRER in summer 2016 using the DJI Phantom 3 and Phantom 4, each weighing ∼1.5 kg (Gillan et al., 2017). A third party software Altizure for DJI (Everest Innovation Technology, 2017), was used

TABLE 2 | Structure from Motion, platforms, sensors, image size [megapixels (M)], estimated Ground Sample Distance (GSD) pixel size, and SfM photogrammetry points per square meter (ppsm), and the sites of deployments: Santa Rita Experimental Range (SRER) and Walnut Gulch Experimental Watershed (WGEW).


TABLE 1 | Lidar platforms (TLS-terrestrial lidar scanner, ALS-Aerial lidar scanner, sUAS small Unmanned Airborne System), sensors, collection pulse rate in kilohertz (kHz), laser divergence (mrad), heigh above ground level (AGL), pulse size (centimeters [cm]), root mean square error (RMSE) in the vertical (z) and horizontal (h) plane (cm), laser pulse return density per meter square (ppsm), and the sites of deployments: Santa Rita Experimental Range (SRER) and Walnut Gulch Experimental Watershed (WGEW).


to program the flight missions with 80% forward and side overlap of the imagery.

The fixed-wing FireFLY6 flights were performed at the SRER UA-C site on March 17, 2016 and June 28, 2016 at an average flight altitude of 110 m AGL with 65 and 75% forward and side overlap between image tiles. Most of the aerial images were collected facing approximately nadir because the camera is fixed to the belly of the FireFLY6. Pitch, yaw, and roll during the flights resulting in images 0◦–15◦ degrees off nadir.

The eBee is a fixed wing (140 cm wingspan, 750 g mission weight), electric platform with a single pusher propeller at the rear (SenseFly, Switzerland). Flights were performed at an average altitude of 110 m AGL with 80% forward overlap between successive image and 70% side overlap respectively between adjacent flight lines at Kendall grassland site on October 8, 2016. The multispectral Sensfly multiSpec 4C camera records four bands: green (520–580 nm), red (630–690 nm), red edge (720–750 nm), and NIR (760–820 nm). The geo-located, spatially paired, and radiometrically calibrated images were combined to generate an orthomosaic and DSM of the entire flight area in Pix4D (SenseFly, Switzerland). Separately a 3D point cloud was generated for each band and then merged together to generate a single point cloud dataset. The eBee sUAS multispectral images were processed in eMotion and Pix4D (SenseFly, Switzerland) and in Photoscan (AgiSoft, 2017).

#### Post-hoc Point Cloud Registration

Comparing two or more point clouds required alignment using at least three GCPs shared between datasets. In the case of our sUAS plots, 10-13 GCPs were needed to ensure high enough accuracy (<5 cm) to successfully detect vegetation change over time (Gillan et al., 2017).

We did not establish GCPs for the eBee SfM photogrammetry in WGEW or the FireFLY6 in SRER. Features that were clearly identifiable in the terrestrial lidar or manned aerial lidar data, e.g., eddy-covariance flux towers, water sampling flumes, fence posts, road features, and small boulders were used post-hoc to align point clouds in CloudCompare (Girardeau-Montaut, 2011) with the translation/rotation tool and three-point-picking tool.

# Detection Bias

For bare earth analyses locations with barren ground that were clearly observable by the terrestrial lidar were used as references, i.e., roadways and bare patches with no herbaceous component. For the vegetation analyses locations scanned with the terrestrial lidar with no obstruction of the bare earth at the base of the reference plant up to its apical leader were used as references. Terrestrial lidar reference point cloud data were not used in areas with high incidence of occlusion and shadowing due to increasing distance from the scanner.

#### Raster Differencing

We produced DEM of Difference (DoD) (Lane et al., 2003; Pelletier and Orem, 2014) for the minimum elevation DEMs and maxima elevation DSMs. DEMs were generated by "cloth simulation filter (CSF)" (Zhang et al., 2016) and "Rasterization" tools in CloudCompare (Girardeau-Montaut, 2011). DEMs were generated using the minimum height function, or bare earth classified points (Class 2, Heidemann, 2012), in the case of the aerial lidar. The DSMs use a maximum height function after outlier points are removed. DEM and DSM rasters were produced at 0.5 m resolution; this was the finest possible resolution for comparison to the aerial manned lidar given its ground return point density. DEM and DSM were exported to QGIS as geotiff (.tif) rasters in their original datum (Table S2). In QGIS we used the raster calculator to generate (a) bare earth DoD1: DEM1–DEM<sup>2</sup> (Equation 1); and (b) vegetation DoD2: DSM1–DSM<sup>2</sup> (Equation 2) for the different point cloud types. The DoD assess bias within the AOI by establishing the absolute difference between each platform; however DoD differences alone did not intrinsically identify which platform as the source of the bias.

#### Cloud-to-Cloud Differencing

Lague et al. (2013) created the direct point cloud-to-cloud "multiscale model to model cloud comparison" (M3C2) tool to compare change across time in point cloud data. The M3C2 is successful at measuring surface and volume differences in geomorphological applications (James et al., 2017; Midgley and Tonkin, 2017). M3C2 computes local distances between "core points" from a "reference" cloud (Lague et al., 2013), we used the terrestrial lidar as the reference cloud and the other clouds as the "test" for differencing. M3C2 uses a "normal" flat surface to estimate the confidence interval based on cloud roughness and registration error (Lague et al., 2013). The difference between two points can be measured in the vertical (z) or horizontal (x and y) axis, or as a linear multi-scale distance based on the normal. In our analysis we compared the vertical (z) change. The significance of the measure is based on the registration error for each core point. Our registration error used the instrument precision of the lidar (**Table 1**) and the uncertainty of the RTK GNSS (Table S3).

#### Mesh Creation and Object Height Measurement

To improve the estimate of vegetation height in a fused data comparison (Objective 2), we used a cloth simulation filter (CSF) technique (Zhang et al., 2016) in CloudCompare (Girardeau-Montaut, 2011), to calculate a minimum elevation (bare earth) model as a mesh. One benefit of the mesh was that it fills space beneath large areas of vegetation where no bare earth surface is remotely sensed. We also used CloudCompare's rasterization tool to generate maximum (DSM) models for object heights. To measure the height of points above the mesh bare earth level we used CloudCompare's Mesh-to-Cloud function.

### RESULTS

#### Feature Class Detection Bias

Bare earth elevation is only clearly resolved by the terrestrial lidar in barren areas or along roads. The propagated uncertainty (Supplementary Materials) of any observation of bare earth derived from manned aerial lidar to a terrestrial lidar fused measurement was ±8.24 to 9.4 cm RMSE<sup>z</sup> (**Table 3**). Aerial lidar (manned and sUAS) was found to be best at detecting bare ground, in part because of its unique point of view, penetrating TABLE 3 | Vertical (z) differences in point clouds by platform using M3C2 with reference cloud uncertainty of 5 cm; mean ± 99% CI observed difference (cm).


*Negative mean values represent systematic under measurement by the test cloud relative to the terrestrial lidar reference cloud. Positive values are associated with the reference cloud failing to observe elevations at depth, e.g., no returns beneath dense vegetation. n/a value are for no temporal alignment for comparison (Santa Rita). The SfM point clouds derived from: † Sensefly eBee, ‡ DJI Phantom 4,* §*DJI Osmo.*

<sup>(</sup>A), terrestrial lidar compared to manned aerial lidar (B), and terrestrial lidar compared to sUAS lidar (C).

down through the vegetation. Conversely, the sUAS fails to detect bare ground beneath grass and shrub vegetation (**Figure 2**), in part due to the GSD of the imagery used to generate it. However, ALS did not always penetrate the canopies of dense leaf-on woody shrubs >2 m tall, nor does it detect sense the fine branches of the shrubs or trees (**Table 3**). The ALS does resolve larger features on tall woody vegetation (>2 m) with typical underestimates of 5–25 cm (**Table 3**).

The octocopter sUAS lidar had lower positional certainty than the manned aerial lidar, due in part to the sUAS lidar having lower quality GNSS (50+ cm RMSEz), and unresolved GPS-IMU drift aboard the octocopter platform (Sankey et al., 2017a,b). When the sUAS lidar point cloud was adjusted using fine alignment tools in CloudCompare its features were comparable to the manned aerial lidar (**Figure 2**, **Table 3**). The sUAS lidar resolved both grasses and bare surface in herbaceous and shrubs in the Kendall site, while the manned aerial lidar does not. Despite the increased point density relative to the manned aerial lidar (8–24 ppsm) the sUAS lidar (35–115 ppsm) was still too sparse to identify individual herbaceous plants or shrubs based on only the point cloud.

The Ebee SfM failed at sensing bare earth elevation for areas with dense vegetation on the WGEW. This was not surprising at Kendall Grassland where the herbaceous cover was high. The eBee SfM also tended to under-predict height of woody vegetation in the same way as the manned aerial lidar: it failed to discriminate diffuse outer branches and maximum tree height (**Figure 2**, **Table 3**). The initial GNSS referenced eBee SfM point cloud relative to the RTK referenced terrestrial lidar was not accurate and required realignment using the 3-point-picking and fine alignment tools in CloudCompare.

The Phantom 3 and Phantom 4 sUAS SfM photogrammetry also produced point clouds suitable for discriminating large woody vegetation or herbaceous vegetation. However, the resultant SfM point clouds were less accurate than the terrestrial lidar, under-estimating grass height on average by 15 and 4 cm for the larger woody shrubs (**Table 3**). SfM photogrammetry based on the FireFLY6 imagery was not able to accurately resolve tall woody vegetation (mesquite trees taller than 2 m) on the SRER during the spring leaf-off period, but was improved during the leaf on period in late summer (Supplementary Materials).

#### DEM/DSMs of Difference for Detection Bias

Our objective #1 was to establish detection bias between point cloud data and derivatives. In the case of raster data generated from point clouds we establish the difference between the same type of measurement for two different platforms, i.e., DEM1- DEM<sup>2</sup> (DoD1) & DSM1-DSM<sup>2</sup> (DoD2).

The DoD<sup>1</sup> for manned aerial lidar (**Figure 3A**) vs. terrestrial lidar (**Figure 3B**) are ±8.24 cm RMSE in areas immediately surrounding the terrestrial laser scan locations in the AOI (**Figure 3D**). Bare earth elevations are >100 cm different, where bare ground was not visible to the terrestrial lidar sensor (**Figure 3D**). In these locations the manned aerial lidar and SfM (**Figure 3E**) are lower (cooler colors) than the terrestrial lidar estimate of bare ground (**Figure 3F**).

The aerial lidar DSM (**Figure 4A**) under estimate height of grass and woody shrubs by 10 to 120 cm when subtracted from the terrestrial lidar DSM (**Figure 4D**). There was also a consistent difference across both the SfM-ALS DoD<sup>2</sup> (**Figure 4E**), due to the failure of the aerial lidar to sense the herbaceous component. There was an under prediction of height for SfM-TLS DoD<sup>2</sup> (**Figure 4F**), with the eBee SfM DSM typically 4 to 52 cm lower than the terrestrial lidar DSM.

#### Data Fusion Accuracy

The area immediately around the terrestrial lidar scanner had the most similar bare elevation as aerial lidar DEM (**Figure 5A**). The associated histogram below **Figure 5A** shows a mean of approximately zero with standard deviation of 25 cm—similar to the results shown in **Table 3** for the M3C2. The weakness of the aerial lidar in measuring vegetation is apparent in **Figure 5B** where the average height of the point cloud is only 10 cm in grass—an approximate 55 cm under measurement relative to the terrestrial lidar in the same locations (**Table 4**), visible in **Figure 5C**. The eBee SfM produced DSM was a more accurate representation of grass height (**Figure 5D**) when using the aerial lidar derived DEM, and was only 24 cm lower on average than the terrestrial lidar scans (**Table 4**).

Using the mesh-to-cloud height measure in CloudCompare we produced a point cloud based height value for the terrestrial lidar (**Figure 6**). The bare earth mesh was derived from the CSF (Zhang et al., 2016) (Supplementary Materials). Our anecdotal example, shown in **Figure 6**, demonstrates one potential method

terrestrial lidar DoD1, (E) eBee SfM minus aerial lidar DoD1, and (F) eBee SfM minus terrestrial lidar DoD1.

terrestrial lidar DoD2, (E) eBee SfM minus aerial lidar DoD2, and (F) eBee SfM minus terrestrial lidar DoD2.

for improving the characterization of vegetation through two platform data fusion: terrestrial lidar and SfM point clouds measure vegetation height above ground but fail to sense bare earth. When they incorporate an aerial lidar point cloud derived bare earth mesh surface the height measure improves. Note in **Figure 6** the Ebee SfM points are almost entirely along the top of the grasses and woody shrub (Yucca elata, soaptree yucca) as scanned by the terrestrial lidar, while the larger diameter black points from the aerial lidar have few returns in the vertical profile except for the flowering stalk of the yucca. Also note, terrestrial lidar has few or no points low in the vertical profile near ground level. This was due to the flat angle of the horizontal scan profile being obscured by dense herbaceous cover.

#### DISCUSSION

No single observation platform solved Levin's "problem of pattern and scale" because each had its own limitations in

TABLE 4 | Average (mean ± 95% CI) object height measurements (cm) taken on the Kendall Grassland site; examples of the aerial lidar (ALS) height model is shown in Figure 5B, the terrestrial lidar DSM minus the aerial lidar DEM (TLS-ALS) in Figure 5C, and eBee DSM minus aerial lidar DEM (SfM-ALS) in Figure 5D.


(1) detecting patterns of important objects in the dryland ecosystem (bare ground, grass, and shrubs), (2) the accuracy of representing the heights of grass and shrubs, and (3) the extent of area represented in a typical data collection campaign. Nonetheless, the combination or fusion of information from the different platforms appears to make greater progress toward solving "problems of pattern and scale" than any single platform alone. Specifically, we suggest the combination of sUAS SfM and either manned or sUAS aerial lidar as providing the best solution at large spatial scales amongst the platforms studied.

To represent spatial and temporal patterns of vegetation, sUAS SfM provides a high quality representation of the presence of grass and shrub vegetation at centimeter scale, but fails to reliably detect bare earth or DEM beneath that vegetation, thus limiting the ability to represent the height and volume of those plants. Aerial lidar on the other hand, provides that needed high quality representation of the pattern of bare earth or DEM. For scale, manned aerial lidar provides spatial resolutions of 0.25–1.0 m2 , and sUAS Velodyne lidar spatial resolution of 0.1–0.25 m<sup>2</sup> which is slightly coarser than the terrestrial lidar or sUAS SfM. However, from the perspective of scale of extent or coverage, most aerial lidar data collection campaigns represent hundreds to thousands of km<sup>2</sup> , and there is less need to frequently repeat those campaigns because DEM change in dryland ecosystems are less frequent than the height and volume of grass and shrub vegetation. On the other hand, the extent of a sUAS SfM campaign is at least two orders of magnitude smaller than an aerial lidar campaign, but it is easier and cheaper to repeat several times per year in specific areas of interest, e.g., eddy-covariance flux tower footprints.

mesh (transparent green) with the smaller diameter black points are derived from the eBee SfM point cloud. Note, there are no eBee SfM points low in the profile.

## Point Cloud Bias amongst Platforms

The availability of high density lidar point clouds, particularly terrestrial lidar, allowed us to carefully examine differences amongst sUAS lidar, manned aerial lidar, and SfM photogrammetry point clouds. We found the manned aerial lidar, sUAS lidar, and sUAS SfM data all tended to underestimate herbaceous vegetation height in the grassland and shrub areas.

The accuracy of the terrestrial lidar measurements are over an order of magnitude finer than the manned aerial lidar. When comparing the various point clouds to the terrestrial lidar data it was apparent large biases (positional shifts) had been introduced by the various GNSS for both the aerial lidar and SfM photogrammetry. It was also apparent that terrestrial lidar had difficulty resolving bare earth elevations along flat trajectories further away from the scan locations when dense herbaceous vegetation was present (**Figures 5**, **6**). The sUAS lidar was able to penetrate the grass to bare earth elevations and characterize the grass vertical profile, similar to that observed in the terrestrial lidar, but was unable to segment individual plants due to lower density. The manned aerial lidar was not suitable for measuring or monitoring herbaceous grass and small shrubs, as highlighted in **Figure 5B**. Most of the WGEW manned aerial lidar data are single return at the near ground level in herbaceous grasses (**Figure 6**). It is likely the ∼13.5 cm footprint of the aerial lidar meant smaller features such a grass panicles and small diameter woody stems were not large enough to be detected or were not at the threshold level for discretization from the full waveform data. The largest woody plants, e.g., leaf-on mesquites were detected, but their diffuse branches were not, resulting in an overall 10 to 25 cm height under-estimate of tree height relative to the terrestrial lidar (**Figure 5B**, **Table 4**). These results are similar to Luscombe et al. (2014), who compared manned aerial lidar and terrestrial lidar and found similar under-estimates in height for taller vegetation.

In bare ground areas, sUAS SfM data were equivalent to aerial lidar and terrestrial lidar, which is consistent with other recent studies (Lucieer et al., 2012; Westoby et al., 2012; Anderson and Gaston, 2013; Fonstad et al., 2013; James et al., 2017). In dense herbaceous cover SfM photogrammetry point clouds failed to accurately resolve bare ground elevations beneath vegetation accurately. Error in sUAS SfM point clouds are attributed in parts to (1) barrel or pin cushion lens distortion (James and Robson, 2014), and (2) algorithm error when interpreting between bare earth and vegetation (James et al., 2017).

#### Influence of Canopy Cover on DEM generation

We found the sUAS SfM point cloud M3C2 values (**Table 3**) were all within the uncertainty of the aerial lidar for bare ground (±9.6 cm RMSEz) and large woody vegetation (±18 cm RMSEz). In the case of the sUAS Phantoms and Osmo SfM, values are within the range of uncertainty of the terrestrial lidar for bare ground (Table S3). Our results were essentially the same as Nouwakpo et al. (2015), who compared soil texture and elevation differences between ground-based SfM and terrestrial lidar and were able to stay within 5 mm RMSE for bare earth patches but not in areas of vegetation with >50% canopy cover.

The sUAS SfM (eBee, Phantoms, Firefly6) data could not be used to generate bare earth elevations beneath herbaceous vegetation (grasses and low shrubs) with any confidence. Another problem in densely vegetated herbaceous sites such as the WGEW Kendall Grassland and the SRER UA-C, occurred when attempting to generate bare earth models with imagery from time periods of vegetation senescence. The resultant models were more likely to be biased because the senescent vegetation cannot be differentiated from soil, such as during the dry season (e.g., the March 2016 Firefly6 flights at UA-C). Standard attempts at removing vegetation using a filter or masking technique, such as NDVI (Cunliffe et al., 2016; James et al., 2017), can only be executed during the leaf-on rainy season in these areas when canopy cover is highest and is spectrally responsive to NDVI.

## Influence of Canopy Cover on DSM Generation

Many recent SfM research papers (Cruzan et al., 2016; Cunliffe et al., 2016; James et al., 2017) report on how SfM does very well in areas with low vegetation cover. In our study, the sUAS SfM was not as accurate at sensing the maximum herbaceous vegetation height as the terrestrial lidar, rather the SfM tended to under predict max heights of the fine herbaceous and woody component. However, SfM at high resolution in hand-held DJI Osmo or low flying Phantoms, was capable of resolving fine herbaceous features and woody stems (Supplementary Materials). Kato et al. (2015) compared SfM vs. terrestrial lidar for tree canopy structure and reported a strong correlation between canopy shape for both technologies. We found similar patterns where the sUAS and handheld SfM reconstruct the shape of large woody canopy shrubs, but are unable to penetrate deeply into the canopy of individual trees to reveal their internal structure.

# Raster Differencing (DoD) vs. Cloud-to-Cloud Comparison (M3C2)

The DoD show vertical change in both the bare elevation and maximum height of vegetation in the two surface models (**Figures 4**, **5**); meanwhile the M3C2 was able to diagnose changes in multiscale and horizontal distance both beneath vegetation and across canopy (Table S3). M3C2 was able to more clearly diagnose errors of horizontal change than the DoD which must be inferred by looking at rasters for distinct vertical changes around the vegetation.

Both methods have utility: DoD rasters are useful in ecohydrology applications, while the M3C2 maybe more useful for geomorphology (bare elevation change), and monitoring of vegetation dynamics (growth, mortality, abundance and composition). Notably, the computational overhead required for M3C2 is orders of magnitude greater than for raster based modeling but is reduced by the use of GPU computing.

#### Data Management

Our large data sets created a computing challenge when comparing the largest point clouds (>50 GB) in a single computing instance. Cyberinfrastructure storage, such as the CyVerse.org DataCommons (http://datacommons.cyverse.org, Walls, 2017), the NSF lidar repository OpenTopography.org, and USGS National Map are available to users who wish to publically archive processed point clouds. The original images (.JPG or.TIF files) generated by sUAS or handheld collection should be archived at the highest resolution possible, with associated time of flight mission metadata, i.e., GNSS location and camera specifications. One benefit of retaining the imagery for SfM photogrammetry over delivered (i.e., processed discrete) lidar data are their availability for reanalysis with new versions of SfM software which have improved optimization techniques as well as camera and lens correction algorithms. Lidar data, both discrete and full waveform, available in their raw (time-of-flight) format should similarly be stored for future reanalysis.

To help with processing we used CyVerse Atmosphere and NSF Jetstream (Stewart et al., 2015) virtual machine instances. We found the largest instances were ideal for large distributed jobs running point cloud analyses (without GPU). For users who are not able to purchase workstation class PCs, access to cloud or HPC solutions are ideal alternatives at a fraction of the initial start-up cost. We expect the availability of cloud computing and cyberinfrastructure to become more common as computing technology further develops. At the same time, we expect the volume of high-resolution 3D data to increase, as sUAS technology becomes more widely used by geoscientists and ecologists.

Fusion of point cloud data from different sensors and formats presents a problem for ensuring data standards and metadata attribution continue through the life cycle of data. Currently, lidar point cloud data are managed under the LAS standard developed from the American Society for Photogrammetry and Remote Sensing (ASPRS). SfM point clouds, stored as LAS/LAZ files, have fundamental differences from lidar LAS/LAZ, including the lack of scan angle, intensity, or return number.

#### Costs (Time, Human, Computing)

The SfM point cloud generation became prohibitive with very large image collections (250–5,000 high quality images) requiring from 12 to 192 h to generate the ultra-high resolution point clouds, also maxing out the available RAM on the workstations without chunking the data into tiles and further increasing processing time. In addition, the photogrammetrist typically had to work with individual images to geo-locate ground control points from anywhere between 4 and 16 h for each model. Conversely, the terrestrial lidar required only a few hours of post-processing by the technicians to resolve the ground-control, typically being completed within 24 h of the initial collection. Aerial lidar data require a significant investment, however that step was shouldered by the contracted vendor and was reflected in the cost of the aerial lidar collection.

# FUTURE OUTLOOK

The scale of remotely sensed 3D data from hand held platforms, sUAS, and manned aircraft allows us to ask new questions which 25 years ago were unassailable for Levin (1992). However, we must recognize the strengths and weaknesses inherent to the technology and the data when applying them to the measurement of ecological features. We found aerial lidar from both sUAS and manned aircraft to be more accurate at sensing bare ground elevation than terrestrial lidar or sUAS SfM in dense vegetation, but less accurate at measuring fine and woody herbaceous vegetation. By using the aerial lidar to derive an accurate bare earth model and a second technology (terrestrial lidar, sUAS SfM photogrammetry) to measure herbaceous vegetation height and structure, we were able to better characterize vegetation than by any single technology alone. Quantifying propagation of uncertainty when fusing or combining point cloud (Supplementary Materials) or derivative data is likely to be one of the most important steps researchers take when attempting to make ecologically important measurements.

Newer constellations of "dove" satellites which form "flocks" and record the entire global surface at weekly to daily time scale solve a temporal limitation to remote sensing at scale, with nominal spatial resolution (1–3 m) similar to aerial orthophotography (Butler, 2014; McCabe et al., 2016; Zimmerman et al., 2017). In the United States, nationally available lidar data sets (Stoker et al., 2016) provide bare earth models at 1–2 m resolution. Our methods show how these data could be used for more accurately measuring dense vegetation cover, height, and volume from sUAS derived lidar or SfM photogrammetry data and scaling out spatially with satellite products. Despite the utility of sUAS for monitoring vegetation phenology and structure at orders of magnitude higher resolution, the overhead of computational processing of these imagery can became a limiting factor at progressively larger spatial and temporal resolution, suggesting the need for distributed HPC and cloud based solutions like those provided by Stewart et al. (2015).

Keeping pace with innovation and the breadth of remotely sensed information available for conducting ecological research can be challenging for any single field ecologist, particularly for those looking to add new skills or maintain technical proficiency. Establishing where the limitations of these new remote sensing technology are was one goal of this research. We hope to have revealed a few useful, scalable solutions to these problems through data fusion, and provided insight into improved monitoring programs in dryland ecosystems.

# AUTHOR CONTRIBUTIONS

TLS, TTS, PH, MN, and MM conceptualized and designed the research ideas. TLS, TTS, JG, MN, and JM acquired and analyzed the data. TLS wrote the draft with contributions from TTS, PH, MN, MM, and JG.

# ACKNOWLEDGMENTS

The Southwest Watershed Research Center USDA-ARS, provided the terrestrial lidar equipment used in this study. We also thank two reviewers for improving an earlier versions of the manuscript.

This material is based upon work supported by the U.S. Department of Agriculture, Agricultural Research Service, under Agreement No. 58-2022-5-13. Any opinions, findings, conclusion, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the U.S. Department of Agriculture.

This material is based upon work supported by the National Science Foundation under Award Numbers DBI-0735191 and

#### REFERENCES


Butler, D. (2014). Many eyes on earth. Nature 505, 143–144. doi: 10.1038/505143a

Carrivick, J. L., Smith, M. W., and Quincey, D. J. (2016). Structure from Motion in the Geosciences. Hoboken, NJ: John Wiley & Sons.


FFmpeg Developers (2016). ffmpeg Tool (Version be1d324) [Software]. Available online at: http://ffmpeg.org/


DBI-1265383, www.cyverse.org; and Jetstream (Award number ACI-144506) cloud resources.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpls.2017. 02144/full#supplementary-material

advance critical zone science. Hydrol. Earth Syst. Sci. Discuss. 12, 1017–1058. doi: 10.5194/hessd-12-1017-2015


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Swetnam, Gillan, Sankey, McClaran, Nichols, Heilman and McVay. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.