SPECIALTY GRAND CHALLENGE article
Specialty Grand Challenge: Remote Sensing Time Series Analysis
- Department of Geography, University of Florida, Gainesville, FL, United States
The globe is currently undergoing a range of alarming changes related to social and environmental systems, and the links between the two. Our ability as researchers to study the dynamics of these ongoing processes is essential for real-world understanding and application of management strategies that can mitigate potentially negative outcomes. The scale of change and its associated impact generated by natural and anthropogenic drivers varies across the landscape, such as local degradation of ecosystem services, regional deforestation, large scale urbanization, and widespread yet geographically specific changes yielded by vagaries in climate. Understanding such critical changes is of paramount importance for the future wellbeing of the coupled human-natural systems that we are all a part of and on which we all depend.
Historically, one of the greatest limitations in our ability to study these systems with remote sensing technology has been inadequate availability of time series datasets that provide fine enough spatial and temporal resolution capable of identifying processes of global environmental change (GEC). However, with the advances in sensors used for environmental remote sensing, as well as the improvements in data storage and distribution, we now have the capacity to employ time series techniques for detecting GEC and addressing the multitude of questions surrounding its impacts. Currently, many of the time series methodologies being applied to examine this suite of issues are still in development, and as such, there is significant space for growth, innovation, and exploration in the field of time series remote sensing analysis (TSRSA).
Only a few decades ago, what was considered a detailed TSRSA may have involved three or more Landsat images and corresponding land cover classifications with a set time interval, such as a decadal study (Southworth et al., 2004; Mondal and Southworth 2009; Cassidy et al., 2010; Gibbes et al., 2010; Adhikari and Southworth 2012). This has given way to analyses with much finer temporal resolution, including remotely sensed information with daily global coverage dating as far back as 30 years (Zhu and Southworth, 2012; Campo-Bescós et al., 2013; Haro-Carrión et al., 2020; Herrero et al., 2020). The tremendous amount of data now available for remote sensing research cannot be efficiently utilized with traditional methodologies of analysis, creating a need for new approaches and techniques. Use of artificial intelligence (AI), specifically related to issues of big data and machine learning, including deep learning, are all possible innovations within this field. As such, we are now on the cusp of being able to effectively investigate some of the most pressing environmental concerns of our time at a temporal scale relevant to climatological, ecological, and social systems. The coming decade will surely present landmark innovations, introduce novel approaches, and yield breakthroughs in understanding our world. Such advances will undoubtedly be facilitated by the enhanced accessibility of remotely sensed datasets with greater temporal range, which will enable more effective monitoring and detection of GEC.
Traditionally, the most common approach used in remote sensing studies has been a selection of two to five dates, for which individual landcover classifications were produced and change was determined by comparing classifications over time (Southworth et al., 2004; Mondal and Southworth 2009; Cassidy et al., 2010; Gibbes et al., 2010; Adhikari and Southworth 2012). While these studies were very useful and often linked directly with ancillary information from surveys for greater insight into the exhibited change on the landscape, the true density of landcover data was limited. Despite the large number of studies using such methods for landcover change analysis, we have now surpassed the limited scope offered by this approach, with advances in sensor technology extending the possibilities of remote sensing science.
The launch of improved sensors led to new questions and opportunities within Earth science. The Advanced Very High Resolution Radiometer (AVHRR) managed by the National Oceanic and Atmospheric Administration (NOAA) offered daily coverage at a global scale beginning in 1981, albeit with a very coarse spatial resolution. It has since been succeeded by the Visible Infrared Imaging Radiometer Suite (VIIRS), which provides better spatial resolution and a larger swath. Similarly, the Landsat fleet revolutionized Earth observation (EO) after its initial launch of the multispectral scanner (MSS) in 1972. This was followed by the thematic mapper (TM), the enhanced thematic mapper (ETM), and the operational land imager (OLI), the latter of which will soon be succeeded by OLI-2 aboard Landsat nine that is set to launch in September of 2021. These sensors have allowed scientists to capture almost 50 years of information, beginning in 1973 to present, serving to increase the data density with much finer spatial and spectral resolutions over time (Kuezner et al., 2015). Furthermore, the provision of open access to the existing Landsat archive in 2008 marked a turning point within environmental remote sensing of land surface processes, as data became widely available at a level previously unprecedented. This publicly available data was corrected and registered by the United States Geological Survey (USGS), allowing many researchers to exploit a data source that was previously unavailable to them due to technical limitations (Weng 2018).
The AVHRR and Landsat satellites have acted as pillars of environmental remote sensing, paving the way for most future research. Since their inception there has been development of newer sensors, such as the Moderate Resolution Imaging Spectroradiometer (MODIS) onboard the Aqua and Terra platforms managed by the National Aeronautics Space Agency (NASA). As well as the European Space Agency’s Advanced Along Track Scanning Radiometer (AATSR) and the Medium Resolution Imaging Spectrometer (MERIS) onboard the European Environmental Satellite (ENVISTAT). While these more recent sensors generate valuable information, they are more temporally limited, with the Terra platform having been launched in 1999 and ENVISTAT only lasting from 2002 to 2012. Despite an abundance of new sensors, now contributing to the array of environmental remote sensing resources, their utility in TSRSA is constrained by relatively small-time frames for which they have gathered data. Given that much of the GEC occurring on our planet is intricately linked with climate and other processes that unfold over long periods of time, it is critically important to analyze EO with sufficient historical depth. One way to address the temporal limitations is to develop a time series of data from across a suite of image platforms and sources, which are then integrated and resolved to a single analysis, and as such, these data fusion approaches and techniques do offer potential for the enhanced monitoring of our Earth system.
With the greater availability of these long-term remotely sensed datasets, the innate challenges of time series analysis are further complicated due to immense volumes of data and computational requirements, making the prospects for innovative and novel time series based remote sensing analysis most opportune. There have long been unique difficulties to working with EO caused by the spatial dependence and nonstationary temporal nature that is often inherent in remotely sensed data (Muir et al., 2021). This includes issues of spatial and serial correlation, the latter of which is particularly troubling for dense time series (Kuezner et al., 2015). In the past, researchers relying on traditional approaches for time series analysis of remotely sensed data have on occasion excluded necessary model parameters to account for such characteristics in the data, fundamentally altering the results of their research. This has promoted further growth in the realm of time-space modeling, though the development in both computer programming and statistical analyses can pose additional challenges for those researchers undertaking advanced time series remote sensing studies.
Additionally, satellite imagery in its current form of plentiful images with very fine temporal and spatial scales, has inevitably become an issue of big data. The use of standard computing hardware to analyze all relevant images is no longer efficient, or in some cases not feasible, making high-performance computing essential to analyses of EO. Yet, the computing power needed to investigate dense time-series and adequate storage of big data can be prohibitive to researchers, particularly those in less developed nations who have historically faced impediments to obtaining and working with large geospatial datasets. Furthermore, costs associated with accessing commercial data collections, such as those with very high resolution, may prove to be a financial constraint to scientists across the globe.
The temporal limitations of the sensors themselves presents several obstacles in time series remote sensing, such as cross-platform comparisons, integration, and data fusion, which may be needed to generate observations with a temporal scale necessary for studying the major issues facing our planet. However, some sensor-related issues remain beyond the control of those using the imagery gathered by sensors. Even when data is available over adequate time durations, the sensors themselves may be affected by problems such as geolocation errors, orbital drift, inaccuracies in sensor calibration, and long-term issues related to sensor lines. This is compounded by the well-established geographically specific challenges linked to atmospheric profile and conditions during data collection, correctional climatological data and cloud cover, and impacts from regional topography.
Despite the aforementioned challenges, major gains have been made in several areas of time series remote sensing (Kuezner et al., 2015). Modern techniques have pioneered new ways of measuring useful remotely sensed information from Earth’s surface, integrating information across different datasets, and revolutionary analytical methodologies. Previous studies of remotely sensed time series have employed a limited number of variables or indices. Among these, vegetation indices (VI) have been widely used in environmental remote sensing given their ability to estimate vegetation cover, vigor, and growth (Xue and Su 2017). More than 40 VI’s have been created over the last 50 years and used in a varied range of studies, including those related to wildfires, land degradation, and for enhanced discrimination between land cover classes (Bannari et al., 1995; Yengoh et al., 2014). Model data has also been utilized to develop time series of vegetation-related variables, such as MODIS-based Net Primary Productivity spanning 20 years, global measurements of Vegetation Directional Persistence derived from NDVI, and multiple time series products that estimate Leaf Area Index (Southworth et al., 2016; Xiao et al., 2017). These metrics provide valuable insights as time series products for monitoring Earth’s system over time due to the direct connection between vegetation and various drivers of GEC, such as climate and anthropogenic land use. Other subfields of remote sensing have developed their own indices to evaluate and monitor change over time. In urban remote sensing spectral indices have been applied to evaluate the rate of urbanization through the urban index (UI), normalized difference built-up index (NDBI), the normalized difference impervious surface index (NDISI), among others (Kawamura et al., 1996; Zha et al., 2003; Xu, 2008). Hydrologists and climatologists have overcome problems of spatial heterogeneity in climatological data and accelerated the process of identifying drought by exploiting remotely sensed observations to estimate climatological indices (Liu et al., 2020). Spectral indices have become critical scientific tools across the numerous subfields utilizing TSRSA, with some studies employing multiple indices for a more holistic understanding of GEC.
Owing to the greater data availability and the creation of new metrics, there has been a strong emphasis on the development of new approaches, which have enabled researchers to examine temporal trends in EO more readily. Change detection algorithms have been especially promising for studying GEC, facilitating identification of shifts in time series data occurring at different rates across temporal and spatial scales. Although most algorithms operate at the pixel level, some have been built to incorporate information from neighboring pixels, permitting greater insight into the agents of landcover change and a means to reduce issues of seasonal variation (Kennedy et al., 2015; Hamunyela et al., 2016). Zhu (2017a) classified change detection algorithms for TSRSA into six major categories based on; 1) frequency of observations, 2) index used to detect change, 3) number of variables, 4) offline or online detection, 5) magnitude of change, and 6) the scale of analysis (i.e. subpixel/pixel/spatial). The algorithms can be defined through their different methodologies for detecting change, such as thresholding, differencing, segmentation, trajectory classification, regression, and statistical boundaries. The latter of these methods includes the widely used Breaks for Additive Seasonal and Trend (BFAST) and the Continuous Change Detection and Classification (CCDC) (Verbesselt et al., 2010; Zhu and Woodcock 2014), which are based on the theory that any significant deviation from the expected statistical boundary in the time series indicates change. Also, among the commonly used algorithms is the Landsat-based detection of Trends in Disturbance and Recovery (Land-Trendr), developed by Kennedy et al. (2010). This method reveals abrupt change through a process of segmentation along a time series, whereby specific types of landcover change can be identified through their distinct temporal signatures captured in spectral space. Selection of a change detection algorithm is highly dependent on the scale of the change target and the anticipated rate of change, either abrupt or gradual, where the former may be better suited to identify processes such as deforestation and the latter to monitor shifts in species composition and broader ecosystem health (Hayes and Sader 2001; Fickas et al., 2016; Hamunyela et al., 2016).
As with any technologically dependent discipline, AI has played an integral role in advancing TSRSA. The information available for extraction in large remotely sensed datasets implores the use of AI techniques. Many in the field are familiar with AI through preliminary machine learning (ML), such as random forest classifications (Belgiu and Drăguţ 2016). However, the possibilities are expanding through deep learning (DL) that is characterized by neural networks (NNs), which typically involves data containing two or more layers. The superior density now available in remote sensing imagery has shifted the focus to greater consideration of the time variable, rather than individual image analysis (Ma et al., 2019). This move towards time-series processing will be greatly facilitated by the creation of innovative network architectures capable of simultaneously utilizing spatial, temporal, and spectral information from EO (Zhu et al., 2017b). AI has also progressed processing of big data and data-mining with DL as a data-driven strategy for knowledge discovery, capable of obtaining more information than what was previously achievable with rule-based approaches (Liu 2015; Wang et al., 2020). AI offers channels to improve data fusion and integration through augmenting spatial and temporal resolution, analysis of multi-source and multimodal data, and streamlined processing (Cao et al., 2020; Peng et al., 2020). This is exemplified by methods such as reconstruction of imagery from the Landsat archive through feature-level fusion (Chen et al., 2021), as well as multimodal-temporal fusion for generating image series with greater temporal resolution (Liu et al., 2019). These advanced techniques have now improved researcher’s abilities to analyze GEC across the realms of Earth’s natural systems, and tools are already available in a variety of programming languages and software packages (Lary et al., 2016; Maxwell et al., 2018; Yuan et al., 2020). The introduction of AI and DL in remote sensing has prompted the proposal of three major directions in time series research, including new methods for constructing time series datasets, data extraction, and environmental applications for time series analysis (Ma et al., 2019; Jeon et al., 2020).
Recent advances in the collection and distribution of remotely sensed imagery have profoundly advanced our ability to understand the complex nature of our planet and the link between humans and their environment (Liu 2015). Looking forward, growth of TSRSA and the assimilation of AI in environmental remote sensing analyses will only deepen, empowering researchers to understand Earth’s system and the coupled socio-ecological processes taking place within it. As scientists more fully embrace and expand on the computational capacities impelled by AI (machine learning, deep learning, data fusion, cloud storage etc.) and the resolution of the temporal axis in remote sensing analyses becomes finer, former obstacles no longer limit the questions that can now be addressed with TSRSA. The greatest challenges facing our planet, those of changing climate and land systems, protecting ecological diversity, limiting ecological degradation, and increasing rates of urban expansion globally can be more accurately examined and better understood using time series approaches, and the satellite based EO represents one of the most powerful tools at our disposal. Coupled with our increasing ability to reveal long term land surface dynamics through the use of increasingly innovative and novel time series approaches, we are no longer hampered by limitations in computational power or data availability, placing us at the cusp of realizing major advancements within the field of time series analysis.
JS developed the outline and ideas for the paper and CM and JS cowrote the paper.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Belgiu, M., and Drăguţ, L. (2016). Random forest in remote sensing: A review of applications and future directions. ISPRS J. Photogrammetry Remote Sensing 114, 24–31. doi:10.1016/j.isprsjprs.2016.01.011
Campo-Bescós, M., Muñoz-Carpena, R., Southworth, J., Waylen, L. P., and Bunting, E. (2013). Combined Spatial and Temporal Effects of Environmental Controls on Long-Term Monthly NDVI in the Southern Africa Savanna. Remote Sensing 5, 6513–6538. doi:10.3390/rs5126513
Cao, R., Tu, W., Yang, C., Li, Q., Liu, J., Zhu, J., et al. (2020). Deep learning-based remote and social sensing data fusion for urban region function recognition. ISPRS J. Photogrammetry Remote Sensing 163, 82–97. doi:10.1016/j.isprsjprs.2020.02.014
Cassidy, L., Binford, M., Southworth, J., and Barnes, G. (2010). Social and ecological factors and land-use land-cover diversity in two provinces in southeast Asia. J. Land Use Sci. 5 (4), 277–306. doi:10.1080/1747423x.2010.500688
Fickas, K. C., Cohen, W. B., and Yang, Z. (2016). Landsat-based monitoring of annual wetland change in the Willamette Valley of Oregon, USA from 1972 to 2012. Wetlands Ecol. Manage. 24, 73–92. doi:10.1007/s11273-015-9452-0
Gibbes, C., Adhikari, S., Rostant, L., Southworth, J., and Qiu, Y. (2010). Application of Object Based Classification and High Resolution Satellite Imagery for Savanna Ecosystem Analysis. Remote Sensing 2 (12), 2748–2772. doi:10.3390/rs2122748
Hamunyela, E., Verbesselt, J., and Herold, M. (2016). Using spatial context to improve early detection of deforestation from Landsat time series. Remote Sensing Environ. 172, 126–138. doi:10.1016/j.rse.2015.11.006
Haro-Carrión, X., Waylen, P., and Southworth, J. (2020). Spatiotemporal changes in vegetation greenness across continental Ecuador: a Pacific-Andean-Amazonian gradient, 1982-2010. J. Land Use Sci. 16, 18–33. doi:10.1080/1747423X.2020.1866705
Hayes, D. J., and Sader, S. A. (2001). Comparison of change detection techniques for monitoring tropical forest clearing and vegetation regrowth in a time series. Photogramm. Eng. Remote Sens. 67 (9), 1067–1075.
Herrero, H., Southworth, J., Muir, C., Khatami, R., Bunting, E., and Child, B. (2020). An Evaluation of Vegetation Health in and around Southern African National Parks during the 21st Century (2000-2016). Appl. Sci. 10 (7), 2366. Special Issue: Dynamics of the Global Savanna and Grasslands Biomes. doi:10.3390/app10072366
Jeon, G., Bellandi, V., and Chehri, A. (2020). Editorial for the Special Issue "Advanced Machine Learning for Time Series Remote Sensing Data Analysis". Remote Sensing 12 (17), 2815. doi:10.3390/rs12172815
Kawamura, M., Jayamana, S., and Tsujiko, Y. (1996). Relation between social and environmental conditions in Colombo Sri Lanka and the urban index estimated by satellite remote sensing data. Int. Arch. Photogramm. Remote Sens. 31 (Part B7), 321–326.
Kennedy, R. E., Yang, Z., Braaten, J., Copass, C., Antonova, N., Jordan, C., et al. (2015). Attribution of disturbance change agent from Landsat time-series in support of habitat monitoring in the Puget Sound region, USA. Remote Sensing Environ. 166, 271–285. doi:10.1016/j.rse.2015.05.005
Kennedy, R. E., Yang, Z., and Cohen, W. B. (2010). Detecting trends in forest disturbance and recovery using yearly Landsat time series: 1. LandTrendr - Temporal segmentation algorithms. Remote Sensing Environ. 114 (12), 2897–2910. doi:10.1016/j.rse.2010.07.008
Liu, Q., Zhang, S., Zhang, H., Bai, Y., and Zhang, J. (2020). Monitoring drought using composite drought indices based on remote sensing. Sci. Total Environ. 711, 134585. doi:10.1016/j.scitotenv.2019.134585
Liu, X., Deng, C., Zhao, B., and Chanussot, J., (2019). Multimodal-Temporal Fusion: Blending Multimodal Remote Sensing Images to Generate Image Series With High Temporal Resolution, 2019, pp. 10083–10086. doi:10.1109/IGARSS.2019.8898453
Ma, L., Liu, Y., Zhang, X., Ye, Y., Yin, G., and Johnson, B. A. (2019). Deep learning in remote sensing applications: A meta-analysis and review. ISPRS J. Photogrammetry Remote Sensing 152, 166–177. doi:10.1016/j.isprsjprs.2019.04.015
Maxwell, A. E., Warner, T. A., and Fang, F. (2018). Implementation of machine-learning classification in remote sensing: An applied review. Int. J. Remote Sensing 39 (9), 2784–2817. doi:10.1080/01431161.2018.1433343
Mondal, P., and Southworth, J. (2009). Protection vs. Commercial Management: Spatial and Temporal Analysis of Land Cover Changes in the Tropical Forests of Central India. For. Ecol. Manag. 259 (5), 1009–1017.
Muir, C., Southworth, J., Khatami, R., Herrero, H., and Akyapı, B. (2021). Vegetation Dynamics and Climatological Drivers in Ethiopia at the Turn of the Century. Remote Sensing 13 (16), 3267. doi:10.3390/rs13163267
Peng, M., Zhang, L., Sun, X., Cen, Y., and Zhao, X. (2020). A Fast Three-Dimensional Convolutional Neural Network-Based Spatiotemporal Fusion Method (STF3DCNN) Using a Spatial-Temporal-Spectral Dataset. Remote Sensing 12 (23), 3888. doi:10.3390/rs12233888
Southworth, J., Nagendra, H., Carlson, L. A., and Tucker, C. (2004). Assessing the impact of Celaque National Park on forest fragmentation in Western Honduras. Appl. Geogr. 24, 303–322. doi:10.1016/j.apgeog.2004.07.003
Southworth, J., Zhu, L., Bunting, E., Ryan, S. J., Herrero, H., Waylen, P., et al. (2016). Changes in vegetation persistence across global savanna landscapes, 1982–2010. J. Land Use Sci. 11 (1). doi:10.1080/1747423x.2015.1071439
Verbesselt, J., Hyndman, R., Newnham, G., and Culvenor, D. (2010). Detecting trend and seasonal changes in satellite image time series. Remote Sensing Environ. 114 (1), 106–115. doi:10.1016/j.rse.2009.08.014
Yengoh, G. T., Dent, D., Olsson, L., Tengberg, A. E., and Tucker, C. J. (2014). The use of the Normalized Difference Vegetation Index (NDVI) to assess land degradation at multiple scales: A review of the current status, future trends, and practical considerations. Sweden: Lund University Center for Sustainability Studies LUCSUS, and The Scientific and Technical Advisory Panel of the Global Environment Facility STAP/GEF.
Yuan, Q., Shen, H., Li, T., Li, Z., Li, S., Jiang, Y., et al. (2020). Deep learning in environmental remote sensing: Achievements and challenges. Remote Sensing Environ. 241, 111716. doi:10.1016/j.rse.2020.111716
Zha, Y., Gao, J., and Ni, S. (2003). Use of normalized difference built-up index in automatically mapping urban areas from TM imagery. Int. J. Remote Sensing 24 (3), 583–594. doi:10.1080/01431160304987
Zhu, L., and Southworth, J. (2012). Disentangling the relationships between net primary production and precipitation in southern Africa savannas using satellite observations from 1982 to 2010. Remote Sensing 5, 3803–3825. doi:10.3390/rs5083803
Zhu, X. X., Tuia, D., Mou, L., Xia, G.-S., Zhang, L., Xu, F., et al. (2017b). Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources. IEEE Geosci. Remote Sens. Mag. 5 (4), 8–36. doi:10.1109/MGRS.2017.2762307
Zhu, Z. (2017a). Change detection using landsat time series: A review of frequencies, preprocessing, algorithms, and applications. ISPRS J. Photogrammetry Remote Sensing 130, 370–384. doi:10.1016/j.isprsjprs.2017.06.013
Keywords: global environmental change (GEC), artificial intelligence, deep learning, big data, time series, Data Fusion
Citation: Southworth J and Muir C (2021) Specialty Grand Challenge: Remote Sensing Time Series Analysis. Front. Remote Sens. 2:770431. doi: 10.3389/frsen.2021.770431
Received: 03 September 2021; Accepted: 14 September 2021;
Published: 24 September 2021.
Edited and reviewed by:Jose Antonio Sobrino, University of Valencia, Spain
Copyright © 2021 Southworth and Muir. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.