A survey on autonomous environmental monitoring approaches: towards unifying active sensing and reinforcement learning

The environmental pollution caused by various sources has escalated the climate crisis making the need to establish reliable, intelligent, and persistent environmental monitoring solutions more crucial than ever. Mobile sensing systems are a popular platform due to their cost-effectiveness and adaptability. However, in practice, operation environments demand highly intelligent and robust systems that can cope with an environment’s changing dynamics. To achieve this reinforcement learning has become a popular tool as it facilitates the training of intelligent and robust sensing agents that can handle unknown and extreme conditions. In this paper, a framework that formulates active sensing as a reinforcement learning problem is proposed. This framework allows unification with multiple essential environmental monitoring tasks and algorithms such as coverage, patrolling, source seeking, exploration and search and rescue. The unified framework represents a step towards bridging the divide between theoretical advancements in reinforcement learning and real-world applications in environmental monitoring. A critical review of the literature in this field is carried out and it is found that despite the potential of reinforcement learning for environmental active sensing applications there is still a lack of practical implementation and most work remains in the simulation phase. It is also noted that despite the consensus that, multi-agent systems are crucial to fully realize the potential of active sensing there is a lack of research in this area.


Platform
Open Table S1.Feature comparison of reinforcement learning development platforms.

TensorFlow
TensorFlow is a comprehensive open-source ML framework maintained by Google.In TensorFlow users create static data flow graphs that describe how data moves Abadi et al. (2015).The nodes of the graphs represent mathematical operations and the edges represent the flow of data.By manipulating these graphs users are provided a high amount of flexibility when building machine learning models.Keras is a popular tool for the implementation of Neural Networks (NN) with TensorFlow as it aims to be a minimalist and intuitive framework and can integrate with pre-built TensorFlow models to facilitate deep learning.TensorFlow comes with a steeper learning curve than other platforms, especially for users new to machine learning models.However, there is a large community of support and a pool of free learning resources due to its widespread popularity.

PyTorch
While Tensorflow has been widely recognized as an industry standard, PyTorch, which is a newer tool, is often considered more user-friendly and 'pythonic ' Paszke et al. (2019).It is good for quickly prototyping models and is gaining popularity in the research community.While the graphs in TensorFlow are static, PyTorch uses dynamic graphs that are executed during run-time.While this can be useful for debugging execution time can be longer for bigger systems.PyTorch has its own in-built deep learning framework that provides the equivalent functionality to Keras in TensorFlow.

Gymnasium
OpenAI's GYM is a popular open-source platform for RL as it is fully compatible with both TensorFlow and PyTorch Brockman et al. (2016).It provides pre-built benchmark-type environments that allow users to easily test their learning algorithm, some example environments are showing in Figure S1.Tools like GYM are not only useful to help streamline RL development but also offer means of standardization, a necessity that is lacking in RL and DRL research.In GYM's early life maintenance and development were poor and as of October 2022, OpenAI GYM is being maintained by the Farama Foundation whose aims are to provide standardization and long-term maintenance of RL libraries1 .The future of Gym will take place under the name of Gymnasium (although the names are often used interchangeably).The Farama Foundation is coordinating future development with other big RL projects like StableBaselines which we discuss in this section.The environments provided by Gymnasium have defined actions, observations and rewards.Different environments allow users to explore different action and observation spaces both continuous and discrete providing 2D visual renders of the environment.GYM also provides the tools for the user to build their own environments.It can also interface with the Robot Operating System (ROS) making it popular for robotics and extending the application to three-dimensional dynamics.

StableBaselines and StableBaselines3
StableBaselines is a set of implemented and up-to-date RL algorithms that is an expansion upon the original baselines by OpenAI Hill et al. (2018).Originally made in TensorFlow, a more recent package StableBaselines3 was realised for PyTorch Raffin et al. (2021).This is a comprehensive and easy-touse package that works well with OpenAI GYM.StableBaselines and StableBaselines3 allow users to easily implement state-of-the-art RL algorithms easily making it a strong platform for research with quick prototypes and intuitive use.This package if updated regularly can help researchers standardize benchmarking with state-of-the-art algorithms.

PettingZoo
PettingZoo Terry et al. (2021) is another project from the Farama Foundation that provides a simple and easy-to-use Python API.It provides similar benchmark environments and environment-building tools but for MARL, some example environments are shown in Figure S2.It provides both turn-based and parallel learning environments.Since MARL is a relatively new research area it provides a good standardization for bench-marking from the get-go.In terms of robotic applications, there aren't any standard environments that allow the training of a robot-like agent.However, there are a number of relevant third-party environments available through the petting zoo online documentation.For example, Crazy-RL provides an environment for RL application on crazyflie drones 2 .

MATLAB
MATLAB has long been a standard tool in engineering research and it offers a well-developed RL Toolbox 3 .The toolbox includes neural networks for representing policies (DRL) and a number of standard RL algorithms available to train these policies.MATLAB is generally considered easy to learn due to its simplicity, comprehensive documentation and active professional support.This makes MATLAB's RL Toolbox a good place for people to learn the foundations of RL.Furthermore, through the ONNX model format, the MATLAB toolbox is compatible with TensorFlow Keras and PyTorch, allowing users to take advantage of both MATLAB and pythonicc platforms.MATLAB also has Simulink which is a graphical user-friendly environment that can make the design process more intuitive.However, MATLAB doesn't have the same wide scope that Python has to offer in terms of community support, ongoing projects, extra libraries and ease of implementation with many other platforms.MATLAB is also a closed-source and proprietary which may make it void for the individual.