Deep learning methods for 3D tracking of fish in challenging underwater conditions for future perception in autonomous underwater vehicles

Martin Føre^*Emilia May O'BrienEleni Kelasidi

• Norwegian University of Science and Technology, Trondheim, Norway

Due to their utility in replacing workers in tasks unsuitable for humans, unmanned underwater vehicles (UUVs) have become increasingly common tools in the fish farming industry. However, earlier studies and anecdotal evidence from farmers imply that farmed fish tend to move away from and avoid intrusive objects such as vehicles that are deployed and operated inside net pens. Such responses could imply a discomfort associated with the intrusive objects that in turn can lead to stress and impaired welfare in the fish. To avoid this, vehicles and their control systems should be designed to automatically adjust operations when they perceive that they are repelling the fish. A necessary first step in this direction is to develop on-vehicle observation systems for assessing object/vehicle-fish distances in real-time that can provide inputs to the control algorithms. Due to their small size and low weight, modern cameras are ideal for this purpose. Moreover, the ongoing rapid developments within deep learning methods are enabling increasingly sophisticated methods for analysing footage from cameras. To explore this potential, we developed three new pipelines for the automated assessment of fish-camera distances in video and images. These methods were complemented by a recently published method, yielding four pipelines in total, SegmentDepth, BBoxDepth and SuperGlue that were based on stereo-vision and DepthAnything that was monocular. Overall performance was investigated using field data and comparing the fish-object distances from the methods with data measured using a sonar. The four methods were then benchmarked by comparing the number of objects detected, and the quality and overall accuracy of the stereo matches (only stereo-based methods). SegmentDepth, DepthAnything, and SuperGlue performed well in comparison with the sonar data, yielding mean absolute errors (MAE) of 0.205 m (95% CI: 0.050-0.360), 0.412 m (95% CI: 0.148-0.676) and 0.187 m (95% CI: 0.073-0.300) respectively and were integrated into the ROS2 framework to enable real-time application on fish behaviour identification and control of robotic vehicles such as UUVs.