Fish Keypoint Detection for Offshore Aquaculture: A Robust Deep Learning Approach with PCA-based Shape Constraint

Li Gen^1,2Lian Anji¹Yao Zidan³Hu Yu^1,2Pang Guoliang^1,2Yuan Taiping^1,2Li Zhenhua³Huang Xiaohua^1,2*Wang Gang⁴

• ¹South China Sea Fisheries Research Institute, Chinese Academy of Fishery Sciences (CAFS), Guangzhou, China
• ²Sanya Tropical Fisheries Research Institute, Lingshui Li Autonomous County, China
• ³School of Marine Engineering Equipment, Zhejiang Ocean University, Zhoushan, Zhejiang Province, China
• ⁴Fisheries Engineering Institute, Chinese Academy of Fishery Sciences, Beijing, China

Fish keypoint detection is a prerequisite for accurate fish behavior analysis and biomass weight estimation, and is therefore crucial for efficient and intelligent offshore aquaculture. Traditional keypoint detection networks typically employ coordinate regression methods, which do not impose any constraints on the output of the regression head or the training process of the neural network. As a result, output keypoints of such networks do not always conform to the shape of a fish and the training process can be affected by incorrect labels, leading to errors in subsequent tasks. To address these issues, this paper proposes a robust deep learning approach characterized by three improvements. 1) A shape model of fish that includes the average shape of fish, principal components of fish keypoints, and corresponding eigenvalues is constructed using principal component analysis (PCA) and unscented transform. 2) A customized version of anchor boxes is introduced and referred to as "anchor fish", which along with the shape model, can be used to encode and decode fish keypoints. 3) Shape variation loss, calculated based on the eigenvalues in the shape model, is added as part of the loss function to constrain the output of the regression head. Moreover, we built a fish keypoint dataset using infrared cameras mounted on a truss-structure net cage. Comparative experiments on our dataset using the keypoint evaluation method from COCO are conducted. The results show that our method achieves an AP50 value of 0.656, significantly outperforming the well-designed YOLO5Face, which produces an AP50 value of 0.503. Furthermore, we have comprehensively explored the impact of key hyperparameters on detection performance and robustness to labeling outliers in the training set. The code is available at https://github.com/LMX-BY/fish_landmark_detection_using_PCA_based_fish_shape_model.