AUTHOR=Li Gen , Lian Anji , Yao Zidan , Hu Yu , Pang Guoliang , Yuan Taiping , Li Zhenhua , Huang Xiaohua , Wang Gang 

TITLE=Fish keypoint detection for offshore aquaculture: a robust deep learning approach with PCA-based shape constraint

JOURNAL=Frontiers in Marine Science

VOLUME=Volume 12 - 2025

YEAR=2025

URL=https://www.frontiersin.org/journals/marine-science/articles/10.3389/fmars.2025.1619457

DOI=10.3389/fmars.2025.1619457

ISSN=2296-7745

ABSTRACT=IntroductionFish keypoint detection is a prerequisite for accurate fish behavior analysis and biomass weight estimation, and is therefore crucial for efficient and intelligent offshore aquaculture. Traditional keypoint detection networks typically employ coordinate regression methods, which do not impose any constraints on the output of the regression head or the training process of the neural network. As a result, output keypoints of such networks do not always conform to the shape of a fish and the training process can be affected by incorrect labels, leading to errors in subsequent tasks.MethodsTo address these issues, this paper proposes a robust deep learning approach characterized by three improvements. 1) A shape model of fish that includes the average shape of fish, principal components of fish keypoints, and corresponding eigenvalues is constructed using principal component analysis (PCA) and unscented transform. 2) A customized version of anchor boxes is introduced and referred to as “anchor fish”, which along with the shape model, can be used to encode and decode fish keypoints. 3) Shape variation loss, calculated based on the eigenvalues in the shape model, is added as part of the loss function to constrain the output of the regression head. Moreover, we built a fish keypoint dataset using infrared cameras mounted on a truss-structure net cage.Results and discussionComparative experiments on our dataset using the keypoint evaluation method from COCO are conducted. The results show that our method achieves an AP50 value of 0.656, significantly outperforming the well-designed YOLO5Face, which produces an AP50 value of 0.503. Furthermore, we have comprehensively explored the impact of key hyperparameters on detection performance and robustness to labeling outliers in the training set. The code is available at https://github.com/LMX-BY/fish_landmark_detection_using_PCA_based_fish_shape_model.