We collected 211 adult P. trituberculatus from a crab farm in Ningbo in March 2020. The appendages of the adult P. trituberculatus were intact. The average body weight of the experimental crab was 318.90 ± 38.07 g (mean ± SD), the full carapace width was 16.73 ± 0.75 cm, and the length was 8.65 ± 0.41 cm. After numbering the P. trituberculatus, the carapaces were pictured with a mobile phone (Huawei P30 rear camera).

The P. trituberculatus were numbered from 1 to 211, and the carapace images were used to compose a dataset named crab-back-211. The carapace pictures were taken in multiple scenes to augment the diversity of dataset crab-back-211. The description of the scenes is shown in Table 1. In Table 1, “25” represents the ID of P. trituberculatus in crab-back-211, and “0–11” are the IDs of 12 scenes. In the 12 scenes, ID 0 is a standard scene without any processing. By diversity augmentation, the crab-back-211 was expanded to 2,532 images. Figure 1 shows the carapace images of the 25th P. trituberculatus in 0–11 scenes, where 25_0 is the standard scene.

We divided the crab-back-211 into the training set and test set, including 154 crabs and 57 crabs, respectively. The training set was used to train ReID model, while the test set was used to evaluate the performance of the trained model. The training set contained 154 P. trituberculatus with a total of 1,848 images, and the test set had 57 P. trituberculatus with a total of 684 images. The test set consisted of a query set and a gallery set. P. trituberculatus ReID aimed at matching images of a specified P. trituberculatus in gallery set, given a query image in the query set. The correct matches could be found by the similarities of carapace features obtained by Euclidean distance. First, carapace features of P. trituberculatus in query or gallery set were obtained by pyramid-based ReID (PR). Then, all the similarities of carapace features between query images and gallery images could be obtained by Euclidean distances. By the similarities, the match of a query image was found from the gallery images. In this experiment, 57 query images were selected as the query set to find the correct match across 627 gallery images in the gallery set. Table 2 shows the configuration of the experiment platform.

We tested the proposed method on crab-back-211. The experiment consisted of two stages, namely, the model training stage and the model inference stage.

For P. trituberculatus ReID, the traditional ReID method takes the whole P. trituberculatus carapace image as input, to extract the whole feature. This extracted feature represents P. trituberculatus individual. By the feature similarity between query images and gallery images, the matching result of a query image can be found in the gallery set. However, these traditional methods overemphasize the global features of the P. trituberculatus individual and ignore some insignificant detailed features. These methods could fail to distinguish similar P. trituberculatus individuals. Therefore, many pieces of research comprehensively consider global features and local features and focus on the contribution of key local areas to the whole feature. To extract local features, an image should be divided into fixed parts. For example, in the field of pedestrian ReID, a pedestrian image is divided into three fixed parts, namely, head, upper body, and lower body, and the divided local features are extracted, respectively. In terms of part-based methods, different part-based frameworks were adopted to improve the performance of ReID (Sun et al., 2018; Fu et al., 2019). The critical point of part-based methods is to align the divided fixed parts.

For P. trituberculatus, there were many carapace characteristics, such as protrusions, grooves, ridges, and irregular spots. Figure 2 shows two P. trituberculatus carapaces. The carapace texture and spots have significant individual discrimination. Therefore, we focused on the local texture and spot characteristics of P. trituberculatus carapaces to improve ReID accuracy.

Backbone was a network for feature extraction. The objective of the backbone was to extract original features and feed the features into the next. ResNet framework (He et al., 2016) is an effective backbone with a strong feature extraction ability on image processing, such as object classification and segmentation. Therefore, we used ResNet to extract original features on P. trituberculatus carapace. Furthermore, many backbone networks, such as Resnet-50 and Resnet-101, can also be used as the basic network, and these backbones were compared with each other in the experiments.

To extract the global and local features of P. trituberculatus, a part-based strategy was used to divide the original features into six fixed parts as shown in Figure 3, after backbone. In Figure 5, the input of the feature pyramid was the original features extracted by the backbone, and then 18 groups in 4 levels by the six fixed parts could be obtained by the pyramid-based strategy.

The basic block used in this study is shown in Figure 5, which includes the pooling layer, convolutional layer, batch normalization (BN), rectified linear units (ReLU), full connection (FC), and softmax layers. There were 18 blocks in the basic block, which could process the 18 features of 18 groups, after the feature pyramid. For each block in the basic block, global average pooling (GAP) and global maximum pooling (GMP) were used to capture the characteristics of different channels, such as protrusions, grooves, ridges, and irregular spots. Then, two features in the same channel were added into a vector. Later, a convolution layer was followed by BN and ReLU activation. The features by 18 blocks were concatenated, as shown in Figure 6. ID loss and triplet loss were adopted to train the PFFM model and to discriminate the subtle discrimination of overall features. In the inference stage, the concatenated feature was used to identify P. trituberculatus.

When triplet loss for model training was used, each P. trituberculatus individual was selected as an anchor. For each anchor, the sample with the same ID and the lowest feature similarity to the anchor was selected as a positive sample. Conversely, the sample with the different ID and the highest feature similarity was selected as a negative sample. By using the above process, the selected three samples (i.e., anchor sample, positive sample, and negative sample) formed a triplet tuple. Figure 7 shows the positive and negative sample selection processes.

As shown in Figure 7, each P. trituberculatus was photographed in different scenes. x_j represents the pictures of P. trituberculatus j, where xji is the picture of P. trituberculatus j in scene i. When xia was selected as anchor sample, xip was a sample of ID i. If xip had the lowest similarity with xia, xip was selected as positive sample. As shown in Figure 7, the ID of xkn was k, and if xkn had the highest similarity with xia, xkn was selected as negative sample. The selected xia,xip,xkn(a≠p,i≠k) composed a triplet tuple. The tuple composed by anchor and positive sample, denoted as (xia,xip), was positive sample pair, while the tuple by anchor and negative sample, denoted as (xia,xkn), was negative sample pair. The purpose of triplet loss was to minimize the feature distance of the positive sample pair in each scene and maximize the feature distance of the negative sample pair. Triplet loss was expressed in the following:

where f(⋅) is the feature extraction function, ||⋅|| is the Euclidean distance function, f(xia), f(xip), f(xkn) denote the features of anchor sample xia, positive sample xip, and negative sample xkn, respectively, α is a hyperparameter, and the value here is 0.2.

The purpose of this experiment was to verify the effectiveness of PFFM. This experiment used three backbones (i.e., Resnet18, Resnet50, and Resnet101) to extract the original image feature and made six groups for the comparison methods, such as Resnet18, Resnet18-pyramid, Resnet18, Resnet50-pyramid, Resnet50, and Resnet101-pyramid. The groups of Resnet18-pyramid, Resnet50-pyramid, and Resnet101-pyramid adopted the proposed PFFM as the ReID model. To verify the robustness of each method, we occluded the left half part of the images in the query set and carried out the six comparison methods above on the retained right half part, to find the ReID target in the gallery set. The experimental results are shown in Table 3. The methods using PFFM, such as Resnet18-pyramid, Resnet50-pyramid, and Resnet101-pyramid, performed better than those not using PFFM. The mAP and rank-1 of Resnet18-pyramid increased by 39.6 and 19.3% compared with Resnet18. Among these comparison methods, the methods using PFFM performed best on mAP and rank-1.

The Resnet18, Resnet50, and Resnet101 models mainly used global features of P. trituberculatus for ReID. When the query set was occluded, a large global feature deviation would occur, which led to a decrease in mAP, rank-1, and rank-5. The proposed PFFM utilized local features to compensate for global features, to alleviate the global feature deviation problem, and to strengthen its robustness. This experiment also showed that PFFM with Resnet101 as backbone performed better than other comparison methods.

In contrast, to verify the optimal division strategy of PFFM, this experiment used two-division strategies, i.e., grid 2 × 3 and grid 2 × 4. This experiment used three backbones (i.e., Resnet18, Resnet50, and Resnet101) to extract the original image feature, and the results obtained using grid 2 × 3 and grid 2 × 4 are shown in Table 4. The model with grid 2 × 4 had more parameters than the model with grid 2 × 3. In accuracy contrast, the model with grid 2 × 3 also had an advantage. For grid 2 × 4, the divided parts were too small to maintain the continuity of local features, which could affect ReID accuracy and increase the burden of model training. By using this experiment, the optimal division of PFFM was grid 2 × 3.

To verify the compatibility of the PFFM, this experiment used different backbones with PFFM to analyze the feature distance of P. trituberculatus. This experiment calculated the feature distances of the samples with the same ID, and the feature distances of the samples with the different ID. Figure 8 shows the feature distance distribution of P. trituberculatus by resnet18, resnet50, and resnet101. In Figure 8, the x-axis indicates P. trituberculatus IDs, and the y-axis is the distance that shows the distances between P. trituberculatus. The dark scatter points represent the feature distances among the samples with the same ID, and the scatter light points represent the feature distances among the samples with a different ID. The dark scatter points are mainly distributed at the bottom, indicating that the distances among the samples with the same ID were small. The distances of the samples with different IDs were greater and had a large fluctuation. By the above analysis, the samples with the same ID were closer, while the samples were farther with different IDs. In addition, the PFFM provided a high discriminative ability, especially when aggregated from fixed part features. The discriminating features reflected the specificity of each P. trituberculatus, which was more suitable for P. trituberculatus ReID. By using this experiment, the PFFM could achieve better results on crab-back-211 with various backbones, so the proposed PFFM had better compatibility with P. trituberculatus ReID.

This experiment compared PFFM with other methods on crab-back-211. Table 5 shows the comparison results. In this experiment, we occluded the left or top half of the images in the query set, to test the robustness of these methods. The occlusion was used to simulate the worse scenes in practice. We selected Vgg16, Resnet50, and PCB as comparison methods. For ReID models, the backbone was the preceding process used to generate original features, such as Vgg16 and Resnet50. The ReID models extracted features of P. trituberculatus from the original features. The Euclidean distance between the extracted features was used to identify P. trituberculatus individuals. For example, PCB was a ReID model that divided the original feature into two horizontal blocks for training and inference. Our study proposed PFFM using Resnet50 as the backbone. It was seen from Table 5 that the mAPs of Vgg16 and Resnet50 were much lower than those of PCB and PFFM that implied the models using part-based strategy had better robustness. Therefore, the method that only considered global features could not accurately identify P. trituberculatus with the incomplete carapace. The mAP of PFFM used in this experiment was 9.6% higher than that of PCB in the upper half occlusion case, while the rank-1 was 26.3% higher and the rank-5 was 7% higher. Therefore, the PFFM proposed had better robustness and could effectively identify the P. trituberculatus individual with the incomplete carapace.