Abstract
Aquatic biodiversity monitoring relies on species recognition from images. While deep learning (DL) streamlines the recognition process, the performance of these method is closely linked to the large-scale labeled datasets, necessitating manual processing with expert knowledge and consume substantial time, labor, and financial resources. Semi-supervised learning (SSL) offers a promising avenue to improve the performance of DL models by utilizing the extensive unlabeled samples. However, the complex collection environments and the long-tailed class imbalance of aquatic species make SSL difficult to implement effectively. To address these challenges in aquatic species recognition within the SSL scheme, we propose a Wavelet Fusion Network and the Consistency Equilibrium Loss function. The former mitigates the influence of data collection environment by fusing image information at different frequencies decomposed through wavelet transform. The latter improves the SSL scheme by refining the consistency loss function and adaptively adjusting the margin for each class. Extensive experiments are conducted on the large-scale FishNet dataset. As expected, our method improves the existing SSL scheme by up to 9.34% in overall classification accuracy. With the accumulation of image data, the improved SSL method with limited labeled data, shows the potential to advance species recognition for aquatic biodiversity monitoring and conservation.
1 Introduction
Aquatic biodiversity plays a crucial role in maintaining the structural integrity, stability, and overall health of ecosystems (). However, anthropogenic pressures from human activities have progressively intensified in recent decades, posing gradual challenges to the preservation of aquatic biodiversity (; ). A critical step in conserving aquatic biodiversity is monitoring the information regarding the abundance and distribution of aquatic animals, which relies heavily on extensive collections of underwater images and videos. Deep learning (DL) techniques have recently demonstrated significant progress in several computer vision tasks (), and offer a promising solution to automatic and effective species recognition from images (). Due to the profound influence of dataset size and diversity on the accuracy of DL methods, many previous efforts have focused on building extensive and publicly available labeled image datasets specifically for aquatic species recognition (; ; ). Unfortunately, the intricate taxonomy of species typically demands a high level of expertise in the aquatic domain, meanwhile the annotation process proves to be tedious and time-consuming ().
It is estimated that more than 300,000 hours of underwater video footage have been collected worldwide so far, with only less than 15% of the data annotated by biological and ecological experts (). As the pace of data collection accelerates annually, the substantial backlog exacerbates. Several strategies, including transfer learning (), data augmentation (), weakly supervised learning (), and active learning (), have been made to tackle this problem. For example, transfer learning necessitates fine-tuning newly labeled aquatic species datasets to maximize accuracy. Weakly-supervised learning, on the other hand, relies on a limited form of supervision, where the labels may be noisy, incomplete, or imprecise. Nonetheless, these studies still require access to large-scale labeled training sets. The significance of diversity and comprehensiveness in the training dataset undoubtedly plays a pivotal role in achieving high recognition accuracy during real-world model deployment. Given the existence of unlabeled data, the marine community has emphasized the need for a powerful approach to training DL methods on vast amounts of data without annotated labels. In contrast, semi-supervised learning (SSL) can handle scenarios with both labeled and unlabeled data, providing more flexibility and potentially better performance when limited labeled data is available (). To date, although numerous studies explore SSL to address the high cost of annotated labels in aquatic domain (; ; ), its application in aquatic environments for species recognition remains scarce.
Two major challenges conspire to hinder the use of SSL scheme for aquatic species recognition. The first challenge stems from the unique characteristics of collected environments, including diverse lighting, variable water turbidity, and complex visual backgrounds that can obscure visual information (; ). Furthermore, the movement of objects in an uncontrolled environment can introduce distortion, deformation, occlusion, and overlapping (; ). These factors increase complexities and hinder the ability of DL models to employ effectively from labeled to unlabeled data. The need for robust feature extraction methods tailored to the above challenges becomes paramount to ensure the practical applicability of the SSL scheme. The second challenge arises from the long-tailed class imbalance of aquatic species in collected images (). As shown in Figure 1A, a limited subset of species are characterized by a substantial number of samples (referred to as head classes), while others are linked to only a few samples (referred to as tail classes). The limited sample information of tail classes poses a significant hurdle for SSL scheme, as there is a risk that the model being biased toward head classes due to the abundance of samples ().
Figure 1
In this work, we propose a novel SSL scheme for aquatic species recognition, which is based on the existing SSL algorithm, FixMatch (). Specifically, to mitigate the complexities inherent in heterogeneous collected environments, we propose a robust wavelet fusion network (WFN) equipped with wavelet transform. The proposed network comprises two frequency-aware streams, one is dedicated to capturing subtle image details by focusing on high-frequency (HF) information, while the other aims to extract high-level semantics from low-frequency (LF) information. These streams are subsequently integrated through a FusionBlock, which facilitates attentive interactions between the LF and HF streams. Furthermore, for the problem of long-tailed nature when using unlabeled data, we design a new Consistency Equilibrium Loss (CEL) that refines the pseudo-labels and adaptively adjusts the margin for each aquatic species class. We find that replacing the unsupervised loss with CEL could ensure that the SSL algorithm achieves relative classification equilibrium, even if the collected data distribution is biased toward the head classes. Extensive experiments demonstrate the proposed method attains superior results on a large-scale aquatic species recognition dataset. In addition, the WFN and CEL are assessed to highlight their advantages over current common practices.
2 Related work
2.1 Aquatic species recognition with deep learning
In recent years, DL-based aquatic species recognition has emerged as a promising tool for assisting marine scientists and ecologists in better understanding and managing marine environments. Accurate species recognition serves as the cornerstone of aquatic biodiversity research, playing a crucial role in estimating species size and quantity. A seminal contribution in this field is the development of the filtering deep convolutional network (FDCNet) (), which effectively classifies deep-sea objects such as sea urchins, crabs, sharks, and shrimps. Due to the complexity and dynamics of the marine environment, DL methods encounter challenges in recognizing interesting objects based on visual characteristics. To overcome this issue, the literature () proposes an invariant feature-based species classification method for distinguishing octopus and crabs. Similarly, the study () introduces an improved fish recognition network along with a novel loss function, FishFace, designs to focus more attention on fish details. More recently, automated plankton recognizing method based on DL has been developed for continuous monitoring of living plankton abundance in aquatic environments (). A comprehensive review () is recommended for researchers to seek an in-depth understanding of DL-based aquatic species recognition methods. However, most existing methods are constrained by their reliance on a relatively small portion of labeled data, posing a challenge to their practical application in real-world scenarios (). Therefore, there is an urgent need to develop a new paradigm capable of effectively utilizing extensive unlabeled data with a small amount of labeled data to accurately identify a broader range of aquatic species, thereby supporting aquatic biodiversity conservation efforts.
2.2 Semi-supervised learning
SSL methods have garnered significant attention from both industry and academia for use unlabeled data during the training process, particularly when the amount of labeled data is scarce. Recent SSL research has generally been categorized into two main groups. The first category of consistency regularization methods imposes a classification invariance loss on unlabeled data following perturbation (; ). In the second category, pseudo-labeling extends model training data beyond labeled samples to contain additional unlabeled data, augmented with credible pseudo-labels (; ). Techniques like FixMatch () and RemixMatch () combine pseudo-labeling with consistency regularization, yielding superior performance compared to many other SSL algorithms in image recognition tasks. Furthermore, several studies have been conducted experiments on long-tailed SSL. For example, DARP () proposes eliminating biased pseudo-labels through distribution alignment, which refines the pseudo-labels based on the labeled data distribution. Additionally, an auxiliary balanced classifier learned by down-sampling the head class is used to enhance generalization capabilities (). The above designs largely promote the overall performance of long-tailed semi-supervised methods, but the performance of natural long-tailed SSL problems in aquatic species recognition is still unsatisfactory, and no research has been found that effectively addresses this issue.
2.3 Wavelet-based deep learning
The integration of wavelet transform with deep neural networks (DNNs) has gained traction due to its robust frequency and spatial representation capabilities. Common strategies involve utilizing wavelet transform as either a pre-processing or post-processing step (; ), as well as substituting specific layers in DNNs (). Previous research has also explored the application of the dual-tree complex wavelet transform to extract robust features from Synthetic Aperture Radar images (). More recently, Wave-ViT () uses the wavelet transform to down-sample keys/values in a Transformer (). The Multi-level Wavelet CNN () integrates wavelet package transform into the DNN to concatenate the LF and HF components and process them in a unified manner, despite the notable disparity between these components. In contrast, we employ wavelet transform as an effective approach to tackle image complexity. Further, none of these studies has attempted to design a fusion block specially tailored for the wavelet transform paradigm to obtain attentive feature representation.
2.4 Loss function for long-tailed learning
Re-weighting and Re-margining loss functions serve as key components in tackling long-tailed class imbalanced challenges (). These methods are primarily implemented by adjusting margins or loss weights based on the distribution of training data. For instance, seminal works (; ) reweight the loss functions according to the sampling frequency of each class. Recent literature () enhances the robustness of SSL to long-tailed class imbalanced problems by designing weights in the unsupervised loss based on estimating the learning difficulty of each class. In contrast, several studies (; ; ) have attempted to adjust the loss margins of each class. The Label-Distribution-Aware-Margin () approach motivates tail classes to have larger margins based on label frequencies. Additionally, the study () replaces the margin term with mean classification score for long-tailed object detection. While our CEL function is inspired by the above pioneer studies, it differs significantly in two aspects. Firstly, to the best of our knowledge, the CEL function is the first to utilize the mean classification score to extend the existing consistency loss in SSL. Secondly, our key idea involves refining pseudo-labels via the mean classification score to match the true data distribution. With the proposed CEL function, our approach demonstrates superior performance in aquatic species recognition based on the SSL scheme.
3 Method
In this section, we first revisit the formulation of the SSL scheme in Section 3.1. After that, we illustrate the process of generating LF and HF entities using wavelet transform in Section 3.2, and provide detailed insights into our FusionBlock in Section 3.3. Lastly, along with the SSL scheme, we introduce the CEL function for unlabeled samples in Section 3.4. An overview of the framework is shown in Figure 2.
Figure 2
3.1 Semi-supervised learning setup
The basic technique utilized in FixMatch () revolves around pseudo-labeling and consistency-regularization, where unlabeled samples with high confidence are selected as training samples. Suppose we have a labeled dataset where is the training sample is the corresponding label with C classes, and L is the number of labeled samples. represents a dataset comprising unlabeled samples, where U is the number of unlabeled samples. Both XL and XU share identical semantic labels. The loss function is composed of two terms: , where denotes the supervised loss applied to labeled data, is the consistency loss for unlabeled data, and is a scalar hyperparameter.
The supervised loss is defined as: , where η denotes the weak augmentation, B is the batch size, H is the cross-entropy loss, and p(·) is the output of logits in DNN. Pseudo-labels are generated from weakly augmented unlabeled samples, guiding the prediction of model on strongly augmented samples. The consistency loss Lu can be formally expressed as: , where represents strong augmentation, governs the proportion of labeled to unlabeled samples in a minibatch, and II is the indicator function; 0 if the highest probability of unlabeled samples is below the confidence threshold τ and 1 otherwise.
3.2 Wavelet transform
The wavelet transform serves as an effective frequency analysis tool, establishing extensive applications in signal processing (). A wavelet is linked with wavelet and scaling functions, which establish a relationship with the low-pass and high-pass filters to facilitate data decomposition. In practice, the images represent discrete non-stationary signals, involving various frequency intervals and spatial location information. Single-level 2D discrete wavelet transform (Equation 1) with four filters (and ) are often used to decompose an image x to obtain its LF component LF and three HF components ,
where , denote the convolution operation with the typical filter fi and downsampling operation, receptively. The components acquired through wavelet transform contain distinct information about the raw images of aquatic species (see Figures 3A, B). Our method strives to leverage wavelet transform to generate robust information as the input of DNN to extract LF and HF features. As such, a LF entity is represented solely by a LF component (Equation 2), while a HF entity is represented as a set of HF components in various directions (Equation 3) similar to those used in ():
Figure 3
Note that our average HF components aim to reduce computational costs by decreasing the number of subsequent encoders. Ideally, each HF component would be feature-extracted by a specific encoder, but this is computationally expensive. In contrast, our average strategy is orthogonal and complements previous practical approaches for handling HF components, such as element-wise addition (), concatenation (; ; ), and maximum (). We refer to Section 4.5 for further details on these.
Figures 3C, D visually illustrate the LF and HF entities as defined above. By using the LF entity as input, DNN can focus more on LF semantics due to its less noise. In contrast, the HF entity, while exhibiting more noise, offers clearer object boundaries and shapes, enabling DNN to concentrate on HF details. A similar perspective has been adopted by who argues that HF information typically represents image details, while LF information often embodies abstract semantics.
3.3 FusionBlock
Given the entities processed by wavelet transform, we employ parallel encoders equipped with ResNet-50 () to respectively generate high-level LF and HF features. These features are then passed through the FusionBlock to generate attentive features from one stream to another. We argue that applying a cross-stream attention strategy to high-level features can capture the connection between conceptual entities in the LF and HF streams, helping subsequent modules in recognizing aquatic objects in images.
Taking and as example to illustrate the details of FusionBlock (where w, h, c and b denote width, height, channel number, and batch size), we use a cross-stream attention strategy to explore correlations between the two streams. Specifically, as shown in Figure 4, the two features are passed through four 1 × 1 convolutional layers to generate query and key matrices. We reshape the query and key matrices into 3D spatial feature maps (w × h × cb), and then concatenate them to obtain the fused key and query as (Equations 4, 5):
Figure 4
where R denotes the reshape operation, and . After that, the attention map is computed by performing a dot-product and applying the softmax activation function (Equation 6).
where σ is the softmax activation function. In this way, the feature from one stream could serve to augment another stream. Additionally, to preserve the original information of each stream, a residual connection is employed to fuse the enhanced features with their original counterparts. As such, we obtain the cross-stream attentive features for the two streams as (Equation 7):
where Ⓜ denotes element-wise multiplication, Bconv1×1(·) represents a sequential operation combining a 1 × 1 convolutional layer and batch normalization. Once obtaining the cross-stream feature representation, we concatenate these features and apply the dropout operator to the fused feature fS. Finally, two fully connected layers are utilized to output the final logits.
3.4 Consistency equilibrium loss
In a recent study (), it was demonstrates that the learning status of a class can be inferred through the mean classification scores. When we take a deeper look into Figure 1B, it is evident that the head classes exhibit higher mean classification scores, whereas tail classes illustrate lower mean classification scores. Based on this observation, we follow the finding of utilizing the mean classification score to adjust the learning effectiveness of each class throughout the training process. The update process of the mean classification score during training can be illustrated as (Equation 8):
where denotes the mean classification score, initialized for each class using is the mean predicted probability of the sample in a mini-batch, and m is a hyper-parameter.
Previous research () has revealed that the performance of SSL scheme is highly sensitive to the quality of pseudo-label, and a long-tailed data distribution leads to biased predictions favoring head classes. Utilizing these pseudo-labels in the SSL scheme can be harmful for tail classes. Instead of solely adjusting the class-dependent margin by deriving the mean classification score, the confirmation bias in pseudo-labels should be alleviated at the same time. To this end, we first refine the original pseudo-labels via mean classification score so that match the true data distribution (Equation 9):
where θ is a hyper-parameter. Simultaneously, we adaptively adjust the margin by encouraging the tail classes to have larger margins. According to the mean classification score, we add a tunable term to balance the classification, similar to the previous study (). As such, the CEL can be written as (Equation 10):
We can control the training process through hyper-parameter θ to ensure the model remains unbiased towards the head classes and does not neglect tail classes. In particular, we increase the larger margin with lower mean classification scores for tail classes, mitigating the suppression of head classes over tail classes to balance the consistency loss.
4 Results
4.1 Dataset and evaluation metrics
Extensive experiments are conducted using the large-scale FishNet dataset (), comprising 94,532 images encompassing 17,357 distinct species. Each species is represented by at least one associated image, which span 8 taxonomic classes, 83 orders, 463 families, and 3,826 genera. To validate the effectiveness and universality of our proposed method, we focus on the family classification task. The FishNet dataset categorizes family classes into three groups based on the class frequencies: common, medium, and rare. There are a total of 6 categories in the common group, 52 categories in the medium group, and 405 categories in the rare group. In our experiments, we report the class average accuracy for each group, as well as the overall accuracy over all categories followed as official metrics. The FishNet contains 75,631 images in the training set and 18,901 images in the test set. Unless otherwise stated, we conduct the experiments with a ratio of 20% labeled samples in the training set as labeled data, and the remaining 80% data in the training set as unlabeled data, adhering to the common semi-supervised experimental partition.
4.2 Implementation details
We implement our model using PyTorch (), with both training and inference procedures conducted on the NVIDIA GeForce RTX 3090 GPU. We use 200 epochs in the training process. In each training step, our batch contains 12 labeled examples and 48 unlabeled examples, maintaining a ratio µ = 4 to support the SSL scheme. To ensure a smooth start, we incorporate linear learning rate warm-up for the first 50 steps, progressively increasing the initial value to 0.004. Subsequently, we decay the learning rate at epochs 30, 60, 100, and 150 by multiplying it by 0.1. For all experiments, the two-stream encoders are initialized with weights pre-trained on the ImageNet dataset (). We adapt a relatively larger coefficient m = 0.99 for the mean classification score update. For the unsupervised loss function CEL, the weight parameter (λ) increases linearly per epoch according to , and the confidence threshold τ is set to 0.95. As in previous works (; ), we employ an exponential moving average of model parameters to generate the final performance. We keep other hyper-parameters the same as the ImageNet experiments in FixMatch, except for those mentioned above.
4.3 Comparison of aquatic species recognition performance
Several experiments are conducted to elaborate the findings: (a) the baseline utilizing only labeled images for aquatic species recognition based on ResNet-50 (); (b) an improved version of (a) incorporating our proposed WFN to enhance image features with wavelet transform; (c) the baseline utilizing both labeled images and unlabeled images based on the representative SSL scheme FixMatch (); (d) the proposed WFN integrated into the FixMatch scheme; (e-i) evaluation of state-of-the-art methods designed for long-tailed SSL on the FishNet dataset; (j) utilization of CEL combined with FixMatch; (k) is the final version of our proposed methods incorporating both CEL and WFN into the FixMatch scheme. Table 1 presents the performance of different methods on the FishNet dataset. Based on these results, several observations emerge regarding the overall progress of the proposed method and variations among different supervised types.
Table 1
| Method | SSL | LT | Common | Medium | Rare | All | |
|---|---|---|---|---|---|---|---|
| a) | ResNet-50 () | – | – | 70.63 | 57.65 | 21.41 | 26.12 |
| b) | WFN | – | – | 72.73 | 58.60 | 25.47 | 29.80 |
| c) | FixMatch () | ✓ | – | 79.07 | 64.94 | 22.24 | 27.77 |
| d) | FixMatch + WFN | ✓ | – | 81.65 | 67.61 | 27.99 | 33.13 |
| e) | Fixmatch+CReST () | ✓ | ✓ | 68.19 | 67.26 | 24.93 | 30.24 |
| f) | Fixmatch+ABC () | ✓ | ✓ | 69.14 | 66.71 | 24.98 | 30.24 |
| g) | Fixmatch+DARP () | ✓ | ✓ | 69.74 | 67.42 | 26.19 | 31.38 |
| h) | FixMatch+SAW () | ✓ | ✓ | 64.54 | 67.18 | 27.31 | 32.27 |
| i) | FixMatch+DASO () | ✓ | ✓ | 65.74 | 67.70 | 27.07 | 32.13 |
| j) | FixMatch+CEL | ✓ | ✓ | 67.75 | 68.83 | 28.30 | 33.36 |
| k) | FixMatch+CEL+WFN | ✓ | ✓ | 69.58 | 68.36 | 32.61 | 37.11 |
Comparison with supervised, semi-supervised, and long-tailed semi-supervised methods on the FishNet dataset.
From a → b, it is evident that the WFN significantly improves overall performance. WFN achieves competitive performance, with average classification accuracy of 72.73%, 58.60%, 25.47%, and 29.80%, surpassing the ResNet-50 by 2.1%, 0.95%, 4.06%, and 3.68% over four metrics. The experiment demonstrates that WFN equipped with wavelet transform and FusionBlock, has better generalization than previous ResNet-50 architecture, which allows the model to tackle the challenges posed by the heterogeneous aquatic environment. From a → c, we can observe that the use of SSL yields a notable enhancement compared to the model trained solely using labeled data. The gain from unlabeled data becomes evident in the aquatic species recognition. SSL enables the DL model to leverage the abundance of unlabeled images, further refining its understanding of various species and environmental conditions.
From b → d, we can infer a similar conclusion to a → c. Furthermore, the combination of WFN and SSL yields a synergistic effect, tackling the challenges posed by the heterogeneity of aquatic environments while leveraging the benefits afforded by unlabeled data. Incorporating WFN into the SSL scheme enables the DL model to acquire robust features across diverse aquatic conditions. In other words, it is crucial to acknowledge that enhanced performance of the robust feature extraction method within SSL extends beyond the initial finding observed in a to c.
Table 1 also compares the proposed CEL function with several other methods: CReST (), which oversample tail classes generated by pseudo-labels, ABC (), utilizing an auxiliary balanced classifier of a single layer, DARP (), refining pseudo-labels to match the true distribution of unlabeled data, SAW , adjusting weights based on the estimated learning difficulty of each class in unsupervised loss, and DASO (), employing a blending pseudo-labels strategy to mitigate the overall bias. Since these methods were originally experiment with in the long-tailed SSL domain, we evaluate their performance on the FishNet dataset. We utilize publicly available code to train each method and report the best results obtained from multiple runs, fine-tuning their hyper-parameters to ensure optimal performance. From (e, f, g, h, i) → j, we observe that our CEL achieves competitiveness with other methods on the FishNet dataset. From c → (e, f, g, h, i, j), the long-tailed extensions yield performance gains of varying degrees for all methods, such as a notable 4.36% increase in average classification accuracy for DASO, demonstrating the importance of long-tailed distribution as a general issue for the task of aquatic species recognition.
Lastly, group (k) demonstrates that integrating WFN and CEL within SSL enhances overall performance for the aquatic species recognition task. The collaborative integration of WFN and CEL could leverage the strengths of each component. WFN enhances the feature extraction capabilities of the model, enabling better handling of the complexities of the aquatic environment. Meanwhile, CEL guides the training process, ensuring that the model benefits from unlabeled data and mitigating long-tailed class imbalanced problems. Through rigorous evaluation, we demonstrate that the combined strength of WFN and CEL contributes to a more robust and accurate aquatic species recognition system, paving the way for advancements in the field of aquatic biodiversity research and conservation.
4.4 Ablation study
4.4.1 Impact of different wavelet bases in WFN
Table 2 presents an analysis of various wavelet bases trained on labeled data, including Dmey, Haar, Daubechies 2, Coiflets, Biorthogonal 1.5, and Biorthogonal 2.4. The results we obtained show that the Daubechies 2 wavelet has better classification accuracy, and the Haar wavelet presents better border accuracy. As such, we select the Daubechies 2 wavelet basis as the default for our experiments.
Table 2
| Wavelet bases | Common | Medium | Rare | All |
|---|---|---|---|---|
| Dmey | 58.86 | 44.36 | 18.48 | 21.91 |
| Haar | 70.84 | 55.90 | 25.08 | 29.13 |
| Coiflets | 60.57 | 46.59 | 21.44 | 24.77 |
| Biorthogonal 1.5 | 58.35 | 45.43 | 18.98 | 22.46 |
| Biorthogonal 2.4 | 60.18 | 44.65 | 19.38 | 22.74 |
| Daubechies 2 | 72.73 | 58.60 | 25.47 | 29.80 |
Ablation study for the wavelet bases in WFN.
4.4.2 Ablation study of different coefficient θ in CEL
We perform an ablation study on the CEL function with various values of θ to evaluate the impact of model performance. As shown in Figure 5, an improper proportion of the term, either too large or too small, impedes the attainment of optimal performance. Observing the CEL function reveals a significant variation in the impact of θ. When the value of θ is set to 0, the CEL is equivalent to the consistency loss of FixMatch. However, excessively large values of θ may hinder the ability of model to focus attention on the data, whereas too small values inadequately addresses the bias in long-tailed SSL problem. The trade-off between model performance and CEL when θ = 0.4 achieves the relatively best performance.
Figure 5
4.4.3 Comparison of fusion strategies for WFN
We further examine the effectiveness of the proposed FusionBlock in Table 3. We utilize different feature fusion strategies to train the DNNs on the labeled images combined with wavelet transform. The proposed FusionBlock achieves better performance on the FishNet test set compared with element-wise add operation and concatenate features along with channel dimension. We believe that the cross-stream attention fusion strategy is more effective for learning interactive features, making it well-suited for the diverse and challenging environment in aquatic species recognition.
Table 3
| Fusion strategy | Common | Medium | Rare | All |
|---|---|---|---|---|
| concatenate | 75.14 | 58.78 | 22.88 | 27.59 |
| Element-wise add | 72.68 | 58.93 | 23.42 | 28.05 |
| FusionBlock | 72.73 | 58.60 | 25.47 | 29.80 |
Ablation study for feature fusion strategies in WFN.
4.5 Analysis of different frequency components
Since the main semantic information is conveyed in the LF component, previous studies have often used the LF component alone in certain tasks (; ). However, researchers have attempted to aggregate HF components with methods such as concatenation (; ; ), maximum (), or element-wise addition (), and incorporate them into DNNs to improve model performance. To verify the effectiveness of our WFN, we compare the performance of experiments conducted by using different components alone and the ways to connect the HF components. We report the results on FishNet’s labeled data in Table 4.
Table 4
| Raw | LF | HFSum | HFMax | HF Concate | HFAverage | Common | Medium | Rare | All |
|---|---|---|---|---|---|---|---|---|---|
| ✓ | 70.63 | 57.65 | 21.41 | 26.12 | |||||
| ✓ | 69.18 | 56.12 | 22.52 | 26.90 | |||||
| ✓ | 56.25 | 40.65 | 14.28 | 17.78 | |||||
| ✓ | 45.22 | 30.88 | 11.17 | 13.83 | |||||
| ✓ | 48.95 | 29.72 | 12.85 | 15.22 | |||||
| ✓ | 57.19 | 41.09 | 14.14 | 17.72 | |||||
| ✓ | ✓ | 69.10 | 55.48 | 23.60 | 27.77 | ||||
| ✓ | ✓ | 71.71 | 56.25 | 23.16 | 27.50 | ||||
| ✓ | ✓ | 68.90 | 56.30 | 24.41 | 28.57 | ||||
| ✓ | ✓ | 72.73 | 58.60 | 25.47 | 29.80 |
Analysis of different frequency components.
The results show that both HF and LF entities are important for aquatic species recognition, as both HF and LF only attain relatively good performance. From Table 4, we find that LF alone achieves better performance than that of only using raw images. One reason for this phenomenon may be the LF entity has less data noise, which enhances the noise-robustness of the DNN by neglecting HF components (). The results also demonstrate despite the noise-robustness in LF leads to quite high performance, the details information conveyed in the HF entity is critical for aquatic species recognition. Furthermore, we compare the strategy of averaging HF components with strategies such as maximum, addition, and concatenation. As illustrated, the model with an averaging connection for HF components used in WFN achieves better performance.
4.6 Sensitivity analysis of dataset partition
As shown in Table 5, we examine the impact of varying number of labeled and unlabeled data. We set the ratios of labeled data in the training set to 10%, 20%, 30%, and 100%, thereby determining the corresponding ratios of arbitrary unlabeled data. With the entire training dataset labeled (100% labeled data) in supervised learning, the WFN achieves an average classification accuracy of 49.41% across all aquatic species. Furthermore, the overall average classification accuracy of SSL increases by 8.52%, 7.31%, and 10.06% compared to supervised methods when using 10%, 20%, and 30% labeled data and the remaining unlabeled data. Moreover, our method exhibits improved performance with increasing amounts of unlabeled data. Training with 20% labeled data and 40%, 60%, and 80% unlabeled data result in overall average classification accuracy improvements of 5.97%, 6.82%, and 7.31% over the baseline. The empirical results confirm the proficiency of our method in generating pseudo-labels using arbitrary quantities of labeled data. Additionally, the robustness of the proposed method under diverse conditions has been comprehensively validated.
Table 5
| Labeled | Unlabeled | Common | Medium | Rare | All |
|---|---|---|---|---|---|
| 10% | 0% 90% | 66.41 59.78 | 47.35 59.72 | 14.97 23.23 | 19.28 27.80 |
| 20% | 0% 40% 60% 80% | 72.73 69.27 67.10 69.58 | 58.60 67.48 68.23 68.36 | 25.47 31.30 32.11 32.61 | 29.80 35.77 36.62 37.11 |
| 30% | 0% 70% | 75.93 73.05 | 62.35 71.42 | 28.05 38.43 | 32.53 42.59 |
| 100% | 0% | 83.31 | 74.49 | 45.68 | 49.41 |
Sensitivity analysis results of dataset partition strategies.
4.7 Replacing HF entity with edge information
Wavelet transform and edge detectors such as Canny and Sobel serve similar purposes in extracting detailed information within images. To further demonstrate the effectiveness of the HF entity, we replace the HF entity with the information generated by the edge detector. As shown in Table 6, we can see using HF entity outperforms the previous edge detection algorithm by a large margin. To be specific, WFN improves the optimal classification accuracy by over 2.46% in the Canny edge detector, and 1.96% in the Sobel edge detector, respectively. The performance degradation of both experiments illustrates the HF entity extracted by wavelet transform contains rich information about fine details and textures in the image. Besides, Canny and Sobel detectors can be sensitive to noise, especially in low-quality underwater images or those with uneven illumination and complex visual backgrounds, which might lead to false edge detection or noisy information. Through the experiments, we also conclude that WFN has a stronger ability for feature extraction than using raw images with edge information, which can be beneficial for heterogeneous image-collected environments in aquatic species recognition.
Table 6
| Raw | LF | HF | Canny | Sobel | Common | Medium | Rare | All |
|---|---|---|---|---|---|---|---|---|
| ✓ | 70.63 | 57.65 | 21.41 | 26.12 | ||||
| ✓ | ✓ | 67.92 | 54.55 | 19.56 | 24.11 | |||
| ✓ | ✓ | 69.66 | 56.01 | 20.27 | 24.93 | |||
| ✓ | ✓ | 70.33 | 56.34 | 22.97 | 27.34 | |||
| ✓ | ✓ | 70.95 | 56.05 | 23.58 | 27.84 | |||
| ✓ | ✓ | 72.73 | 58.60 | 25.47 | 29.80 |
Ablation on effectiveness of various information, including raw image, LF entity, HF entity, and information generated by edge detector.
4.8 Comparison of model size and computation cost
We showcase the performance of models trained on the labeled images along with model size and computational cost. Given that the proposed CEL function is designed for pseudo-labels, its computational complexity is negligible compared to that of fully-supervised training methods. As shown in Table 7, WFN requires two encoders for various frequency awareness, which significantly increased the computation cost as the acquired information increased. Furthermore, to illustrate that the performance enhancement stems from well-designed components, we expand ResNet-50 to match the number of parameters and computational costs of WFN. Our results indicate that while increased computational complexity yields positive effects, it still falls shorts of matching the performance of WFN.
Table 7
| Method | Params (M) | Flops (G) | Common | Medium | Rare | All |
|---|---|---|---|---|---|---|
| ResNet-50 | 24.46 | 4.14 | 70.63 | 57.65 | 21.41 | 26.12 |
| ResNet-50∗ | 59.09 | 11.63 | 72.70 | 57.35 | 23.09 | 27.58 |
| WFN | 64.11 | 9.11 | 72.73 | 58.60 | 25.47 | 29.80 |
Comparison of model sizes and computational cost.
∗ indicates increasing the number of convolutions and channels.
5 Discussion
A previous study () demonstrates that recognition task typically requiring four ecologists approximately 3 to 6 weeks for manual analysis can be completed in just 24 hours using DL methods. Their research also concludes that this accelerated approach does not compromise accuracy, as abundance estimates obtained through DL were within 1% of those derived from manual analysis by experts. Computer analysis has the potential to substantially streamline the investigative analysis process. Our study concurs with this point but also underscores the significant challenges in data labeling, as evidenced by previous representative studies (; ). This paper introduces a novel technique based on a SSL scheme, where DNN are learned using a limited number of labeled data and extensive unlabeled data, thereby alleviating the burden of manually labeling large dataset for researchers. This is enabled by two simple-to-implement but crucial modifications (1) using a robust feature extraction method, (2) replacing original consistency loss with CEL function. These modifications enable the DL method, trained on a limited amount of labeled data, to effectively address a diverse aquatic environment, as well as the long-tailed distribution of aquatic species.
Aquatic species recognition based on DL serves as a foundation for specific application, particularly biomass estimation and species habitat monitoring (). This information is crucial for informed decision-making in conservation management, including the establishment of protected areas, restoration effort, and mitigation of anthropogenic impacts. Furthermore, our research presents promising applications for long-tailed distribution of aquatic species in the natural world, which can significantly contribute to marine biodiversity conservation efforts. Species distribution and abundance follow a highly skewed rule, with a small number of species exhibiting high abundant, while numerous species are present in relatively low numbers (; ). The complex image collection environment poses challenges for commonly used methods such as data augmentation or data generation to be effective, particularly when labeled data is limited. As recommendations for improvement concerning existing conservation measures, we propose integrating our method into established monitoring frameworks. We have conducted both quantitative and qualitative experiments demonstrating the utility of the our method across a variety of diverse aquatic environments using large-scale species recognition datasets. The results of the above experiments instill confidence in our ability to collaborate with existing conservation monitoring programs.
While this work represents progress in developing a robust and effective SSL scheme for real-world aquatic species recognition applications, it has also revealed some limitations that future research should address. Firstly, the CEL enhances the performance of tail-class at the expense of lower performance for head-class. Given the importance of all aquatic species in real environments, it is worthwhile to explore strategies for significantly improving the performance of tail species while maintaining or even enhancing the performance of head species. Secondly, while our study has confirmed the effectiveness of WFN combined with single-level 2D discrete wavelet transform for aquatic species recognition, it is worth developing a DNN equipped with multilevel wavelet packet transform in future research because it could benefit from the hierarchical representation. Lastly, it would be interesting to apply our algorithm to more practical task, such as aquatic species detection, behavior analysis, and trait prediction. By deploying these application in real-world aquatic environments, we can develop increasingly intelligent solutions to address some of the most pressing issues of our time.
6 Conclusion
In this work, we have introduced a robust feature extractor, WFN, and a novel loss function, CEL, based on the SSL scheme FixMatch, for aquatic species recognition. Our proposed methods have demonstrated effectiveness in addressing the challenges of high-quality recognition in complex image-collected environments and the long-tailed class imbalanced nature of aquatic species, even with a limited number of labeled data. This is achieved through dedicated components, using the output of wavelet transform of one to train the DNN, and applying the CEL function at the stage where pseudo-labels come into play. The proposed method has consistently shown performance gains in both quantitative and qualitative experiments. We thus believe that our study can serve as a valuable resource for future research efforts in aquatic species recognition.
Statements
Data availability statement
The original contributions presented in the study are included in the article/supplementary material. Further inquiries can be directed to the corresponding authors.
Author contributions
DM: Writing – original draft. JW: Visualization, Writing – original draft. LZ: Writing – review & editing. FZ: Writing – review & editing, Supervision. HW: Writing – original draft. XC: Methodology, Writing – review & editing. YL: Funding acquisition, Writing – original draft. ML: Conceptualization, Writing – review & editing.
Funding
The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This work was supported by the International Research Center of Big Data for Sustainable Development Goals (No. CBAS2022GSP07), and the National Natural Science Foundation of China (No. 42230505, 42206148).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1
BellK. L. C.QuinzinM. C.AmonD.PoultonS.HopeA.SartiO.et al. (2023). Exposing inequities in deep-sea exploration and research: results of the 2022 global deep-sea capacity assessment. Front. Mar. Sci. 10, 1217227. doi: 10.3389/fmars.2023.1217227
2
BerthelotD.CarliniN.CubukE. D.KurakinA.SohnK.ZhangH.et al. (2019a). Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785.
3
BerthelotD.CarliniN.GoodfellowI.PapernotN.OliverA.RaffelC. A. (2019b). “Mixmatch: A holistic approach to semi-supervised learning,” in Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS '19) (NY, USA), 454, 5049–5059. doi: 10.5555/3454287.3454741
4
CaiL.McGuireN. E.HanlonR.MooneyT. A.GirdharY. (2023). Semi-supervised visual tracking of marine animals using autonomous underwater vehicles. Int. J. Comput. Vision131, 1406–1427. doi: 10.1007/s11263-023-01762-5
5
CaoK.WeiC.GaidonA.ArechigaN.MaT. (2019). “Learning imbalanced datasets with label-distribution-aware margin loss,“ in Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS'19) (NY, USA). 140, 1567–1578. doi: 10.5555/3454287.3454427
6
ChenZ.DuM.YangX.-D.ChenW.LiY.-S.QianC.et al. (2023). Deep-learning-based automated tracking and counting of living plankton in natural aquatic environments. Environ. Sci. Technol. 57 (46), 18048–18057. doi: 10.1021/acs.est.3c00253
7
ChoiC.KampffmeyerM.HandegardN. O.SalbergA.-B.BrautasetO.EikvilL.et al. (2021). Semi-supervised target classification in multi-frequency echosounder data. ICES J. Mar. Sci.78, 2615–2627. doi: 10.1093/icesjms/fsab140
8
CuiY.JiaM.LinT.-Y.SongY.BelongieS. (2019). “Class-balanced loss based on effective number of samples,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (Long Beach, CA, USA), 9268–9277. doi: 10.1109/CVPR.2019.00949
9
DengJ.DongW.SocherR.LiL.-J.LiK.Fei-FeiL. (2009). “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (Miami, FL, USA), 248–255. doi: 10.1109/CVPR.2009.5206848
10
de Souza BritoA.VieiraM. B.De AndradeM. L. S. C.FeitosaR. Q.GiraldiG. A. (2021). Combining max-pooling and wavelet pooling strategies for semantic image segmentation. Expert Syst. Appl.183, 115403. doi: 10.1016/j.eswa.2021.115403
11
DitriaE. M.Lopez-MarcanoS.SieversM.JinksE. L.BrownC. J.ConnollyR. M. (2020). Automating the analysis of fish abundance using object detection: optimizing animal ecology with deep learning. Front. Mar. Sci.7, 429. doi: 10.3389/fmars.2020.00429
12
DuanY.LiuF.JiaoL.ZhaoP.ZhangL. (2017). Sar image segmentation based on convolutional-wavelet neural network and markov random field. Pattern Recognition64, 255–267. doi: 10.1016/j.patcog.2016.11.015
13
FengC.ZhongY.HuangW. (2021). “Exploring classification equilibrium in long-tailed object detection,” in Proceedings of the IEEE/CVF International conference on computer vision (ICCV) (Montreal, QC, Canada), 3417–3426. doi: 10.1109/ICCV48922.2021.00340
14
HeK.ZhangX.RenS.SunJ. (2016). “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) (Las Vegas, NV, USA), 770–778. doi: 10.1109/CVPR.2016.90
15
HuangH.HeR.SunZ.TanT. (2017). “Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution,” in Proceedings of the IEEE international conference on computer vision (ICCV) (Venice, Italy), 1689–1697. doi: 10.1109/ICCV.2017.187
16
IrfanS.AlatawiA. M. M. (2019). Aquatic ecosystem and biodiversity: a review. Open J. Ecol.9, 1–13. doi: 10.4236/oje.2019.91001
17
JahanbakhtM.AzghadiM. R.WalthamN. J. (2023). Semi-supervised and weakly-supervised deep neural networks and dataset for fish detection in turbid underwater videos. Ecol. Inf.78, 102303. doi: 10.1016/j.ecoinf.2023.102303
18
KatijaK.OrensteinE.SchliningB.LundstenL.BarnardK.SainzG.et al. (2022). Fathomnet: A global image database for enabling artificial intelligence in the ocean. Sci. Rep.12, 15914. doi: 10.1038/s41598-022-19939-2
19
KaurM.VijayS. (2023). Deep learning with invariant feature based species classification in underwater environments. Multimedia Tools Appl., 1–22. doi: 10.1007/s11042-023-15896-8
20
KhanF. F.LiX.TempleA. J.ElhoseinyM. (2023). “Fishnet: A large-scale dataset and benchmark for fish recognition, detection, and functional trait prediction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (Paris, France), 20496–20506. doi: 10.1109/ICCV51070.2023.01874
21
KimJ.HurY.ParkS.YangE.HwangS. J.ShinJ. (2020). Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. Adv. Neural Inf. Process. Syst.33, 14567–14579.
22
LaiZ.WangC.GunawanH.CheungS.-C. S.ChuahC.-N. (2022). “Smoothed adaptive weighting for imbalanced semi-supervised learning: Improve reliability against unknown distribution data,” in International Conference on Machine Learning (PMLR). 11828–11843.
23
LaradjiI. H.SalehA.RodriguezP.NowrouzezahraiD.AzghadiM. R.VazquezD. (2021). Weakly supervised underwater fish segmentation using affinity lcfcn. Sci. Rep.11, 17379. doi: 10.1038/s41598-021-96610-2
24
LeCunY.BengioY.HintonG. (2015). Deep learning. nature521, 436–444. doi: 10.1038/nature14539
25
LeeH.ShinS.KimH. (2021). Abc: Auxiliary balanced classifier for class-imbalanced semisupervised learning. Adv. Neural Inf. Process. Syst.34, 7082–7094.
26
LiQ.ShenL.GuoS.LaiZ. (2020). “Wavelet integrated cnns for noise-robust image classification,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (Seattle, WA, USA), 7245–7254. doi: 10.1109/CVPR42600.2020.00727
27
LiQ.ShenL.GuoS.LaiZ. (2021). Wavecnet: Wavelet integrated cnns to suppress aliasing effect for noise-robust image classification. IEEE Trans. Image Process.30, 7074–7089. doi: 10.1109/TIP.2021.3101395
28
LiJ.XuW.DengL.XiaoY.HanZ.ZhengH. (2023). Deep learning for visual recognition and detection of aquatic animals: A review. Rev. Aquaculture15, 409–433. doi: 10.1111/raq.12726
29
LiuL.MengL.PengY.WangX. (2021). A data hiding scheme based on u-net and wavelet transform. Knowledge-Based Syst.223, 107022. doi: 10.1016/j.knosys.2021.107022
30
LiuL.WuJ.ZhengT.ZhaoH.KongH.QuB.et al. (2023). Fish recognition in the underwater environment using an improved arcface loss for precision aquaculture. Fishes8, 591. doi: 10.3390/fishes8120591
31
LiuP.ZhangH.ZhangK.LinL.ZuoW. (2018). “Multi-level wavelet-cnn for image restoration,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops (CVPRW) (Salt Lake City, UT, USA), 773–782. doi: 10.1109/CVPRW.2018.00121
32
LuH.LiY.UemuraT.GeZ.XuX.HeL.et al. (2018). Fdcnet: filtering deep convolutional network for marine organism classification. Multimedia Tools Appl.77, 21847–21860. doi: 10.1007/s11042-017-4585-1
33
MaD.WeiJ.LiY.ZhaoF.ChenX.HuY.et al. (2023). Mldet: Towards efficient and accurate deep learning method for marine litter detection. Ocean Coast. Manage.243, 106765. doi: 10.1016/j.ocecoaman.2023.106765
34
MallatS. G. (1989). A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans. Pattern Anal. Mach. Intell.11, 674–693. doi: 10.1109/34.192463
35
MenonA. K.JayasumanaS.RawatA. S.JainH.VeitA.KumarS. (2020). Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314.
36
MiyatoT.MaedaS.-i.KoyamaM.IshiiS. (2018). Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell.41, 1979–1993. doi: 10.1109/TPAMI.34
37
MollerT.NilssenI.NattkemperT. W. (2017). “Active learning for the classification of species in underwater images from a fixed observatory,” in Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW) (Venice, Italy), 2891–2897. doi: 10.1109/ICCVW.2017.341
38
OhY.KimD.-J.KweonI. S. (2022). “Daso: Distribution-aware semantics-oriented pseudo-label for imbalanced semi-supervised learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (New Orleans, LA, USA), 9786–9796. doi: 10.1109/CVPR52688.2022.00956
39
PaszkeA.GrossS.MassaF.LererA.BradburyJ.ChananG.et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst.32.
40
QiuC.ZhangS.WangC.YuZ.ZhengH.ZhengB. (2018). Improving transfer learning and squeeze-and-excitation networks for small-scale fine-grained fish image classification. IEEE Access6, 78503–78512. doi: 10.1109/Access.6287639
41
RamamonjisoaM.FirmanM.WatsonJ.LepetitV.TurmukhambetovD. (2021). “Single image depth prediction with wavelet decomposition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (Nashville, TN, USA), 11089–11098. doi: 10.1109/CVPR46437.2021.01094
42
RenJ.YuC.MaX.ZhaoH.YiS.et al. (2020). “Balanced meta-softmax for long-tailed visual recognition.“ in Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS '20) (NY, USA), 4175–4186. doi: 10.5555/3495724.3496075
43
RubbensP.BrodieS.CordierT.Destro BarcellosD.DevosP.Fernandes-SalvadorJ. A.et al. (2023). Machine learning in marine ecology: an overview of techniques and applications. ICES J. Mar. Sci.80, 1829–1853. doi: 10.1093/icesjms/fsad100
44
SalaE.MayorgaJ.BradleyD.CabralR. B.AtwoodT. B.AuberA.et al. (2021). Protecting the global ocean for biodiversity, food and climate. Nature592, 397–402. doi: 10.1038/s41586-021-03371-z
45
SalehA.LaradjiI. H.KonovalovD. A.BradleyM.VazquezD.SheavesM. (2020). A realistic fish-habitat dataset to evaluate algorithms for underwater visual analysis. Sci. Rep.10, 14671. doi: 10.1038/s41598-020-71639-x
46
SalehA.SheavesM.JerryD.AzghadiM. R. (2023). Applications of deep learning in fish habitat monitoring: A tutorial and survey. Expert Syst. Appl., 121841.
47
SalehA.SheavesM.Rahimi AzghadiM. (2022). Computer vision and deep learning for fish classification in underwater habitats: A survey. Fish Fisheries23, 977–999. doi: 10.1111/faf.12666
48
SohnK.BerthelotD.CarliniN.ZhangZ.ZhangH.RaffelC. A.et al. (2020). Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Adv. Neural Inf. Process. Syst.33, 596–608.
49
TanJ.WangC.LiB.LiQ.OuyangW.YinC.et al. (2020). “Equalization loss for long-tailed object recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (Seattle, WA, USA), 11662–11671. doi: 10.1109/CVPR42600.2020.01168
50
TorneyC. J.Lloyd-JonesD. J.ChevallierM.MoyerD. C.MalitiH. T.MwitaM.et al. (2019). A comparison of deep learning and citizen science techniques for counting wildlife in aerial survey images. Methods Ecol. Evol.10, 779–787. doi: 10.1111/2041-210X.13165
51
VaswaniA.ShazeerN.ParmarN.UszkoreitJ.JonesL.GomezA. N.et al. (2017). “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17) (NY, USA), 6000–6010. doi: 10.5555/3295222.3295349
52
VillonS.IovanC.MangeasM.VigliolaL. (2022). Confronting deep-learning and biodiversity challenges for automatic video-monitoring of marine ecosystems. Sensors22, 497. doi: 10.3390/s22020497
53
VisbeckM. (2018). Ocean science research is key for a sustainable future. Nat. Commun.9, 690. doi: 10.1038/s41467-018-03158-3
54
WeiC.SohnK.MellinaC.YuilleA.YangF. (2021). “Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (Nashville, TN, USA), 10857–10866. doi: 10.1109/CVPR46437.2021.01071
55
XieQ.DaiZ.HovyE.LuongT.LeQ. (2020a). “Unsupervised data augmentation for consistency training,” in Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS '20) (NY, USA), 6256–6268. doi: 10.5555/3495724.3496249
56
XieQ.LuongM.-T.HovyE.LeQ. V. (2020b). “Self-training with noisy student improves imagenet classification,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (Seattle, WA, USA), 10687–10698. doi: 10.1109/CVPR42600.2020
57
YangX.SongZ.KingI.XuZ. (2022). A survey on deep semi-supervised learning. IEEE Trans. Knowledge Data Eng. 35 (9), 8934–8954. doi: 10.1109/TKDE.2022.3220219
58
YaoT.PanY.LiY.NgoC.-W.MeiT. (2022). “Wave-vit: Unifying wavelet and transformers for visual representation learning,” in European Conference on Computer Vision (ECCV) (Berlin, Heidelberg), 328–345. doi: 10.1007/978-3-031-19806-9_19
59
YinX.XuX. (2021). “A method for improving accuracy of deeplabv3+ semantic segmentation model based on wavelet transform,” in International Conference in Communications, Signal Processing, and Systems (Singapore), 315–320. doi: 10.1007/978-981-19-0390-8_85
60
ZhangY.KangB.HooiB.YanS.FengJ. (2023). Deep long-tailed learning: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 45, 10795–10816. doi: 10.1109/TPAMI.2023.3268118
61
ZhaoY.WangS.ZhangY.QiaoS.ZhangM. (2023). Wranet: wavelet integrated residual attention u-net network for medical image segmentation. Complex intelligent Syst.9, 6971–6983. doi: 10.1007/s40747-023-01119-y
62
ZhouY.HuangJ.WangC.SongL.YangG. (2023). “Xnet: Wavelet-based low and high frequency fusion networks for fully-and semi-supervised semantic segmentation of biomedical images,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (Paris, France), 21085–21096. doi: 10.1109/ICCV51070.2023.01928
63
ZhuangP.WangY.QiaoY. (2020). Wildfish++: A comprehensive fish benchmark for multimedia research. IEEE Trans. Multimedia23, 3603–3617. doi: 10.1109/TMM.2020.3028482
Summary
Keywords
deep learning, semi-supervised learning, aquatic species recognition, wavelet transform, consistency loss
Citation
Ma D, Wei J, Zhu L, Zhao F, Wu H, Chen X, Li Y and Liu M (2024) Semi-supervised learning advances species recognition for aquatic biodiversity monitoring. Front. Mar. Sci. 11:1373755. doi: 10.3389/fmars.2024.1373755
Received
20 January 2024
Accepted
02 May 2024
Published
28 May 2024
Volume
11 - 2024
Edited by
Haiyong Zheng, Ocean University of China, China
Reviewed by
Zhuhua Hu, Hainan University, China
Deepayan Bhowmik, Newcastle University, United Kingdom
Updates
Copyright
© 2024 Ma, Wei, Zhu, Zhao, Wu, Chen, Li and Liu.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Ye Li, yli@geo.ecnu.edu.cn; Min Liu, mliu@geo.ecnu.edu.cn
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.