Natural image restoration based on multi-scale group sparsity residual constraints

Abstract

The Group Sparse Representation (GSR) model shows excellent potential in various image restoration tasks. In this study, we propose a novel Multi-Scale Group Sparse Residual Constraint Model (MS-GSRC) which can be applied to various inverse problems, including denoising, inpainting, and compressed sensing (CS). Our new method involves the following three steps: (1) finding similar patches with an overlapping scheme for the input degraded image using a multi-scale strategy, (2) performing a group sparse coding on these patches with low-rank constraints to get an initial representation vector, and (3) under the Bayesian maximum a posteriori (MAP) restoration framework, we adopt an alternating minimization scheme to solve the corresponding equation and reconstruct the target image finally. Simulation experiments demonstrate that our proposed model outperforms in terms of both objective image quality and subjective visual quality compared to several state-of-the-art methods.

1. Introduction

Unsuitable equipment and other disturbances unavoidably contribute noise in the target images. Image denoising is a crucial area of image processing and has attracted much attention from scholars in related fields recently. Digital image denoising techniques have a wide range of uses, involving disciplines of medicine and industry, and also in spectral images for weather forecasting, remote sensing images, and so on. Taking image denoising as a basis, the method can be introduced to more image restoration problems and be useful in more fields (Buades et al., 2005; Osher et al., 2005; Elad and Aharon, 2006; Zoran and Weiss, 2011; Gu et al., 2014; Zhang et al., 2014b; Liu et al., 2017; Keshavarzian et al., 2019; Ou et al., 2020; Zha et al., 2020a, 2022; Jon et al., 2021). This task aims to generate a latent image x from the degraded version y. The process modeling can be depicted as

Where H is an irreversible linear operator in matrix form and n is the additive white Gaussian noise vector. By requiring H, Eq.(1) can be converted to diverse image restoration problems. For example, Eq.(1) represents the image denoising problem if H is an identity (Elad and Aharon, 2006; Ou et al., 2020); Eq.(1) denotes the image inpainting problem if H is a mask (Liu et al., 2017; Zha et al., 2020a); and Eq.(1) stands for the image CS problem if H is an undersampled random projection matrix (Keshavarzian et al., 2019; Zha et al., 2022). We concentrate on image denoising, inpainting, and CS challenges in this article.

Given that the problem always ill-posed, it is common to use image priors to regularize the model so as to gain excellent restored images. Namely, the Maximum A Posteriori (MAP) approach allows for the image restoration problem to be formulated as a mathematical equation to address the minimization problem:

The former is the data-fidelity term and the latter is the image prior constraint term. The weights between these two terms are regulated by the parameter λ. After establishing the mathematical model, we conceived an optimization algorithm to address various image restoration problems. The method yields a reconstructed image that approximates a clean image after several iterations.

Numerous image prior models have been put forward in earlier studies, mainly classified into local smoothness (Rudin et al., 1992; Osher et al., 2005; Dey et al., 2006), non-local self-similarity (Fazel et al., 2001; Buades et al., 2005; Gu et al., 2014), and sparsity (Zhang et al., 2014b; Ou et al., 2020, 2022a). Yet, the curse of dimensionality makes it difficult to construct a global model for the entire image. Therefore, the approach of building patch priors has become popular in recent years for its efficiency and convenience.

Sparse representation is one of the most representative patch-based priors. Elad and Aharon (2006) proposed K-SVD (K-Singular Value Decomposition) which is a pioneering work in applying sparse coding to image denoising. NSS is another crucial prior information widely used. Buades et al. (2005) proposed the first model using NSS for image denoising. In addition, the high correlation between patches leading to the data matrix of a clean image is as often low-rank. Related studies mainly fall into two categories: low-rank matrix factorization (LRMF) (Srebro and Jaakkola, 2003; Buades et al., 2005) and the Nuclear Norm Minimization (NNM) (Fazel et al., 2001; Hu et al., 2012). NNM is the more popular one in most cases. Gu et al. presented the Weighted Nuclear Norm Minimisation model (WNNM) (Gu et al., 2014) which dramatically enhances the flexibility of NNM, and it remains among most widespread image denoising methods. Apart from this, RRC (Zha et al., 2019), which makes use of low-rank residuals for modeling, has also achieved good quality in various image restoration problems.

Some studies have combined image sparsity and self-similarity to modeling, and these algorithms have shown great potential in image restoration research. For instance, in the study by Dabov et al. (2007), BM3D applies NSS to cluster patches before collaborative filtering, which is a benchmark method in the current area of image denoising. Both NCSR (Dong et al., 2012b) and GSR (Zhang et al., 2014b) use the NSS property to aggregate image patches into groups, and then perform sparse coding on the self-similar groups. Mairal J et al. devised the LSSC (Mairal et al., 2009) to force all self-similar groups to be imposed with the same dictionary. Zha et al. (2017) designed an efficient GSRC model that converts the task of image denoising into one of minimizing group sparse residuals. In addition, Zha et al. (2020a) also proposed a GSRC-NLP model with a better image restoration result based on the above.

Another groundbreaking patch-based image recovery method is Expected Patch Log Likelihood (EPLL) (Zoran and Weiss, 2011) which restores images by learning a Gaussian mixture model(GMM). Later on, Zoran et al. introduces a multi-scale EPLL (Papyan and Elad, 2015) model, which can improve the performance of image restoration further. Subsequently, image denoising methods using external GMM priors have been widely used. Most of the relevant studies have combined external GMM with internal NSS for modeling, such as Xu et al. (2015) proposed PGPD, Chen et al. (2015) proposed PCLR, and Zha et al. (2020b) proposed SNSS.

In addition to the above methods, deep convolutional neural networks (CNNs) (Zhang et al., 2017; Zhang and Ghanem, 2018) is an emerging approach in recent years, but it requires learning in an external database before restoring damaged images.

It is not comprehensive to only consider the sparsity or low-rankness property of the image. Hence, with the aim of obtaining a higher-quality restored image, our study uses the low-rank property of similar groups as a constraint in combination with sparsity to design the model. Furthermore, based on the NSS property, we can not only find similar patches for a specified patch on a single scale image but also extend the search window to multi-scales. Finally, we propose a novel Multi-scale Group Sparsity Residual Constraint (MS-GSRC) model with the following innovations:

• We propose a novel MS-GSRC model that provides a simple yet effective approach for image restoration: find neighbor patches with an overlapping scheme for the input degraded image using a multi-scale strategy and perform a group sparse coding on these similar patches with a low-rank constraint.

• An alternating minimization mechanism with an automatically tuned parameter scheme is applied to our proposed model, which guarantees a closed-form solution at each step.

• Our proposed MS-GSRC model is validated on three tasks: denoising, inpainting, and compressed sensing. The model performs competitive in both objective image quality and subjective visual quality compared to several state-of-the-art image restoration methods.

The remainder of this article is as follows: In Section 2, after the brief overview of the GSRC framework and LR methods, we introduce a novel MS-GSRC model. Section 3 adopts an alternating minimization scheme with self-adjustable parameters to resolve our proposed algorithm. Section 4 lists extensive experimental results that prove the feasibility of our model. Conclusion is presented in Section 5.

2. Models

In this part, we briefly review some relevant knowledge and present our new model.

2.1. Group-based sparse representation

Principles of the GSR model can be described as follows: divide the image into many overlapping patches, find self-similarity groups for each image patch using the NSS property, perform sparse coding for each self-similarity group, and finally reconstruct the image (Dong et al., 2012b; Zha et al., 2020a; Ou et al., 2022a).

Specifically, the image x ∈ ℝ^M is divided into m overlapping patches , where . Next, for each overlapping patch x_i, we use the K-Nearest Neighbor classification (KNN) algorithm (Keller et al., 1985; Xie et al., 2016) to select k neighbor patches from a W×W search window to form the group K_i. Subsequently, stack all K_i into a data matrix ; this matrix contains each element of K_i as its column, i.e., X_i = {x_{i, 1}, x_{i, 2}…x_{i, k}}, where denotes the k-th similar patch of the k-th group. Each similarity group X_i is represented sparsely as , where D_i denotes the dictionary.

Nevertheless, solving the 0-norm minimization problem is NP-hard, so for the ease of making the solution, The sparse code is obtained from the following equation (Zhang et al., 2014b):

It is well-known that clean images x are unavailable in image restoration problems. Thus, we replace x with degenerate images y ∈ ℝ^{M × M}. Eq.(3) can be transformed into the problem of recovering the group sparse code A_i from Y_i:

The restored X_i is obtained by , and the final complete image X can be gained by simple averaging .

2.2. Group sparsity residual constraint

After observing the GSR model, it is clear that the closer the computed A approximates to B, the better the quality of the final restoration image. Consequently, the following definition of the group sparsity residual constraint (GSRC) (Zha et al., 2017) is given: R = A − B. Then, Eq.(4) for solving the group sparse coefficients A_i can be converted into:

This model uses BM3D to restore the the degenerate observation y to the image z. Moreover, z can be viewed as a good approximation of the target x considering BM3D has an excellent denoising performance. Thus, the group sparsity coefficients B_i can be obtained from z. In the study by Zha et al. (2020a), GSRC-NLP uses NLP before constraining the input image.

2.3. Low-rank approximation

According to Gu et al. (2014), Zha et al. (2019), and Zha et al. (2020b), it can be found that NNM is a popular low-rank approximations methods. For X, define the i-th singular value as σ_i(x), and the nuclear norm as ‖X‖_* = Σ_iσ_i(x). The specific solution for X is:

Equation (6) yields a simple solution: , where Ŷ = UΣV^T is the SVD for Y and S_τ(Σ) is a soft-thresholding (Cai et al., 2010) function. Namely, S_τ(Σ)_ii = max(Σ_ii − τ, 0), where Σ_ii is the diagonal element of Σ.

2.4. Multi-scale GSRC

The established GSRC model has performed well in image denoising, but it requires additional pre-processing of degraded images for obtaining the group sparsity coefficients B. Thus, we combine group sparsity and low-rank property to build a model. Furthermore, the GSRC model only focuses on a single scale. However, it is evident that NSS can appear not only on the original scale of an image but also on a coarse scale, so we can find neighbor patches for the original image patch at multi-scales (Yair and Michaeli, 2018; Ou et al., 2022a,b). The specific steps of our proposed new Multi-Scale Group Sparse Residual Constraint (MS-GSRC) model are as follows:

(a) First, we use KNN to find a specified number of similar patches from both the original scale and scaled-down version for the overlapping patches of the input image.

(b) Then, these similar patches are stacked separately into groups.

(c) Next, the low-rank constraint is imposed on each group to obtain good group sparsity coefficients B_i.

(d) After estimating the group sparsity coefficients A_i by using the group sparsity residuals R_i, each group was recovered in sequence.

(e) Finally, we select the patch belonging to the original image from each group, and aggregate the complete image by simple averaging.

We propose the following constraint function:

is a multi-scale similarity group, which is a matrix with k nearest neighbor patches matched for each original image patch. These similar patches are derived from both the original and coarse scales of the image. The window size is W × W in the original scale, and it isχW × χW in the other scale images, where χ indicates the scale factor (0 < χ < 1). χ will be set to different values in different experiments.

For image denoising, for example, the flowchart of MS-GSRC model is shown in Figure 1.

Figure 1

3. Algorithm for image restoration

This section is a detailed analysis of our proposed MS-GSRC model. The solution of this algorithm is obtained using an alternating minimization method whose parameter is self-adjustment.

First, we divide Eq.(7) into three sub-problems:

3.1. A_i sub-problem

Given x and D_i, we get a sub-problem of A_i:

where , α_i, β_i, p_i stand for the vector representations of A_i, B_i, and P_i, respectively. D_i is a dictionary, A crucial step for solving the A_i problem is to design an efficient D_i. The restored image is prone to visual artifacts (Lu et al., 2013) if learning the over-complete dictionary. To reduce this terrible phenomenon, we choose to adopt principal component analysis (PCA) (Abdi and Williams, 2010) for learning the dictionary D_i in this study because PCA is more robust and adjustable.

Equation (8) can be deduced as a closed-form solution:

Soft (·) represents the soft-thresholding operator.

Since x is an unknown target image, it is impossible to gain the group sparse coefficients B_i directly. Consequently, we must utilize methods to gain an approximation value.

The introduction of low-rank constraints into the model is a practical approach. After applying LR constraints to the Y_i group, we can obtain a matrix S_i. The clean group sparsity coefficients B_i can be computed from S_i. It is easy to derive the following equation:

So,we can obtain

where ‖B_i‖_* = Σ_jδ_{i, j}, are singular value of matrix. Apparently, we are able to get a closed-form solution for B_i:

where is the SVD for A_i and Δ_i is the diagonal element of the singular matrix.

3.2. x sub-problem

Given A_i and D_i, subproblem of x in Eq.(7) turns into:

Clearly, Eq.(13) is a quadratic optimization equation. We adopt Alternate Directional Multiplication Method (ADMM) (Boyd et al., 2011) to simplify the optimization process.

First, we bring in an auxiliary variable s = x^MS, and Eq.(13) can be converted into an equivalence equation:

By observing Eq.(14), it is plain that this equation has three unknown variables requiring solutions. Thus, we decompose Eq.(14) into three iterative processes. In the t-th iteration:

The parameter c indicates the Lagrangian multiplier. To make the derivation process look more concise, we omit t in the following formulation expression.

Update  s : Given D_iA_i, x, and c, s can be represented as a closed-form solution by Eq.(16), namely:

Since I is a matrix of identities and represents a diagonal matrix, is positive. So the above formula is valid.

Update  x : Given s and c, Eq.(15) provides a solution to the variable x:

Notably, since H is an unstructured random projection matrix, the cost required to solve x using Eq.(19) directly is too high in CS problem. Hence, after setting step size γ and gradient direction q, we employ the gradient descent method (Ruder, 2016): to rewrite Eq.(19) as:

In addition, it is recommended to compute H^TH and H^Ty in advance to further enhance the algorithm efficiency.

3.3. Parameter settings

In the model we proposed above, there are four parameters (μ, λ, θ, ζ) requiring setting. Here, we set a strategy for the parameters that can be automatically adjusted in each iteration, which allows us to achieve more robust and accurate experimental results.

The noise standard deviation σ_n is automatically updated in each iteration (Osher et al., 2005):

Where ω represents a scaling factor, it is evident from Gu et al. (2014) and Chen et al. (2015) that this approach to regularize σ_e has been implemented in diverse models and has exhibited positive performance.

After setting σ_e, the value of μ is tuned to change in proportion to (Zha et al., 2022):

where ρ denotes a constant.

Moreover, the regularization parameters λ and θ represent the constraint penalties on sparsity and LR, respectively. Inspired by Dong et al. (2012a), they are adjusted in each iteration as follows:

where m_i is the estimated standard variance of R_i and n_i stands for the estimated standard variance of Δ_i. The ε and ϵ are two small constants to avoid zero divisors. α and β are set to two constants. Finally, parameter ζ is also set to a fixed constant.

The detailed procedure of the MS-GSRC algorithm is presented in Algorithm 1.

4. Experiences

In this chapter, extensive trial are conducted on image denoising, inpainting, and CS to verify that our proposed MS-GSRC model possesses better image restoration capabilities compared to some classical methods. To obtain intuitive comparison results, we set on two metrics: peak signal-to-noise ratio (PSNR) and structural self-similarity (SSIM) (Wang et al., 2004).

PSNR is commonly used to measure signal distortion. This parameter is calculated based on the gray scale values of the image pixels. Although sometimes the value of PSNR is not consistent with competent human perception, it remains an important reference evaluation metric. SSIM is a metric intended for assessing similarity between two images, which is an intuitive human standard for evaluating image quality.

If the degraded image is in color, we mainly recover the luminance channel due to the fact that variations in the luminance of color images are more easily perceived by the human eye.

The codes for all comparison algorithms used in this study are obtained from the original author's homepage and uses the given default parameters directly. For reasons of limited space, only a few images frequently used for testing are detailed list in Figure 2. In all tables, the data marked in red represent the best values.

Figure 2

4.1. Image denoising

First, we verify the performance of our MS-GSRC model on the image denoising task. The corresponding parameters are set as follows. We set the search window W × W to 30 × 30, the patch size to 6 × 6, 7 × 7, 9 × 9 for σ ≤ 15, 15 < σ ≤ 30, and 30 < σ ≤ 75, with the number of neighbor patches k to 70, 110, 120 for σ ≤ 30, 30 < σ ≤ 50, 50 < σ ≤ 75, respectively. The parameters (α, β, ω, ζ) are set to (0.03, 1.75, 0.81, 0.085), (0.015, 1.8, 0.86, 0.07), (0.05, 2.2, 0.81, 0.12), (0.006, 2, 0.86, 0.05) for σ ≤ 15, 15 < σ ≤ 30, 30 < σ ≤ 50, 50 < σ ≤ 75. In addition, we set the multi-scale to [1,0.8], [1,0.85], and [1,0.9] for σ ≤ 15, 15 < σ ≤ 50, and 50 < σ ≤ 75, separately.

Our MS-GSRC method is compared with several recently proposed popular denoising methods and classical traditional denoising methods, including BM3D (Dabov et al., 2007), PGPD (Xu et al., 2015), WNNM (Gu et al., 2014), NCSR (Dong et al., 2012b), RRC (Zha et al., 2019), LGSR (Zha et al., 2022) and GSRC-NLP (Zha et al., 2020a). Of all the comparison methods, BM3D is a frequently adopted benchmarking method, NCSR, PGPD, and GSRC-NLP all use GSR as a prior, and WNNM and RRC exploit low-rankness knowledge. And LGSR combines GSR and LR. Besides, both GSRC-NLP and our proposed model use the GSRC framework. Taking 12 frequently used images as an example, Table 1 lists the PSNR and SSIM results for various denoising methods at different noise levels. It is observed that our proposed MS-GSRC method produced superior performance. Specifically, the average PSNR and SSIM we achieve are improved by (0.47 dB, 0.0149) compared to BM3D, (0.38 dB, 0.0107) compared to PGPD, (0.07 dB, 0.0032) compared to WWNM, (0.42 dB, 0.0149) compared to NCSR, (0.25 dB, 0.0066) compared to RRC, (0.1 dB, 0.0005) compared to LGSR, and (0.19 dB, 0.0054) compared to GSRC-NLP.

We also utilize the BSD68 dataset (Wang et al., 2004) to assess the denoising ability of all compared approaches. We can observe from Table 2 that the average PSNR gains obtained by our proposed MS-GSRC method in comparison to the BM3D, PGPD, WNNM, NCSR, RRC, GSRC-NLP, and LGSR methods are 0.24 dB, 0.16 dB, 0.02 dB, 0.27 dB, 0.24 dB, 0.23 dB, and 0.03 dB. Meanwhile, on average, the proposed MS-GSRC achieve an SSIM improvement of 0.0153 on BM3D, 0.0169 on PGPD, 0.0064 on WNNM, 0.0137 on NCSR, 0.0228 on RRC, 0.0027 on LGSR, and 0.0194 on GSRC-NLP. Evidently, our proposed MS-GSRC method yields better PSNR and SSIM in almost all noise cases. Our method is only 0.01 dB lower than WWNM in PSNR, but 0.0052 higher than in SSIM at σ = 75. Beyond objective metrics, the subjective perception of the human body is also a crucial criterion for assessing the quality of an image. Consequently, we present the visual contrast between the two images of starfish and 223,061 restored by different methods in Figures 3, 4, respectively. Figure 3 indicates that BM3D, PGPD, WNNM, and RRC are likely to over-smooth the restored image, whereas NCSR, GSRC-NLP, and LGSR can lead to the appearance of some undesired visual artifacts. As can be seen in Figure 4, although the image restored by WNNM has a higher PSNR, the image restored by our MS-GSRC method has a higher SSIM value and presents a better visual effect. PGPD, NCSR, RRC, and GSRC-NLP are susceptible to loss of detail in the restored images, while BM3D, WNNM, and LGSR may result in undesirable artifacts.

Figure 3

Figure 4

4.2. Image inpainting

Next, we verify the superiority of the MS-GSRC model on inpainting. We likewise compare the proposed MS-GSRC method with many classical or recently popular methods, such as SAIST (Afonso et al., 2010), TSLRA (Guo et al., 2017), GSR (Zhang et al., 2014b), JSM (Zhang et al., 2014c), JPG-SR (Zha et al., 2018b), LGSR (Zha et al., 2022), and IDBP (Tirer and Giryes, 2018). Among these, SAIST is one of the earliest proposed methods for image restoration, GSR, JPG-SR, LGSR, TSLRA, and JSM use the NSS prior, and IDBP is a deep learning-based method. In simulation experiments, we test images by randomly generated masks that included missing pixels of 80%, 70%, 60%, and 50%. Following are the parameters that we set for the MS-GSRC model in different cases. We set the patch size to 7 × 7, the search window size to 25, and the non-local similar patches to 60. In addition, for all cases, we set the multi-scales to [1,0.85]. Moreover, we set (0.0002, 0.0001, 1.5, 15) and (0.0001, 0.0001, 1.5, 15) as parameters (ω, ζ, α, β) when the missing pixels are 0.8 and others, respectively. In addition, for all experiences.

Table 3 illustrates the PSNR and SSIM results for each method on the 12 frequently used test images. As observed in Table 3, our proposed method exceeds the comparison algorithm virtually often when it comes to image inpainting performance. The proposed MS-GSRC outperforms SAIST, JSM, GSR, TSLRA, JPG-SR, LGSR, and IDBP approaches in average PSNR performance, with gains of 5.8 dB, 1.43 dB, 0.51 dB, 1.82 dB, 0.51 dB, 0.13 dB, and 1.26 dB, respectively. Additionally, on average, the proposed MS-GSRC surpasses SAIST by 0.0776, JSM by 0.0216, GSR by 0.0025, TSLRA by 0.0238, JPG-SR by 0.007, LGSR by 0.0012, and IDBP by 0.022.

Similarly, two images are selected for detailed visual analysis. The image butterfly with a 80% loss of pixels restored by different methods are presented in Figure 5. Moreover, Figure 6 displays the outcomes of a visual comparison of image flowers with a 70% loss of pixels restored with different algorithms. By analyzing the visual comparison images, we can find that images restored using SAIST, JSM, TSLRA, IDBP GSR, and JPG-SR are susceptible to excessive smoothing, and images restored using LGSR tend to show excessive visual artifacts. The images restored using our proposed MS-GSRC model have significantly better restoration capabilities with regard to image detail and edges.

Figure 5

Figure 6

4.3. Image compressed sensing

Finally, we validate the restoration capability of our proposed MS-GSRC model on the image compressed sensing problem. In this part of experiments, we use the Gaussian random projection matrix (Zhang et al., 2014b) to generate blocks of size 32 × 32 to test the CS restoration effects. The parameters set for the MS-GSRC model are as follows: For all cases, the patch size is set to be 8 × 8, the patch number to 80, the search window size to be 25, and the multi-scales to be [1,0.75]. In addition, (0.004, 0.00002, 0.6, 25), (0.0014, 0.00005, 0.9, 15), (0.0015, 0.00001, 0.5, 10), and (0.0015, 0.00001, 1.4, 6) are set for (ζ,ω, α,β) when subrate is 0.1N, 0.2N, 0.3N, and 0.4N.

BSC (Mun and Fowler, 2009), RCOS (Zhang et al., 2012), ALSB (Zhang et al., 2014a), GSR (Zhang et al., 2014b), ASNR (Zha et al., 2018a), and LGSR (Zha et al., 2022) are choosen as competing methods. Among them, GSR performs a sparse representation on similar groups of images, ASNR is an image of the CS method that extends on the basis of NCSR, and LGSR combines sparsity and LR. Similarly, we selected 12 images frequently used in image restoration experiments as test images. Table 4 presents the average outcomes of PSNR and SSIM of the restored images using different method. To be concrete, the proposed MS-GSRC model over BCS, RCOS, ALSB, GSR, ASNR, and LGSR methods are 5.36 dB, 2.34 dB, 1.24 dB, 0.46 dB, 0.45 dB, and 0.14d B in PSNR and 0.1591, 0.0634, 0.0023, 0.0056, 0.0082, and 0.0029 in SSIM, respectively.

Due to the other competing algorithms used in this thesis, all use BCS to pre-process CS images, and here we use the BCS-processed images as corrupted images. Figure 7 shows the visual contrast of the image fence with 0.1 N CS measurements, and we can observe that RCOS and ALSB are less capable of restoring details, GSR and LGSR lead to over-smooth, and ASNR generates some redundant artifacts. Figure 8 illustrates the visual comparison of the image leaves measured with 0.1N CS. All comparison images have strong ringing phenomena and present terrible artifacts. In Figure 9, we have selected the image airplane processed with 0.2N CS for detailed analysis. It is obvious that the details of the images restored by ALSB and LGSR are seriously missing. The images restored by RCOS, GSR, and ASNR produced more artifacts. In the above three cases, our proposed MS-GSRC algorithm significantly outperforms other competing algorithms in recovering the image overall and some texture details.

Figure 7

Figure 8

Figure 9

5. Conclusion

In this study, we propose a novel model Multi-Scale Group Sparse Residual Constraint Model (MS-GSRC) for image restoration. This model introduces the low-rank property into the group sparse residual framework and finds similar patches for overlapping patches of the input image using a multi-scale strategy. Furthermore, under the MAP restoration framework, an alternating minimization method with adaptive tunable parameters is used to deliver a robust optimization solution for our MS-GSRC method. We employ the MS-GSRC model to three image restoration problems, namely, denoising, inpainting, and compressed sensing. Extensive simulation trials show that our novel model performs superior to many classical methods in terms of both objective image quality and subjective visual quality.

References

• 1

  Abdi H. Williams L. J. (2010). Principal component analysis. Wiley Interdiscip. Rev. Comput. Stat. 2, 433–459. 10.1002/wics.101

  • CrossRef
  • Google Scholar

• 2

  Afonso M. V. Bioucas-Dias J. M. Figueiredo M. A. (2010). An augmented lagrangian approach to the constrained optimization formulation of imaging inverse problems. IEEE Trans. Image Proc. 20, 681–695. 10.1109/TIP.2010.2076294

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3

  Boyd S. Parikh N. Chu E. Peleato B. Eckstein J. (2011). Distributed optimization and statistical learning via the alternating direction method of multipliers. Trends Mach. Learn. 3, 1–122. 10.1561/2200000016

  • CrossRef
  • Google Scholar

• 4

  Buades A. Coll B. Morel J.-M. (2005). “A non-local algorithm for image denoising,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). San Diego, CA: IEEE, 60-65.

  • Google Scholar

• 5

  Cai J.-F. Candès E. J. Shen Z. (2010). A singular value thresholding algorithm for matrix completion. SIAM J. Optimizat. 20, 1956–1982. 10.1137/080738970

  • CrossRef
  • Google Scholar

• 6

  Chen F. Zhang L. Yu H. (2015). “External patch prior guided internal clustering for image denoising,” in Proceedings of the IEEE International Conference on Computer Vision (Santiago, CA: IEEE), 603–611.

  • Google Scholar

• 7

  Dabov K. Foi A. Katkovnik V. Egiazarian K. (2007). Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Trans. Image Process. 16, 2080–2095. 10.1109/TIP.2007.901238

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8

  Dey N. Blanc-Feraud L. Zimmer C. Roux P. Kam Z. Olivo-Marin J.-C. et al . (2006). Richardson-lucy algorithm with total variation regularization for 3d confocal microscope deconvolution. Microsc. Res. Tech. 69, 260–266. 10.1002/jemt.20294

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9

  Dong W. Shi G. Li X. (2012a). Nonlocal image restoration with bilateral variance estimation: a low-rank approach. IEEE Trans. Image Process. 22, 700–711. 10.1109/TIP.2012.2221729

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10

  Dong W. Zhang L. Shi G. Li X. (2012b). Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process. 22, 1620–1630. 10.1109/TIP.2012.2235847

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11

  Elad M. Aharon M. (2006). Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15, 3736–3745. 10.1109/TIP.2006.881969

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12

  Fazel M. Hindi H. Boyd S. P. (2001). “A rank minimization heuristic with application to minimum order system approximation,” in Proceedings of the 2001 American Control Conference.(Cat. No. 01CH37148). Arlington, VA: IEEE, 4734–4739.

  • Google Scholar

• 13

  Gu S. Zhang L. Zuo W. Feng X. (2014). “Weighted nuclear norm minimization with application to image denoising,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2862–2869.

  • Google Scholar

• 14

  Guo Q. Gao S. Zhang X. Yin Y. Zhang C. (2017). Patch-based image inpainting via two-stage low rank approximation. IEEE Trans. Vis. Comput. Graph. 24, 2023–2036. 10.1109/TVCG.2017.2702738

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15

  Hu Y. Zhang D. Ye J. Li X. He X. (2012). Fast and accurate matrix completion via truncated nuclear norm regularization. IEEE Trans. Pattern Anal. Mach. Intell. 35, 2117–2130. 10.1109/TPAMI.2012.271

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16

  Jon K. Sun Y. Li Q. Liu J. Wang X. Zhu W. (2021). Image restoration using overlapping group sparsity on hyper-laplacian prior of image gradient. Neurocomputing420, 57–69. 10.1016/j.neucom.2020.08.053

  • CrossRef
  • Google Scholar

• 17

  Keller J. M. Gray M. R. Givens J. A. (1985). A fuzzy k-nearest neighbor algorithm. IEEE transactions on systems, man, and cybernetics. IEEE Trans. Syst. Man. Cybern. 4, 580–585. 10.1109/TSMC.1985.6313426

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18

  Keshavarzian R. Aghagolzadeh A. Rezaii T. Y. (2019). Llp norm regularization based group sparse representation for image compressed sensing recovery. Signal Proc.: Image Commun. 78, 477–493. 10.1016/j.image.2019.07.021

  • CrossRef
  • Google Scholar

• 19

  Liu S. Wu G. Liu H. Zhang X. (2017). Image restoration approach using a joint sparse representation in 3d-transform domain. Digital Signal Proc. 60, 307–323. 10.1016/j.dsp.2016.10.008

  • CrossRef
  • Google Scholar

• 20

  Lu C. Shi J. Jia J. (2013). “Online robust dictionary learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 415–422.

  • Google Scholar

• 21

  Mairal J. Bach F. Ponce J. Sapiro G. Zisserman A. (2009). “Non-local sparse models for image restoration,” in 2009 IEEE 12th International Conference on Computer Vision. Kyoto: IEEE, 2272–2279.

  • Google Scholar

• 22

  Mun S. Fowler J. E. (2009). “Block compressed sensing of images using directional transforms,” in 2009 16th IEEE International Conference on Image Processing (ICIP). Cairo: IEEE, 3021–3024.

  • Google Scholar

• 23

  Osher S. Burger M. Goldfarb D. Xu J. Yin W. (2005). An iterative regularization method for total variation-based image restoration. Multiscale Model. Simul. 4, 460–489. 10.1137/040605412

  • CrossRef
  • Google Scholar

• 24

  Ou Y. Luo J. Li B. Swamy M. S. (2020). Gray-level image denoising with an improved weighted sparse coding. J. Vis. Commun. Image Represent. 72, 102895. 10.1016/j.jvcir.2020.102895

  • CrossRef
  • Google Scholar

• 25

  Ou Y. Swamy M. Luo J. Li B. (2022a). Single image denoising via multi-scale weighted group sparse coding. Signal Proc. 200, 108650. 10.1016/j.sigpro.2022.108650

  • CrossRef
  • Google Scholar

• 26

  Ou Y. Zhang B. Li B. (2022b). Multi-scale low-rank approximation method for image denoising. Multimedia Tools Applicat. 81, 20357–20371. 10.1007/s11042-022-12083-z

  • CrossRef
  • Google Scholar

• 27

  Papyan V. Elad M. (2015). Multi-scale patch-based image restoration. IEEE Trans. Image Process. 25, 249–261. 10.1109/TIP.2015.2499698

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 28

  Ruder S. (2016). An overview of gradient descent optimization algorithms. arXiv. abs/1609.04747. 10.48550/arXiv.1609.04747

  • CrossRef
  • Google Scholar

• 29

  Rudin L. I. Osher S. Fatemi E. (1992). Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenom. 60, 259–268. 10.1016/0167-2789(92)90242-F

  • CrossRef
  • Google Scholar

• 30

  Srebro N. Jaakkola T. (2003). “Weighted low-rank approximations,” in Proceedings of the 20th International Conference on Machine Learning (ICML-03) (Washington, DC: AAAI Press), 720–727.

  • Google Scholar

• 31

  Tirer T. Giryes R. (2018). Image restoration by iterative denoising and backward projections. IEEE Trans. Image Process. 28, 1220–1234. 10.1109/TIP.2018.2875569

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 32

  Wang Z. Bovik A. C. Sheikh H. R. Simoncelli E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612. 10.1109/TIP.2003.819861

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 33

  Xie Y. Gu S. Liu Y. Zuo W. Zhang W. Zhang L. (2016). Weighted schatten p-norm minimization for image denoising and background subtraction. IEEE Trans. Image Process. 25, 4842–4857. 10.1109/TIP.2016.2599290

  • CrossRef
  • Google Scholar

• 34

  Xu J. Zhang L. Zuo W. Zhang D. Feng X. (2015). “Patch group based nonlocal self-similarity prior learning for image denoising,” in Proceedings of the IEEE International Conference on Computer Vision (Santiago, CA: IEEE), 244–252.

  • Google Scholar

• 35

  Yair N. Michaeli T. (2018). “Multi-scale weighted nuclear norm image restoration,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3165–3174.

  • Google Scholar

• 36

  Zha Z. Liu X. Zhang X. Chen Y. Tang L. Bai Y. et al . (2018a). Compressed sensing image reconstruction via adaptive sparse nonlocal regularization. Visual Comp. 34, 117–137. 10.1007/s00371-016-1318-9

  • CrossRef
  • Google Scholar

• 37

  Zha Z. Liu X. Zhou Z. Huang X. Shi J. Shang Z. et al . (2017). “Image denoising via group sparsity residual constraint,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New Orleans, LA: IEEE, 1787–1791.

  • Google Scholar

• 38

  Zha Z. Wen B. Yuan X. Zhou J. Zhu C. Kot A. C. (2022). Low-rankness guided group sparse representation for image restoration. IEEE Trans. Neural. Netw. Learn Syst. 34, 7593–7607. 10.1109/TNNLS.2022.3144630

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 39

  Zha Z. Yuan X. Wen B. Zhou J. Zhang J. Zhu C. (2019). From rank estimation to rank approximation: Rank residual constraint for image restoration. IEEE Trans. Image Process. 29, 3254–3269. 10.1109/TIP.2019.2958309

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 40

  Zha Z. Yuan X. Wen B. Zhou J. Zhu C. (2018b). “Joint patch-group based sparse representation for image inpainting,” in Asian Conference on Machine Learning, 145–160.

  • Google Scholar

• 41

  Zha Z. Yuan X. Wen B. Zhou J. Zhu C. (2020a). Group sparsity residual constraint with non-local priors for image restoration. IEEE Trans. Image Process. 29, 8960–8975. 10.1109/TIP.2020.3021291

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 42

  Zha Z. Yuan X. Zhou J. Zhu C. Wen B. (2020b). Image restoration via simultaneous nonlocal self-similarity priors. IEEE Trans. Image Process. 29, 8561–8576. 10.1109/TIP.2020.3015545

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 43

  Zhang J. Ghanem B. (2018). “Ista-net: Interpretable optimization-inspired deep network for image compressive sensing,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Salt Lake City, UT: IEEE), 1828–1837.

  • Google Scholar

• 44

  Zhang J. Zhao C. Zhao D. Gao W. (2014a). Image compressive sensing recovery using adaptively learned sparsifying basis via l0 minimization. Signal Proc. 103, 114–126. 10.1016/j.sigpro.2013.09.025

  • CrossRef
  • Google Scholar

• 45

  Zhang J. Zhao D. Gao W. (2014b). Group-based sparse representation for image restoration. IEEE Trans. Image Process. 23, 3336–3351. 10.1109/TIP.2014.2323127

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 46

  Zhang J. Zhao D. Xiong R. Ma S. Gao W. (2014c). Image restoration using joint statistical modeling in a space-transform domain. IEEE Trans. Circuits Syst. Video Technol. IEEE. 24, 915–928. 10.1109/TCSVT.2014.2302380

  • CrossRef
  • Google Scholar

• 47

  Zhang J. Zhao D. Zhao C. Xiong R. Ma S. Gao W. (2012). Image compressive sensing recovery via collaborative sparsity. IEEE J. Emerg. Sel. Topics Power Electron. 2, 380–391. 10.1109/JETCAS.2012.2220391

  • CrossRef
  • Google Scholar

• 48

  Zhang K. Zuo W. Gu S. Zhang L. (2017). “Learning deep cnn denoiser prior for image restoration,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Honolulu, HI: IEEE), 3929–3938.

  • Google Scholar

• 49

  Zoran D. Weiss Y. (2011). “From learning models of natural image patches to whole image restoration,” in 2011 International Conference on Computer Vision. Barcelona: IEEE, 479–486.

  • Google Scholar