Natural image restoration based on multi-scale group sparsity residual constraints

The Group Sparse Representation (GSR) model shows excellent potential in various image restoration tasks. In this study, we propose a novel Multi-Scale Group Sparse Residual Constraint Model (MS-GSRC) which can be applied to various inverse problems, including denoising, inpainting, and compressed sensing (CS). Our new method involves the following three steps: (1) finding similar patches with an overlapping scheme for the input degraded image using a multi-scale strategy, (2) performing a group sparse coding on these patches with low-rank constraints to get an initial representation vector, and (3) under the Bayesian maximum a posteriori (MAP) restoration framework, we adopt an alternating minimization scheme to solve the corresponding equation and reconstruct the target image finally. Simulation experiments demonstrate that our proposed model outperforms in terms of both objective image quality and subjective visual quality compared to several state-of-the-art methods.


. Introduction
Unsuitable equipment and other disturbances unavoidably contribute noise in the target images.Image denoising is a crucial area of image processing and has attracted much attention from scholars in related fields recently.Digital image denoising techniques have a wide range of uses, involving disciplines of medicine and industry, and also in spectral images for weather forecasting, remote sensing images, and so on.Taking image denoising as a basis, the method can be introduced to more image restoration problems and be useful in more fields (Buades et al., 2005;Osher et al., 2005;Elad and Aharon, 2006;Zoran and Weiss, 2011;Gu et al., 2014;Zhang et al., 2014b;Liu et al., 2017;Keshavarzian et al., 2019;Ou et al., 2020;Zha et al., 2020aZha et al., , 2022;;Jon et al., 2021).This task aims to generate a latent image x from the degraded version y.The process modeling can be depicted as Where H is an irreversible linear operator in matrix form and n is the additive white Gaussian noise vector.By requiring H, Eq.( 1) can be converted to diverse image restoration problems.For example, Eq.(1) represents the image denoising problem if H is an identity (Elad and Aharon, 2006;Ou et al., 2020); Eq.(1) denotes the image inpainting problem if H is a mask (Liu et al., 2017;Zha et al., 2020a); and Eq.( 1) stands for the image CS problem if H is an undersampled random projection matrix (Keshavarzian et al., 2019;Zha et al., 2022).We concentrate on image denoising, inpainting, and CS challenges in this article.
Given that the problem always ill-posed, it is common to use image priors to regularize the model so as to gain excellent restored images.Namely, the Maximum A Posteriori (MAP) approach allows for the image restoration problem to be formulated as a mathematical equation to address the minimization problem: The former is the data-fidelity term and the latter is the image prior constraint term.The weights between these two terms are regulated by the parameter λ.After establishing the mathematical model, we conceived an optimization algorithm to address various image restoration problems.The method yields a reconstructed image that approximates a clean image after several iterations.
Numerous image prior models have been put forward in earlier studies, mainly classified into local smoothness (Rudin et al., 1992;Osher et al., 2005;Dey et al., 2006), non-local self-similarity (Fazel et al., 2001;Buades et al., 2005;Gu et al., 2014), and sparsity (Zhang et al., 2014b;Ou et al., 2020Ou et al., , 2022a).Yet, the curse of dimensionality makes it difficult to construct a global model for the entire image.Therefore, the approach of building patch priors has become popular in recent years for its efficiency and convenience.
Sparse representation is one of the most representative patchbased priors.Elad and Aharon (2006) proposed K-SVD (K-Singular Value Decomposition) which is a pioneering work in applying sparse coding to image denoising.NSS is another crucial prior information widely used.Buades et al. (2005) proposed the first model using NSS for image denoising.In addition, the high correlation between patches leading to the data matrix of a clean image is as often low-rank.Related studies mainly fall into two categories: low-rank matrix factorization (LRMF) (Srebro and Jaakkola, 2003;Buades et al., 2005) and the Nuclear Norm Minimization (NNM) (Fazel et al., 2001;Hu et al., 2012).NNM is the more popular one in most cases.Gu et al. presented the Weighted Nuclear Norm Minimisation model (WNNM) (Gu et al., 2014) which dramatically enhances the flexibility of NNM, and it remains among most widespread image denoising methods.Apart from this, RRC (Zha et al., 2019), which makes use of low-rank residuals for modeling, has also achieved good quality in various image restoration problems.
Some studies have combined image sparsity and self-similarity to modeling, and these algorithms have shown great potential in image restoration research.For instance, in the study by Dabov et al. (2007), BM3D applies NSS to cluster patches before collaborative filtering, which is a benchmark method in the current area of image denoising.Both NCSR (Dong et al., 2012b) and GSR (Zhang et al., 2014b) use the NSS property to aggregate image patches into groups, and then perform sparse coding on the selfsimilar groups.Mairal J et al. devised the LSSC (Mairal et al., 2009) to force all self-similar groups to be imposed with the same dictionary.Zha et al. (2017) designed an efficient GSRC model that converts the task of image denoising into one of minimizing group sparse residuals.In addition, Zha et al. (2020a) also proposed a GSRC-NLP model with a better image restoration result based on the above.
Another groundbreaking patch-based image recovery method is Expected Patch Log Likelihood (EPLL) (Zoran and Weiss, 2011) which restores images by learning a Gaussian mixture model(GMM).Later on, Zoran et al. introduces a multi-scale EPLL (Papyan and Elad, 2015) model, which can improve the performance of image restoration further.Subsequently, image denoising methods using external GMM priors have been widely used.Most of the relevant studies have combined external GMM with internal NSS for modeling, such as Xu et al. (2015) proposed PGPD, Chen et al. (2015) proposed PCLR, and Zha et al. (2020b) proposed SNSS.
In addition to the above methods, deep convolutional neural networks (CNNs) (Zhang et al., 2017;Zhang and Ghanem, 2018) is an emerging approach in recent years, but it requires learning in an external database before restoring damaged images.
It is not comprehensive to only consider the sparsity or lowrankness property of the image.Hence, with the aim of obtaining a higher-quality restored image, our study uses the low-rank property of similar groups as a constraint in combination with sparsity to design the model.Furthermore, based on the NSS property, we can not only find similar patches for a specified patch on a single scale image but also extend the search window to multi-scales.Finally, we propose a novel Multi-scale Group Sparsity Residual Constraint (MS-GSRC) model with the following innovations: 1. We propose a novel MS-GSRC model that provides a simple yet effective approach for image restoration: find neighbor patches with an overlapping scheme for the input degraded image using a multi-scale strategy and perform a group sparse coding on these similar patches with a low-rank constraint.2.An alternating minimization mechanism with an automatically tuned parameter scheme is applied to our proposed model, which guarantees a closed-form solution at each step.3. Our proposed MS-GSRC model is validated on three tasks: denoising, inpainting, and compressed sensing.The model performs competitive in both objective image quality and subjective visual quality compared to several state-of-the-art image restoration methods.
The remainder of this article is as follows: In Section 2, after the brief overview of the GSRC framework and LR methods, we introduce a novel MS-GSRC model.Section 3 adopts an alternating minimization scheme with self-adjustable parameters to resolve our proposed algorithm.Section 4 lists extensive experimental results that prove the feasibility of our model.Conclusion is presented in Section 5.

. Models
In this part, we briefly review some relevant knowledge and present our new model.

. . Group-based sparse representation
Principles of the GSR model can be described as follows: divide the image into many overlapping patches, find self-similarity groups for each image patch using the NSS property, perform sparse coding for each self-similarity group, and finally reconstruct the image (Dong et al., 2012b;Zha et al., 2020a;Ou et al., 2022a).Specifically, the image x ∈ R M is divided into m overlapping patches {x i } m i=1 , where x i ∈ R n×n .Next, for each overlapping patch x i , we use the K-Nearest Neighbor classification (KNN) algorithm (Keller et al., 1985;Xie et al., 2016) to select k neighbor patches from a W × W search window to form the group K i .Subsequently, stack all K i into a data matrix X i ∈ R n×k ; this matrix contains each element of K i as its column, i.e., X i = x i,1 , x i,2 . . .x i,k , where x i,j k j=1 denotes the k-th similar patch of the k-th group.Each similarity group X i is represented sparsely as Xi = D Bi , where D i denotes the dictionary.
Nevertheless, solving the 0-norm minimization problem is NPhard, so for the ease of making the solution, The sparse code Bi is obtained from the following equation (Zhang et al., 2014b): It is well-known that clean images x are unavailable in image restoration problems.Thus, we replace x with degenerate images y ∈ R M×M .Eq.( 3) can be transformed into the problem of recovering the group sparse code A i from Y i : The restored X i is obtained by Xi = D i Âi , and the final complete image X can be gained by simple averaging {X i } m i=1 . .

. Group sparsity residual constraint
After observing the GSR model, it is clear that the closer the computed A approximates to B, the better the quality of the final restoration image.Consequently, the following definition of the group sparsity residual constraint (GSRC) (Zha et al., 2017) is given: R = A − B. Then, Eq.(4) for solving the group sparse coefficients A i can be converted into: This model uses BM3D to restore the the degenerate observation y to the image z.Moreover, z can be viewed as a good approximation of the target x considering BM3D has an excellent denoising performance.Thus, the group sparsity coefficients B i can be obtained from z.In the study by Zha et al. (2020a), GSRC-NLP uses NLP before constraining the input image.

. . Low-rank approximation
According to Gu et al. (2014), Zha et al. (2019), andZha et al. (2020b), it can be found that NNM is a popular low-rank approximations methods.For X, define the i-th singular value as σ i (x), and the nuclear norm as X * = i σ i (x).The specific solution for X is: Equation ( 6) yields a simple solution: X = US τ V T , where Ŷ = U V T is the SVD for Y and S τ ( ) is a soft-thresholding (Cai et al., 2010) function.Namely, S τ ( ) ii = max( ii − τ , 0), where ii is the diagonal element of .

. . Multi-scale GSRC
The established GSRC model has performed well in image denoising, but it requires additional pre-processing of degraded images for obtaining the group sparsity coefficients B. Thus, we combine group sparsity and low-rank property to build a model.Furthermore, the GSRC model only focuses on a single scale.However, it is evident that NSS can appear not only on the original scale of an image but also on a coarse scale, so we can find neighbor patches for the original image patch at multi-scales (Yair and Michaeli, 2018;Ou et al., 2022a,b).The specific steps of our proposed new Multi-Scale Group Sparse Residual Constraint (MS-GSRC) model are as follows: (a) First, we use KNN to find a specified number of similar patches from both the original scale and scaled-down version for the overlapping patches of the input image.
(b) Then, these similar patches are stacked separately into groups.
(c) Next, the low-rank constraint is imposed on each group to obtain good group sparsity coefficients B i .
(d) After estimating the group sparsity coefficients A i by using the group sparsity residuals R i , each group was recovered in sequence.
(e) Finally, we select the patch belonging to the original image from each group, and aggregate the complete image by simple averaging.
We propose the following constraint function: MS is a multi-scale similarity group, which is a matrix with k nearest neighbor patches matched for each original image patch.These similar patches are derived from both the original and coarse scales of the image.The window size is W × W in the original scale, and it isχ W × χ W in the other scale images, where χ indicates the scale factor (0 < χ < 1).χ will be set to different values in different experiments.
For image denoising, for example, the flowchart of MS-GSRC model is shown in Figure 1.

. Algorithm for image restoration
This section is a detailed analysis of our proposed MS-GSRC model.The solution of this algorithm is obtained using an alternating minimization method whose parameter is selfadjustment.
First, we divide Eq.( 7) into three sub-problems: . .A i sub-problem Given x and D i , we get a sub-problem of A i : where , p i stand for the vector representations of A i , B i , and P i , respectively.D i is a dictionary, A crucial step for solving the A i problem is to design an efficient D i .The restored image is prone to visual artifacts (Lu et al., 2013) if learning the over-complete dictionary.To reduce this terrible phenomenon, we choose to adopt principal component analysis (PCA) (Abdi and Williams, 2010) for learning the dictionary D i in this study because PCA is more robust and adjustable.
Equation ( 8) can be deduced as a closed-form solution:

FIGURE
The widely tested images for experiences.
Soft(•) represents the soft-thresholding operator.Since x is an unknown target image, it is impossible to gain the group sparse coefficients B i directly.Consequently, we must utilize methods to gain an approximation value.
The introduction of low-rank constraints into the model is a practical approach.After applying LR constraints to the Y i group, we can obtain a matrix S i .The clean group sparsity coefficients B i can be computed from S i .It is easy to derive the following equation: where B i * = j δ i,j , δ i,j s j=1 are singular value of matrix.Apparently, we are able to get a closed-form solution for B i : where A i = U i i V T i is the SVD for A i and i is the diagonal element of the singular matrix.

. . x sub-problem
Given A i and D i , subproblem of x in Eq.( 7) turns into: Clearly, Eq.( 13) is a quadratic optimization equation.We adopt Alternate Directional Multiplication Method (ADMM) (Boyd et al., 2011) to simplify the optimization process.
First, we bring in an auxiliary variable s = x MS , and Eq.( 13) can be converted into an equivalence equation: By observing Eq.( 14), it is plain that this equation has three unknown variables requiring solutions.Thus, we decompose Eq.( 14) into three iterative processes.In the t-th iteration: The parameter c indicates the Lagrangian multiplier.To make the derivation process look more concise, we omit t in the following formulation expression.
Update s : Given D i A i , x, and c, s can be represented as a closed-form solution by Eq.( 16), namely: Since I is a matrix of identities and R T i R i represents a diagonal matrix, µI + ζ m i=1 R T i R i is positive.So the above formula is valid.
Update x : Given s and c, Eq.( 15) provides a solution to the variable x: Notably, since H is an unstructured random projection matrix, the cost required to solve x using Eq.( 19) directly is too high in CS problem.Hence, after setting step size γ and gradient direction q, we employ the gradient descent method (Ruder, 2016): x = x − γ q to rewrite Eq.( 19) as: In addition, it is recommended to compute H T H and H T y in advance to further enhance the algorithm efficiency.

. . Parameter settings
In the model we proposed above, there are four parameters (µ, λ, θ , ζ ) requiring setting.Here, we set a strategy for the parameters that can be automatically adjusted in each iteration, which allows us to achieve more robust and accurate experimental results.
The noise standard deviation σ n is automatically updated in each iteration (Osher et al., 2005): Where ω represents a scaling factor, it is evident from Gu et al. (2014) and Chen et al. (2015) that this approach to regularize σ e has been implemented in diverse models and has exhibited positive performance.
After setting σ e , the value of µ is tuned to change in proportion to σ 2 e (Zha et al., 2022): where ρ denotes a constant.
Frontiers in Neuroscience frontiersin.org Moreover, the regularization parameters λ and θ represent the constraint penalties on sparsity and LR, respectively.Inspired by Dong et al. (2012a), they are adjusted in each iteration as follows: where m i is the estimated standard variance of R i and n i stands for the estimated standard variance of i .The ε and ǫ are two small constants to avoid zero divisors.α and β are set to two constants.Finally, parameter ζ is also set to a fixed constant.
The detailed procedure of the MS-GSRC algorithm is presented in Algorithm 1.

. Experiences
In this chapter, extensive trial are conducted on image denoising, inpainting, and CS to verify that our proposed MS-GSRC model possesses better image restoration capabilities compared to some classical methods.To obtain intuitive comparison results, we set on two metrics: peak signal-to-noise ratio (PSNR) and structural self-similarity (SSIM) (Wang et al., 2004).
PSNR is commonly used to measure signal distortion.This parameter is calculated based on the gray scale values of the image pixels.Although sometimes the value of PSNR is not consistent with competent human perception, it remains an important reference evaluation metric.SSIM is a metric intended for assessing similarity between two images, which is an intuitive human standard for evaluating image quality.
If the degraded image is in color, we mainly recover the luminance channel due to the fact that variations in the luminance of color images are more easily perceived by the human eye.
The codes for all comparison algorithms used in this study are obtained from the original author's homepage and uses the given default parameters directly.For reasons of limited space, only a few images frequently used for testing are detailed list in Figure 2. In all tables, the data marked in red represent the best values.
Our MS-GSRC method is compared with several recently proposed popular denoising methods and classical traditional denoising methods, including BM3D (Dabov et al., 2007), PGPD (Xu et al., 2015), WNNM (Gu et al., 2014), NCSR (Dong et al.,  2012b), RRC (Zha et al., 2019), LGSR (Zha et al., 2022) and GSRC-NLP (Zha et al., 2020a).Of all the comparison methods, BM3D is a frequently adopted benchmarking method, NCSR, PGPD, and GSRC-NLP all use GSR as a prior, and WNNM and RRC exploit low-rankness knowledge.And LGSR combines GSR and LR.Besides, both GSRC-NLP and our proposed model use the GSRC framework.Taking 12 frequently used images as an example, Table 1 lists the PSNR and SSIM results for various denoising methods at different noise levels.It is observed that our proposed MS-GSRC method produced superior performance.Specifically, the average PSNR and SSIM we achieve are improved by (0.47 dB, 0.0149) compared to BM3D, (0.38 dB, 0.0107) compared to PGPD, (0.07 dB, 0.0032) compared to WWNM, (0.42 dB, 0.0149) compared to NCSR, (0.25 dB, 0.0066) compared to RRC, (0.1 dB, 0.0005) compared to LGSR, and (0.19 dB, 0.0054) compared to GSRC-NLP.We also utilize the BSD68 dataset (Wang et al., 2004) to assess the denoising ability of all compared approaches.We can observe from Table 2 that the average PSNR gains obtained by our proposed MS-GSRC method in comparison to the BM3D, PGPD, WNNM, NCSR, RRC, GSRC-NLP, and LGSR methods are 0.24 dB, 0.16 dB, 0.02 dB, 0.27 dB, 0.24 dB, 0.23 dB, and 0.03 dB.Meanwhile, on average, the proposed MS-GSRC achieve an SSIM improvement of 0.0153 on BM3D, 0.0169 on PGPD, 0.0064 on WNNM, 0.0137 on NCSR, 0.0228 on RRC, 0.0027 on LGSR, and 0.0194 on GSRC-NLP.Evidently, our proposed MS-GSRC method yields better PSNR and SSIM in almost all noise cases.Our method is only 0.01 dB lower than WWNM in PSNR, but 0.0052 higher than in SSIM at σ = 75.Beyond objective metrics, the subjective perception of the human body is also a crucial criterion for assessing the quality of an image.Consequently, we present the visual contrast between the two images of starfish and 223,061  ).(C) PGPD (PSNR = .dB and SSIM = .
restored by different methods in Figures 3, 4, respectively.Figure 3 indicates that BM3D, PGPD, WNNM, and RRC are likely to oversmooth the restored image, whereas NCSR, GSRC-NLP, and LGSR can lead to the appearance of some undesired visual artifacts.As can be seen in Figure 4, although the image restored by WNNM has a higher PSNR, the image restored by our MS-GSRC method has a higher SSIM value and presents a better visual effect.PGPD, NCSR, RRC, and GSRC-NLP are susceptible to loss of detail in the restored images, while BM3D, WNNM, and LGSR may result in undesirable artifacts. .
Table 3 illustrates the PSNR and SSIM results for each method on the 12 frequently used test images.As observed in Table 3, our proposed method exceeds the comparison algorithm virtually often when it comes to image inpainting performance.The proposed MS-GSRC outperforms SAIST, JSM, GSR, TSLRA, JPG-SR, LGSR, and IDBP approaches in average PSNR performance, with gains of 5.8 dB, 1.43 dB, 0.51 dB, 1.82 dB, 0.51 dB, 0.13 dB, and 1.26 dB, respectively.Additionally, on average, the proposed MS-GSRC surpasses SAIST by 0.0776, JSM by 0.0216, GSR by 0.0025, TSLRA by 0.0238, JPG-SR by 0.007, LGSR by 0.0012, and IDBP by 0.022.
Similarly, two images are selected for detailed visual analysis.The image butterfly with a 80% loss of pixels restored by different methods are presented in Figure 5.Moreover, Figure 6 displays the outcomes of a visual comparison of image flowers with a 70% loss of pixels restored with different algorithms.By analyzing the visual comparison images, we can find that images restored using SAIST, JSM, TSLRA, IDBP GSR, and JPG-SR are susceptible to excessive smoothing, and images restored using LGSR tend to show excessive visual artifacts.The images restored using our proposed MS-GSRC model have significantly better restoration capabilities with regard to image detail and edges.).(C) JSM (PSNR = .dB and SSIM = .
BSC (Mun and Fowler, 2009), RCOS (Zhang et al., 2012), ALSB (Zhang et al., 2014a), GSR (Zhang et al., 2014b), ASNR (Zha et al., 2018a), and LGSR (Zha et al., 2022) are choosen as competing methods.Among them, GSR performs a sparse representation on similar groups of images, ASNR is an image of the CS method that extends on the basis of NCSR, and LGSR combines sparsity and LR.Similarly, we selected 12 images frequently used in image restoration experiments as test images.Table 4 presents the average outcomes of PSNR and SSIM of the restored images using different method.To be concrete, the proposed MS-GSRC model over BCS, RCOS, ALSB, GSR, ASNR, and LGSR methods are 5.36 dB, 2.34 dB, 1.24 dB, 0.46 dB, 0.45 dB, and 0.14d B in PSNR and 0.1591, 0.0634, 0.0023, 0.0056, 0.0082, and 0.0029 in SSIM, respectively.Due to the other competing algorithms used in this thesis, all use BCS to pre-process CS images, and here we use the BCSprocessed images as corrupted images.Figure 7 shows the visual contrast of the image fence with 0.1 N CS measurements, and we can observe that RCOS and ALSB are less capable of restoring details, GSR and LGSR lead to over-smooth, and ASNR generates some redundant artifacts.Figure 8 illustrates the visual comparison of the image leaves measured with 0.1N CS.All comparison images have strong ringing phenomena and present terrible artifacts.In Figure 9, we have selected the image airplane processed with 0.2N CS for detailed analysis.It is obvious that the details of the images restored by ALSB and LGSR are seriously missing.The images restored by RCOS, GSR, and ASNR produced more artifacts.In the above three cases, our proposed MS-GSRC algorithm significantly outperforms other competing algorithms in recovering the image overall and some texture details.

. Conclusion
In this study, we propose a novel model Multi-Scale Group Sparse Residual Constraint Model (MS-GSRC) for image restoration.This model introduces the low-rank property into the group sparse residual framework and finds similar patches for overlapping patches of the input image using a multi-scale strategy.Furthermore, under the MAP restoration framework, an alternating minimization method with adaptive tunable parameters is used to deliver a robust optimization solution for our ).(B) RCOS (PSNR = .dB, SSIM = .
MS-GSRC method.We employ the MS-GSRC model to three image restoration problems, namely, denoising, inpainting, and compressed sensing.Extensive simulation trials show that our novel model performs superior to many classical methods in terms of both objective image quality and subjective visual quality.

TABLE PSNR (
dB) and SSIM comparison of di erent methods for image denoising.Image Airplane Flower Foreman J.Bean Lake Leaves Lena Lin Monarch Starfish Pentagon Peppers Average Image Airplane Flower Foreman J.Bean Lake Leaves Lena Lin Monarch Starfish Pentagon Peppers Average

TABLE PSNR (
dB) and SSIM comparison of di erent methods for image denoising on BSD dataset.

TABLE PSNR (
dB) and SSIM comparison of di erent methods for image CS on test images.
The data marked in red represent the best values.