Frontiers journals are at the top of citation and impact metrics

# Frontiers in Applied Mathematicsand Statistics

## Original Research ARTICLE

Front. Appl. Math. Stat., 27 March 2017 | https://doi.org/10.3389/fams.2017.00003

# Optimal Rates for the Regularized Learning Algorithms under General Source Condition

• Department of Mathematics, Indian Institute of Technology Delhi, New Delhi, India

We consider the learning algorithms under general source condition with the polynomial decay of the eigenvalues of the integral operator in vector-valued function setting. We discuss the upper convergence rates of Tikhonov regularizer under general source condition corresponding to increasing monotone index function. The convergence issues are studied for general regularization schemes by using the concept of operator monotone index functions in minimax setting. Further we also address the minimum possible error for any learning algorithm.

## 1. Introduction

Learning theory [13] aims to learn the relation between the inputs and outputs based on finite random samples. We require some underlying space to search the relation function. From the experiences we have some idea about the underlying space which is called hypothesis space. Learning algorithms tries to infer the best estimator over the hypothesis space such that f(x) gives the maximum information of the output variable y for any unseen input x. The given samples ${\left\{{x}_{i},{y}_{i}\right\}}_{i=1}^{m}$ are not exact in the sense that for underlying relation function f(xi) ≠ yi but f(xi) ≈ yi. We assume that the uncertainty follows the probability distribution ρ on the sample space X × Y and the underlying function (called the regression function) for the probability distribution ρ is given by

where ρ(y|x) is the conditional probability measure for given x. The problem of obtaining estimator from examples is ill-posed. Therefore, we apply the regularization schemes [47] to stabilize the problem. Various regularization schemes are studied for inverse problems. In the context of learning theory [2, 3, 810], the square loss-regularization (Tikhonov regularization) is widely considered to obtain the regularized estimator [9, 1116]. Gerfo et al. [6] introduced general regularization in the learning theory and provided the error bounds under Hölder's source condition [5]. Bauer et al. [4] discussed the convergence issues for general regularization under general source condition [17] by removing the Lipschitz condition on the regularization considered in Gerfo et al. [6]. Caponnetto and De Vito [12] discussed the square-loss regularization under the polynomial decay of the eigenvalues of the integral operator LK with Hölder's source condition. For the inverse statistical learning problem, Blanchard and Mücke [18] analyzed the convergence rates for general regularization scheme under Hölder's source condition in scalar-valued function setting. Here we are discussing the convergence issues of general regularization schemes under general source condition and the polynomial decay of the eigenvalues of the integral operator in vector-valued framework. We present the minimax upper convergence rates for Tikhonov regularization under general source condition Ωϕ, R, for a monotone increasing index function ϕ. For general regularization the minimax rates are obtained using the operator monotone index function ϕ. The concept of effective dimension [19, 20] is exploited to achieve the convergence rates. In the choice of regularization parameters, the effective dimension plays the important role. We also discuss the lower convergence rates for any learning algorithm under the smoothness conditions. We present the results in vector-valued function setting. Therefore, in particular they can be applied to multi-task learning problems.

The structure of the paper is as follows. In the second section, we introduce some basic assumptions and notations for supervised learning problems. In Section 3, we present the upper and lower convergence rates under the smoothness conditions in minimax setting.

## 2. Learning From Examples: Notations and Assumptions

In the learning theory framework [2, 3, 810], the sample space Z = X × Y consists of two spaces: The input space X (locally compact second countable Hausdorff space) and the output space (Y, 〈·, ·〉Y) (the real separable Hilbert space). The input space X and the output space Y are related by some unknown probability distribution ρ on Z. The probability measure can be split as ρ(x, y) = ρ(y|xX(x), where ρ(y|x) is the conditional probability measure of y given x and ρX is the marginal probability measure on X. The only available information is the random i.i.d. samples z = ((x1, y1), …, (xm, ym)) drawn according to the probability measure ρ. Given the training set z, learning theory aims to develop an algorithm which provides an estimator fz : XY such that fz(x) predicts the output variable y for any given input x. The goodness of the estimator can be measured by the generalization error of a function f which can be defined as

where V : Y × Y → ℝ is the loss function. The minimizer of ${E}\left(f\right)$ for the square loss function $V\left(f\left(x\right),y\right)=||f\left(x\right)-y|{|}_{Y}^{2}$ is given by

where fρ is called the regression function. The regression function fρ belongs to the space of square integrable functions provided that

We search the minimizer of the generalization error over a hypothesis space ${H}$,

where f${H}$ is called the target function. In case fρ${H}$, f${H}$ becomes the regression function fρ.

Because of inaccessibility of the probability distribution ρ, we minimize the regularized empirical estimate of the generalization error over the hypothesis space ${H}$,

where λ is the positive regularization parameter. The regularization schemes [47, 10] are used to incorporate various features in the solution such as boundedness, monotonicity and smoothness. In order to optimize the vector-valued regularization functional, one of the main problems is to choose the appropriate hypothesis space which is assumed to be a source to provide the estimator.

### 2.1. Reproducing Kernel Hilbert Space as a Hypothesis Space

Definition 2.1. (Vector-valued reproducing kernel Hilbert space) For non-empty set X and the real Hilbert space (Y, 〈·, ·〉Y), the Hilbert space (${H}$, 〈·, ·〉${H}$) of functions from X to Y is called reproducing kernel Hilbert space if for any x ∈ X and y ∈ Y the linear functional which maps f ∈ ${H}$ toy, f(x)〉Y is continuous.

By Riesz lemma [21], for every xX and yY there exists a linear operator Kx : Y${H}$ such that

Therefore, the adjoint operator ${K}_{x}^{*}:{H}\to Y$ is given by ${K}_{x}^{*}f=f\left(x\right)$. Through the linear operator Kx : Y${H}$ we define the linear operator K(x, t) : YY,

$K(x,t)y:=Kty(x).$

From Proposition 2.1 [22], the linear operator $K\left(x,t\right)\in {L}\left(Y\right)$ (the set of bounded linear operators on Y), K(x, t) = K(t, x)* and K(x, x) is non-negative bounded linear operator. For any m ∈ ℕ, {xi: 1 ≤ im} ∈ X, {yi: 1 ≤ im} ∈ Y, we have that $\sum _{i,j=1}^{m}〈{y}_{i},K\left({x}_{i},{x}_{j}\right){y}_{j}〉\ge 0$. The operator valued function $K:X×X\to {L}\left(Y\right)$ is called the kernel.

There is one to one correspondence between the kernels and reproducing kernel Hilbert spaces [22, 23]. So a reproducing kernel Hilbert space ${H}$ corresponding to a kernel K can be denoted as ${H}$K and the norm in the space ${H}$ can be denoted as ||·||${H}$K. In the following article, we suppress K by simply using ${H}$ for reproducing kernel Hilbert space and ||·||${H}$ for its norm.

Throughout the paper we assume the reproducing kernel Hilbert space ${H}$ is separable such that

(i) Kx : Y${H}$ is a Hilbert-Schmidt operator for all xX and $\kappa :=\sqrt{\underset{x\in X}{sup}Tr\left({K}_{x}^{*}{K}_{x}\right)}<\infty$.

(ii) The real function from X × X to ℝ, defined by (x, t) ↦ 〈Ktv, Kxw${H}$, is measurable ∀v, wY.

By the representation theorem [22], the solution of the penalized regularization problem (5) will be of the form:

Definition 2.2. let ${H}$ be a separable Hilbert space and ${\left\{{e}_{k}\right\}}_{k=1}^{\infty }$ be an orthonormal basis of ${H}$. Then for any positive operator $A\in {L}\left({H}\right)$ we define $Tr\left(A\right)=\sum _{k=1}^{\infty }〈A{e}_{k},{e}_{k}〉$. It is well-known that the number Tr(A) is independent of the choice of the orthonormal basis.

Definition 2.3. An operator $A\in {L}\left({H}\right)$ is called Hilbert-Schmidt operator if Tr(A*A) < ∞. The family of all Hilbert-Schmidt operators is denoted by ${{L}}_{2}\left({H}\right)$. For $A\in {{L}}_{2}\left({H}\right)$, we define $Tr\left(A\right)=\sum _{k=1}^{\infty }〈A{e}_{k},{e}_{k}〉$ for an orthonormal basis ${\left\{{e}_{k}\right\}}_{k=1}^{\infty }$ of ${H}$.

It is well-known that ${{L}}_{2}\left({H}\right)$ is the separable Hilbert space with the inner product,

$〈A,B〉L2(H)=Tr(B*A)$

and its norm satisfies

$||A||L(H)≤||A||L2(H)≤Tr(|A|),$

where $|A|=\sqrt{{A}^{*}A}$ and $||·|{|}_{{L}\left({H}\right)}$ is the operator norm (For more details see [24]).

For the positive trace class operator ${K}_{x}{K}_{x}^{*}$, we have

$||KxKx*||L(H)≤||KxKx*||L2(H)≤Tr(KxKx*)≤κ2.$

Given the ordered set $\text{x}=\left({x}_{1},\dots ,{x}_{m}\right)\in {X}^{m}$, the sampling operator ${S}_{\text{x}}:{H}\to {Y}^{m}$ is defined by Sx(f) = (f(x1), …, f(xm)) and its adjoint ${S}_{\text{x}}^{*}:{Y}^{m}\to {H}$ is given by

The regularization scheme (5) can be expressed as

where $||\text{y}|{|}_{m}^{2}=\frac{1}{m}\sum _{i=1}^{m}||{y}_{i}|{|}_{Y}^{2}$.

We obtain the explicit expression of fz, λ by taking the functional derivative of above expression over RKHS ${H}$.

Theorem 2.1. For the positive choice of λ, the functional (6) has unique minimizer:

Define fλ as the minimizer of the optimization functional,

Using the fact ${E}\left(f\right)=||{L}_{K}^{1/2}\left(f-{f}_{{H}}\right)|{|}_{{H}}^{2}+{E}\left({f}_{{H}}\right)$, we get the expression of fλ,

where the integral operator ${L}_{K}:{{L}}_{{\rho }_{X}}^{2}\to {{L}}_{{\rho }_{X}}^{2}$ is a self-adjoint, non-negative, compact operator, defined as

The integral operator LK can also be defined as a self-adjoint operator on ${H}$. We use the same notation LK for both the operators defined on different domains. It is well-known that ${L}_{K}^{1/2}$ is an isometry from the space of square integrable functions to reproducing kernel Hilbert space.

In order to achieve the uniform convergence rates for learning algorithms we need some prior assumptions on the probability measure ρ. Following the notion of Bauer et al. [4] and Caponnetto and De Vito [12], we consider the class of probability measures ${{P}}_{\varphi }$ which satisfies the assumptions:

(i) For the probability measure ρ on X × Y,

(ii) The minimizer of the generalization error f${H}$ (4) over the hypothesis space ${H}$ exists.

(iii) There exist some constants M, Σ such that for almost all xX,

(iv) The target function f${H}$ belongs to the class Ωϕ, R with

where ϕ is a continuous increasing index function defined on the interval [0, κ2] with the assumption ϕ(0) = 0. This condition is usually referred to as general source condition [17].

In addition, we consider the set of probability measures ${{P}}_{\varphi ,b}$ which satisfies the conditions (i), (ii), (iii), (iv) and the eigenvalues tn's of the integral operator LK follow the polynomial decay: For fixed positive constants α, β and b > 1,

Under the polynomial decay of the eigenvalues the effective dimension ${N}\left(\lambda \right)$, to measure the complexity of RKHS, can be estimated from Proposition 3 [12] as follows,

and without the polynomial decay condition (13), we have

$N(λ)≤||(LK+λI)-1||L(H)Tr(LK)≤κ2λ.$

We discuss the convergence issues for the learning algorithms (zfz${H}$) in probabilistic sense by exponential tail inequalities such that

$Probz{||fz-fρ||ρ≤ε(m)log(1η)}≥1-η$

for all 0 < η ≤ 1 and ε(m) is a positive decreasing function of m. Using these probabilistic estimates we can obtain error estimates in expectation by integration of tail inequalities:

where $||f|{|}_{\rho }=||f|{|}_{{{L}}_{{\rho }_{X}}^{2}}={\left\{{\int }_{X}||f\left(x\right)|{|}_{Y}^{2}d{\rho }_{X}\left(x\right)\right\}}^{1/2}$ and ${E}_{\text{z}}\left(\xi \right)={\int }_{{Z}^{m}}\xi d\rho \left({z}_{1}\right)\dots d\rho \left({z}_{m}\right)$.

## 3. Convergence Analysis

In this section, we analyze the convergence issues of the learning algorithms on reproducing kernel Hilbert space under the smoothness priors in the supervised learning framework. We discuss the upper and lower convergence rates for vector-valued estimators in the standard minimax setting. Therefore, the estimates can be utilized particularly for scalar-valued functions and multi-task learning algorithms.

### 3.1. Upper Rates for Tikhonov Regularization

In General, we consider Tikhonov regularization in learning theory. Tikhonov regularization is briefly discussed in the literature [7, 9, 10, 25]. The error estimates for Tikhonov regularization are discussed theoretically under Hölder's source condition [12, 15, 16]. We establish the error estimates for Tikhonov regularization scheme under general source condition f${H}$ ∈ Ωϕ,R for some continuous increasing index function ϕ and the polynomial decay of the eigenvalues of the integral operator LK.

In order to estimate the error bounds, we consider the following inequality used in the papers [4, 12] which is based on the results of Pinelis and Sakhanenko [26].

Proposition 3.1. Let ξ be a random variable on the probability space $\left(\Omega ,{B},P\right)$ with values in real separable Hilbert space ${H}$. If there exist two constants Q and S satisfying

then for any 0 < η < 1 and for all m ∈ ℕ,

$Prob{(ω1,…,ωm)∈Ωm:‖1m∑i=1m[ξ(ωi)−E(ξ(ωi))]‖H ≤2(Qm+Sm)log(2η)}≥1−η.$

In particular, the inequality (15) holds if

We estimate the error bounds for the regularized estimators by measuring the effect of random sampling and the complexity of f${H}$. The quantities described in Proposition 3.2 express the probabilistic estimates of the perturbation measure due to random sampling. The expressions of Proposition 3.3 describe the complexity of the target function f${H}$ which are usually referred to as the approximation errors. The approximation errors are independent of the samples z.

Proposition 3.2. Let z be i.i.d. samples drawn according to the probability measure ρ satisfying the assumptions (10), (11) and $\kappa =\sqrt{\underset{x\in X}{sup}Tr\left({K}_{x}^{*}{K}_{x}\right)}$. Then for all 0 < η < 1, we have

$||(LK+λI)−1/2{Sx∗y−Sx∗SxfH}||H≤2(κMmλ+Σ2N(λ)m)log(4η)(16)$

and

with the confidence 1 − η.

The proof of the first expression is the content of the step 3.2 of Theorem 4 [12] while the proof of the second expression can be obtained from Theorem 2 in De Vito et al. [25].

Proposition 3.3. Suppose f${H}$ ∈ Ωϕ,R. Then,

(i) Under the assumption that $\varphi \left(t\right)\sqrt{t}$ and $\sqrt{t}/\varphi \left(t\right)$ are non-decreasing functions, we have

(ii) Under the assumption that ϕ(t) and t/ϕ(t) are non-decreasing functions, we have

and

Under the source condition f${H}$ ∈ Ωϕ, R, the proposition can be proved using the ideas of Theorem 10 [4].

Theorem 3.1. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi }$ where ϕ is the index function satisfying the conditions that ϕ(t), t/ϕ(t) are non-decreasing functions. Then for all 0 < η < 1, with confidence 1 − η, for the regularized estimator fz (7) the following upper bound holds:

$||fz,λ-fH||H≤2{Rϕ(λ)+2κMmλ+4Σ2N(λ)mλ}log(4η)$

provided that

Proof. The error of regularized solution fz, λ can be estimated in terms of the sample error and the approximation error as follows:

Now fz, λfλ can be expressed as

$fz,λ-fλ=(Sx*Sx+λI)-1{Sx*y-Sx*Sxfλ-λfλ}.$

Then ${f}_{\lambda }={\left({L}_{K}+\lambda I\right)}^{-1}{L}_{K}{f}_{{H}}$ implies

$LKfH=LKfλ+λfλ.$

Therefore,

$fz,λ-fλ=(Sx*Sx+λI)-1{Sx*y-Sx*Sxfλ-LK(fH-fλ)}.$

Employing RKHS-norm we get,

where ${I}_{1}=||{\left({S}_{\text{x}}^{*}{S}_{\text{x}}+\lambda I\right)}^{-1}{\left({L}_{K}+\lambda I\right)}^{1/2}|{|}_{{L}\left({H}\right)}$, ${I}_{2}=||{\left({L}_{K}+\lambda I\right)}^{-1/2}\left({S}_{\text{x}}^{*}\text{y}-{S}_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}}\right)|{|}_{{H}}$ and ${I}_{3}=||{S}_{\text{x}}^{*}{S}_{\text{x}}-{L}_{K}|{|}_{{L}\left({H}\right)}$.

The estimates of I2, I3 can be obtained from Proposition 3.2 and the only task is to bound I1. For this we consider

which implies

provided that $||{\left({L}_{K}+\lambda I\right)}^{-1}\left({L}_{K}-{S}_{\text{x}}^{*}{S}_{\text{x}}\right)|{|}_{{L}\left({H}\right)}<1$. To verify this condition, we consider

$||(LK+λI)-1(Sx*Sx-LK)||L(H)≤I3/λ.$

Now using Proposition 3.2 we get with confidence 1 − η/2,

$||(LK+λI)-1(Sx*Sx-LK)||L(H)≤4κ2mλlog(4η).$

From the condition (21) we get with confidence 1 − η/2,

Consequently, using (25) in the inequality (24) we obtain with probability 1 − η/2,

From Proposition 3.2 we have with confidence 1 − η/2,

$||Sx*Sx-LK||L(H)≤2(κ2m+κ2m)log(4η).$

Again from the condition (21) we get with probability 1 − η/2,

Therefore, the inequality (23) together with (16), (20), (26), (27) provides the desired bound.

The following theorem discuss the error estimates in ${L}$2-norm. The proof is similar to the above theorem.

Theorem 3.2. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi }$ and fz, λ is the regularized solution (7) corresponding to Tikhonov regularization. Then for all 0 < η < 1, with confidence 1 − η, the following upper bounds holds:

(i) Under the assumption that ϕ(t), $\sqrt{t}/\varphi \left(t\right)$ are non-decreasing functions,

(ii) Under the assumption that ϕ(t), t/ϕ(t) are non-decreasing functions,

provided that

We derive the convergence rates of Tikhonov regularizer based on data-driven strategy of the parameter choice of λ for the class of probability measure ${{P}}_{\varphi ,b}$.

Theorem 3.3. Under the same assumptions of Theorem 3.2 and hypothesis (13), the convergence of the estimator fz, λ (7) to the target function f${H}$ can be described as:

(i) If ϕ(t) and $\sqrt{t}/\varphi \left(t\right)$ are non-decreasing functions. Then under the parameter choice λ ∈ (0, 1], λ = Ψ−1(m−1/2) where $\Psi \left(t\right)={t}^{\frac{1}{2}+\frac{1}{2b}}\varphi \left(t\right)$, we have

and

(ii) If ϕ(t) and t/ϕ(t) are non-decreasing functions. Then under the parameter choice λ ∈ (0, 1], λ = Θ−1(m−1/2) where $\Theta \left(t\right)={t}^{\frac{1}{2b}}\varphi \left(t\right)$, we have

$Probz{||fz,λ-fH||ρ≤C′ϕ(Θ-1(m-1/2))log(4η)}≥1-η$

and

Proof. (i) Let $\Psi \left(t\right)={t}^{\frac{1}{2}+\frac{1}{2b}}\varphi \left(t\right)$. Then it follows,

$limt→0Ψ(t)t=limt→0t2Ψ-1(t)=0.$

Under the parameter choice λ = Ψ−1(m−1/2) we have,

$limm→∞mλ=∞.$

Therefore, for sufficiently large m,

$1mλ=λ12bϕ(λ)mλ≤λ12bϕ(λ).$

Under the fact λ ≤ 1 from Theorem 3.2 and Equation (14) follows that with confidence 1 − η,

where $C=2R+4\kappa M+4\sqrt{\beta b{\Sigma }^{2}/\left(b-1\right)}$.

Now defining $\tau :=Clog\left(\frac{4}{\eta }\right)$ gives

$η=ητ=4e-τ/C.$

The estimate (29) can be reexpressed as

(ii) Suppose $\Theta \left(t\right)={t}^{\frac{1}{2b}}\varphi \left(t\right).$ Then the condition (28) follows that

$mλ≥8κ2log(4/η)λ≥8κ2λ.$

Hence under the parameter choice λ ∈ (0, 1], λ = Θ−1(m−1/2) we have

$1mλ≤λ8κ2m≤λ12+12bϕ(λ)8κ2≤ϕ(λ)8κ2.$

From Theorem 3.2 and Equation (14), it follows that with confidence 1 − η,

where ${C}^{\prime }:=R\left(\kappa +1\right)+M/2\kappa +4\sqrt{\beta b{\Sigma }^{2}/\left(b-1\right)}.$

Now defining $\tau :={C}^{\prime }log\left(\frac{4}{\eta }\right)$ gives

$η=ητ=4e-τ/C′.$

The estimate (31) can be reexpressed as

Then from Equations (30) and (32) our conclusions follow.

Theorem 3.4. Under the same assumptions of Theorem 3.1 and hypothesis (13) with the parameter choice λ ∈ (0, 1], λ = Ψ−1(m−1/2) where $\Psi \left(t\right)={t}^{\frac{1}{2}+\frac{1}{2b}}\varphi \left(t\right)$, the convergence of the estimator fz, λ (7) to the target function f${H}$ can be described as

$Probz{||fz,λ-fH||H≤Cϕ(Ψ-1(m-1/2))log(4η)}≥1-η$

and

The proof of the theorem follows the same steps as of Theorem 3.3 (i). We obtain the following corollary as a consequence of Theorem 3.3, 3.4.

Corollary 3.1. Under the same assumptions of Theorem 3.3, 3.4 for Tikhonov regularization with Hölder's source condition , for all 0 < η < 1, with confidence 1 − η, for the parameter choice $\lambda ={m}^{-\frac{b}{2br+b+1}}$, we have

and for the parameter choice $\lambda ={m}^{-\frac{b}{2br+1}}$, we have

### 3.2. Upper Rates for General Regularization Schemes

Bauer et al. [4] discussed the error estimates for general regularization schemes under general source condition. Here we study the convergence issues for general regularization schemes under general source condition and the polynomial decay of the eigenvalues of the integral operator LK. We define the regularization in learning theory framework similar to considered for ill-posed inverse problems (See Section 3.1 [4]).

Definition 3.1. A family of functions ${g}_{\lambda }:\left[0,{\kappa }^{2}\right]\to ℝ$, 0 < λ ≤ κ2, is said to be the regularization if it satisfies the following conditions:

$\exists D:\underset{\sigma \in \left(0,{\kappa }^{2}\right]}{sup}|\sigma {g}_{\lambda }\left(\sigma \right)|\le D$.

$\exists B:\underset{\sigma \in \left(0,{\kappa }^{2}\right]}{sup}|{g}_{\lambda }\left(\sigma \right)|\le \frac{B}{\lambda }$.

$\exists \gamma :\underset{\sigma \in \left(0,{\kappa }^{2}\right]}{sup}|1-{g}_{\lambda }\left(\sigma \right)\sigma |\le \gamma$.

The maximal p satisfying the condition:

$supσ∈(0,κ2]|1-gλ(σ)σ|σp≤γpλp$

is called the qualification of the regularization gλ, where γp does not depend on λ.

The properties of general regularization are satisfied by the large class of learning algorithms which are essentially all the linear regularization schemes. We refer to Section 2.2 [10] for brief discussion of the regularization schemes. Here we consider general regularized solution corresponding to the above regularization:

Here we are discussing the connection between the qualification of the regularization and general source condition [17].

Definition 3.2. The qualification p covers the index function ϕ if the function $t\to \frac{{t}^{p}}{\varphi \left(t\right)}$ on t ∈ (0, κ2] is non-decreasing.

The following result is a restatement of Proposition 3 [17].

Proposition 3.4. Suppose ϕ is a non-decreasing index function and the qualification of the regularization gλ covers ϕ. Then

Generally, the index function ϕ is not stable with respect to perturbation in the integral operator LK. In practice, we are only accessible to the perturbed empirical operator ${S}_{\text{x}}^{*}{S}_{\text{x}}$ but the source condition can be expressed in terms of LK only. So we want to control the difference $\varphi \left({L}_{K}\right)-\varphi \left({S}_{\text{x}}^{*}{S}_{\text{x}}\right)$. In order to obtain the error estimates for general regularization, we further restrict the index functions to operator monotone functions which is defined as

Definition 3.3. A function ϕ1:[0, d] → [0, ∞) is said to be operator monotone index function if ϕ1(0) = 0 and for every non-negative pair of self-adjoint operators A, B such that ||A||, ||B|| ≤ d and A ≤ B we have ϕ1(A) ≤ ϕ1(B).

We consider the class of operator monotone index functions:

For the above class of operator monotone functions from Theorem 1 [4], given ϕ1Fμ there exists cϕ1 such that

$||ϕ1(Sx*Sx)-ϕ1(LK)||L(H)≤cϕ1ϕ1(||Sx*Sx-LK||L(H)).$

Here we observe that the rate of convergence of ${\varphi }_{1}\left({S}_{\text{x}}^{*}{S}_{\text{x}}\right)$ to ϕ1(LK) is slower than the convergence rate of ${S}_{\text{x}}^{*}{S}_{\text{x}}$ to LK. Therefore, we consider the following class of index functions:

The splitting of ϕ = ϕ2ϕ1 is not unique. So we can take ϕ2 as a Lipschitz function with Lipschitz constant 1. Now using Corollary 1.2.2 [27] we get

$||ϕ2(Sx*Sx)-ϕ2(LK)||L2(H)≤||Sx*Sx-LK||L2(H).$

General source condition f${H}$ ∈ Ωϕ,R corresponding to index class functions ${F}$ covers wide range of source conditions as Hölder's source condition ϕ(t) = tr, logarithm source condition $\varphi \left(t\right)={t}^{p}{log}^{-\nu }\left(\frac{1}{t}\right)$. Following the analysis of Bauer et al. [4] we develop the error estimates of general regularization for the index class function ${F}$ under the suitable priors on the probability measure ρ.

Theorem 3.5. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi }$. Suppose fz is the regularized solution (33) corresponding to general regularization and the qualification of the regularization covers ϕ. Then for all 0 < η < 1, with confidence 1 − η, the following upper bound holds:

provided that

Proof. We consider the error expression for general regularized solution (33),

where rλ(σ) = 1 − gλ(σ)σ.

Now the first term can be expressed as

On applying RKHS-norm we get,

where ${I}_{2}=||{\left({L}_{K}+\lambda I\right)}^{-1/2}\left({S}_{\text{x}}^{*}\text{y}-{S}_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}}\right)|{|}_{{H}}$ and ${I}_{5}=||{\left({S}_{\text{x}}^{*}{S}_{\text{x}}+\lambda I\right)}^{-1/2}{\left({L}_{K}+\lambda I\right)}^{1/2}|{|}_{{L}\left({H}\right)}$.

The estimate of I2 can be obtained from the first estimate of Proposition 3.2 and from the second estimate of Proposition 3.2 with the condition (34) we obtain with probability 1 − η/2,

$||(LK+λI)-1/2(LK-Sx*Sx)(LK+λI)-1/2||L(H)≤1λ||Sx*Sx-LK||L(H)≤4κ2mλlog(4η)≤12.$

which implies that with confidence 1 − η/2,

From the properties of the regularization we have,

Hence it follows,

where ${\nu }_{1}:=B+\sqrt{BD}$.

Therefore, using (16), (37) and (39) in Equation (36) we conclude that with probability 1 − η,

Now we consider the second term,

Employing RKHS-norm we get

$||rλ(Sx*Sx)fH||H≤Rcgϕ(λ)+Rcgcϕ1ϕ2(λ)ϕ1(||LK-Sx*Sx||L(H))+Rμγ||LK-Sx*Sx||L2(H).$

Here we used the fact that if the qualification of the regularization covers ϕ = ϕ1ϕ2, then the qualification also covers ϕ1 and ϕ2 both separately.

From Equations (17) and (34) we have with probability 1 − η/2,

Therefore, with probability 1 − η/2,

Combining the bounds (40) and (42) we get the desired result.

□

Theorem 3.6. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi }$ and fz is the regularized solution (33) corresponding to general regularization. Then for all 0 < η < 1, with confidence 1 − η, the following upper bounds holds:

(i) If the qualification of the regularization covers ϕ,

$||fz,λ-fH||ρ≤{Rcg(1+cϕ1)(κ+λ)ϕ(λ)+4Rμγκ2(κ+λ)m+22ν2κMmλ+8ν22Σ2N(λ)m}log(4η),$

(ii) If the qualification of the regularization covers $\varphi \left(t\right)\sqrt{t}$,

provided that

Proof. Here we establish ${L}$2-norm estimate for the error expression:

$fz,λ-fH=gλ(Sx*Sx)(Sx*y-Sx*SxfH)-rλ(Sx*Sx)fH.$

On applying ${L}$2-norm in the first term we get,

where ${I}_{2}=||{\left({L}_{K}+\lambda I\right)}^{-1/2}\left({S}_{\text{x}}^{*}\text{y}-{S}_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}}\right)|{|}_{{H}}$ and ${I}_{5}=||{\left({S}_{\text{x}}^{*}{S}_{\text{x}}+\lambda I\right)}^{-1/2}{\left({L}_{K}+\lambda I\right)}^{1/2}|{|}_{{L}\left({H}\right)}$.

The estimates of I2 and I5 can be obtained from Proposition 3.2 and Theorem 3.5 respectively. Now we consider

$||LK1/2gλ(Sx∗Sx)(Sx∗Sx+λI)1/2||L(H)≤||LK1/2 −(Sx∗Sx)1/2||L(H)||gλ(Sx∗Sx)(Sx∗Sx+λI)1/2 ||L(H)+||(Sx∗Sx)1/2gλ(Sx∗Sx) (Sx∗Sx+λI)1/2||L(H).$

Since $\varphi \left(t\right)=\sqrt{t}$ is operator monotone function. Therefore, from Equation (41) with probability 1 − η/2, we get

$||LK1/2-(Sx*Sx)1/2||L(H)≤(||LK-Sx*Sx||L(H))1/2≤λ.$

Then using the properties of the regularization and Equation (38) we conclude that with probability 1 − η/2,

From Equations (44) with Equations (16), (37), and (45) we obtain with probability 1 − η,

The second term can be expressed as

Here two cases arises:

Case 1. If the qualification of the regularization covers ϕ. Then we get with confidence 1 − η/2,

$||rλ(Sx∗Sx)fH||ρ≤(κ+λ)(Rcg(1+cϕ1)ϕ(λ) +Rμγ||Sx∗Sx−LK||L2(H)).$

Therefore, using Equation (17) we obtain with probability 1 − η/2,

Case 2. If the qualification of the regularization covers $\varphi \left(t\right)\sqrt{t}$, we get with probability 1 − η/2,

Combining the error estimates (46), (47) and (48) we get the desired results.

□

We discuss the convergence rates of general regularizer based on data-driven strategy of the parameter choice of λ for the class of probability measure ${{P}}_{\varphi ,b}$. The proof of Theorem 3.7, 3.8 are similar to Theorem 3.3.

Theorem 3.7. Under the same assumptions of Theorem 3.5 and hypothesis (13) with the parameter choice λ ∈ (0, 1], λ = Ψ−1(m−1/2) where $\Psi \left(t\right)={t}^{\frac{1}{2}+\frac{1}{2b}}\varphi \left(t\right)$, the convergence of the estimator fz (33) to the target function f${H}$ can be described as

$Probz{||fz,λ-fH||H≤C~ϕ(Ψ-1(m-1/2))log(4η)}≥1-η,$

where $\stackrel{~}{C}=R{c}_{g}\left(1+{c}_{{\varphi }_{1}}\right)+4R\mu \gamma {\kappa }^{2}+2\sqrt{2}{\nu }_{1}\kappa M+\sqrt{8\beta b{\nu }_{1}^{2}{\Sigma }^{2}/\left(b-1\right)}$ and

Theorem 3.8. Under the same assumptions of Theorem 3.6 and hypothesis (13), the convergence of the estimator fz (33) to the target function f${H}$ can be described as

(i) If the qualification of the regularization covers ϕ. Then under the parameter choice λ ∈ (0, 1], λ = Θ−1(m−1/2) where $\Theta \left(t\right)={t}^{\frac{1}{2b}}\varphi \left(t\right)$, we have

$Probz{||fz,λ-fH||ρ≤C~1ϕ(Θ-1(m-1/2))log(4η)}≥1-η$

where ${\stackrel{~}{C}}_{1}=R{c}_{g}\left(1+{c}_{{\varphi }_{1}}\right)\left(\kappa +1\right)+4R\mu \gamma {\kappa }^{2}\left(\kappa +1\right)+{\nu }_{2}M/2\sqrt{2}\kappa +\sqrt{8\beta b{\nu }_{2}^{2}{\Sigma }^{2}/\left(b-1\right)}$ and

(ii) If the qualification of the regularization covers $\varphi \left(t\right)\sqrt{t}$. Then under the parameter choice λ ∈ (0, 1], λ = Ψ−1(m−1/2) where $\Psi \left(t\right)={t}^{\frac{1}{2}+\frac{1}{2b}}\varphi \left(t\right)$, we have

where ${\stackrel{~}{C}}_{2}=2R{c}_{g}\left(1+{c}_{{\varphi }_{1}}\right)+4R\mu \left(\gamma +{c}_{g}\right){\kappa }^{2}+2\sqrt{2}{\nu }_{2}\kappa M+\sqrt{8\beta b{\nu }_{2}^{2}{\Sigma }^{2}/\left(b-1\right)}$ and

$Probz{||fz,λ−fH||ρ ≤ C˜2(Ψ−1(m−1/2))1/2ϕ(Ψ−1(m−1/2)) log(4η)​ }≥ ​1−η,$

We obtain the following corollary as a consequence of Theorem 3.7, 3.8.

Corollary 3.2. Under the same assumptions of Theorem 3.7, 3.8 for general regularization of the qualification p with Hölder's source condition , for all 0 < η < 1, with confidence 1 − η, for the parameter choice $\lambda ={m}^{-\frac{b}{2br+b+1}}$, we have

and for the parameter choice $\lambda ={m}^{-\frac{b}{2br+1}}$, we have

Remark 3.1. It is important to observe from Corollary 3.1, 3.2 that using the concept of operator monotonicity of index function we are able to achieve the same error estimates for general regularization as of Tikhonov regularization up to a constant multiple.

Remark 3.2. (Related work) Corollary 3.1 provides the order of convergence same as of Theorem 1 [12] for Tikhonov regularization under the Hölder's source condition f${H}$ ∈ Ωϕ, R for and the polynomial decay of the eigenvalues (13). Blanchard and Mücke [18] addressed the convergence rates for inverse statistical learning problem for general regularization under the Hölder's source condition with the assumption fρ${H}$. In particular, the upper convergence rates discussed in Blanchard and Mücke [18] agree with Corollary 3.2 for considered learning problem which is referred as direct learning problem in Blanchard and Mücke[18]. Under the fact ${N}\left(\lambda \right)\le \frac{{\kappa }^{2}}{\lambda }$ from Theorem 3.5, 3.6 we obtain the similar estimates as of Theorem 10 [4] for general regularization schemes without the polynomial decay condition of the eigenvalues (13).

Remark 3.3. For the real valued functions and multi-task algorithms (the output space Y ⊂ ℝm for some m ∈ ℕ) we can obtain the error estimates from our analysis without imposing any condition on the conditional probability measure (11) for the bounded output space Y.

Remark 3.4. We can address the convergence issues of binary classification problem [28] using our error estimates as similar to discussed in Section 3.3 [4] and Section 5 [16].

### 3.3. Lower Rates for General Learning Algorithms

In this section, we discuss the estimates of minimum possible error over a subclass of the probability measures ${{P}}_{\varphi ,b}$ parameterized by suitable functions f${H}$. Throughout this section we assume that Y is finite-dimensional.

Let ${\left\{{v}_{j}\right\}}_{j=1}^{d}$ be a basis of Y and f ∈ Ωϕ, R. Then we parameterize the probability measure based on the function f,

where aj(x) = L − 〈f,Kxvj${H}$, bj(x) = L + 〈f,Kxvj${H}$, L = 4κϕ(κ2)R and δξ denotes the Dirac measure with unit mass at ξ. It is easy to observe that the marginal distribution of ρf over X is ν and the regression function for the probability measure ρf is f (see Proposition 4 [12]). In addition to this, for the conditional probability measure ρf(y|x) we have,

$∫Y(e||y-f(x)||Y/M-||y-f(x)||YM-1)dρf(y|x)≤(d2L2-||f(x)||Y2)∑i=2∞(dL+||f(x)||Y)i-2Mii!≤Σ22M2$

provided that

We assume that the eigenvalues of the integral operator LK follow the polynomial decay (13) for the marginal probability measure ν. Then we conclude that the probability measure ρf parameterized by f belongs to the class ${{P}}_{\varphi ,b}$.

The concept of information theory such as the Kullback-Leibler information and Fano inequalities (Lemma 3.3 [29]) are the main ingredients in the analysis of lower bounds. In the literature [12, 29], the closeness of probability measures is described through Kullback-Leibler information: Given two probability measures ρ1 and ρ2, it is defined as

$K(ρ1,ρ2):=∫Zlog(g(z))dρ1(z),$

where g is the density of ρ1 with respect to ρ2, that is, ${\rho }_{1}\left(E\right)={\int }_{E}g\left(z\right)d{\rho }_{2}\left(z\right)$ for all measurable sets E.

Following the analysis of Caponnetto and De Vito [12] and DeVore et al. [29] we establish the lower rates of accuracy that can be attained by any learning algorithm.

To estimate the lower rates of learning algorithms, we generate Nε-functions belonging to Ωϕ,R for given ε > 0 such that (53), (54) holds. Then we construct the probability measures ${\rho }_{i}\in {{P}}_{\varphi ,b}$ from Equation (49), parameterized by these functions fi's (1 ≤ iNε). On applying Lemma 3.3 [29], we obtain the lower convergence rates using Kullback-Leibler information.

Theorem 3.9. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi ,b}$ under the hypothesis dim(Y) = d < ∞. Then for any learning algorithm (zfz${H}$) there exists a probability measure ${\rho }_{*}\in {{P}}_{\varphi ,b}$ and fρ*${H}$ such that for all 0 < ε < εo, fz can be approximated as

$Probz{||fz-fρ*||H>ε/2}≥min{11+e-ℓε/24,ϑe(ℓε48-cmε2ℓεb)}$

where ϑ = e−3/e, $c=\frac{64\beta }{15\left(b-1\right)d{L}^{2}}\left(1-\frac{1}{{2}^{b-1}}\right)$ and ${\ell }_{\epsilon }=⌊\frac{1}{2}{\left(\frac{\alpha }{{\varphi }^{-1}\left(\epsilon /R\right)}\right)}^{1/b}⌋$.

Proof. For given ε > 0, we define

$g=∑n=ℓ+12ℓεσn-ℓenℓϕ(tn),$

where σ = (σ1, …, σ) ∈ {−1, +1}, tn's are the eigenvalues of the integral operator LK, en's are the eigenvectors of the integral operator LK and the orthonormal basis of RKHS ${H}$. Under the decay condition on the eigenvalues $\alpha \le {n}^{b}{t}_{n}$, we get

$||g||H2=∑n=ℓ+12ℓε2ℓϕ2(tn)≤∑n=ℓ+12ℓε2ℓϕ2(αnb)≤ε2ϕ2(α2bℓb).$

Hence f = ϕ(LK)g ∈ Ωϕ,R provided that ||g||${H}$R or equivalently,

For ℓ = ${\ell }_{\epsilon }=⌊\frac{1}{2}{\left(\frac{\alpha }{{\varphi }^{-1}\left(\epsilon /R\right)}\right)}^{1/b}⌋$, choose εo such that ℓεo > 16. Then according to Proposition 6 [12], for every positive ε < εo (ℓε > ℓεo) there exists Nε ∈ ℕ and ${\sigma }_{1},\dots ,{\sigma }_{{N}_{\epsilon }}\in {\left\{-1,+1\right\}}^{{\ell }_{\epsilon }}$ such that

and

Now we suppose fi = ϕ(LK)gi and for ε > 0,

where ${\sigma }_{i}=\left({\sigma }_{i}^{1},\dots ,{\sigma }_{i}^{{\ell }_{\epsilon }}\right)\in {\left\{-1,+1\right\}}^{{\ell }_{\epsilon }}$. Then from Equation (51) we get,

For 1 ≤ i, jNε, we have

where ${c}^{\prime }=\frac{4\beta }{\left(b-1\right)}\left(1-\frac{1}{{2}^{b-1}}\right)$.

We define the sets,

It is clear from Equation (53) that Ai's are disjoint sets. On applying Lemma 3.3 [29] with probability measures , we obtain that either

or

where ${\text{Ψ}}_{{N}_{\epsilon }}\left(p\right)=log\left({N}_{\epsilon }\right)+\left(1-p\right)log\left(\frac{1-p}{p}\right)-plog\left(\frac{{N}_{\epsilon }-p}{p}\right)$. Further,

Since minimum value of x log(x) is −1/e on [0, 1].

For the joint probability measures ${\rho }_{{f}_{i}}^{m}$, ${\rho }_{{f}_{j}}^{m}$ from Proposition 4 [12] and the Equation (54) we get,

where c = 16c′/15dL2.

Therefore, Equations (55), (56), together with Equations (57) and (58) implies

From Equation (52) for the probability measure ρ* such that $p={\rho }_{*}^{m}\left({A}_{i}^{c}\right)$ follows the result.                                                                           □

The lower estimates in ${L}$2-norm can be obtained similar to above theorem.

Theorem 3.10. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi ,b}$ under the hypothesis dim(Y) = d < ∞. Then for any learning algorithm (zfz${H}$) there exists a probability measure ${\rho }_{*}\in {{P}}_{\varphi ,b}$ and fρ*${H}$ such that for all 0 < ε < εo, fz can be approximated as

$Probz{||fz-fρ*||Lν2(X)>ε/2}≥min{11+e-ℓε/24,ϑe(ℓε48-64mε215dL2)}$

where ϑ = e−3/e, ${\ell }_{\epsilon }=⌊{\left(\frac{\alpha }{{\Psi }^{-1}\left(\epsilon /R\right)}\right)}^{1/b}⌋$ and $\psi \left(t\right)=\sqrt{t}\varphi \left(t\right)$.

Theorem 3.11. Under the same assumptions of Theorem 3.10 for ψ(t) = t1/2ϕ(t) and $\text{Ψ}\left(t\right)={t}^{\frac{1}{2}+\frac{1}{2b}}\varphi \left(t\right)$, the estimator fz corresponding to any learning algorithm converges to the regression function fρ with the following lower rate:

where ${A}$ denotes the set of all learning algorithms $l:\text{z}\to {f}_{\text{z}}^{l}$.

Proof. Under the condition ${\ell }_{\epsilon }=⌊{\left(\frac{\alpha }{{\Psi }^{-1}\left(\epsilon /R\right)}\right)}^{1/b}⌋$ from Theorem 3.10 we get,

Choosing ${\epsilon }_{m}=\tau R\psi \left({\text{Ψ}}^{-1}\left({m}^{-1/2}\right)\right)$, we obtain

where $c=\left(\frac{{\alpha }^{1/b}}{48}-\frac{64{R}^{2}{\tau }^{2}}{15d{L}^{2}}\right)>0$ for $0<\tau .

Now as m goes to ∞, ε → 0 and ℓε → ∞. Therefore, for c > 0 we conclude that

□

Choosing ${\epsilon }_{m}=\tau R\varphi \left({\text{Ψ}}^{-1}\left({m}^{-1/2}\right)\right)$ we get the following convergence rate from Theorem 3.9.

Theorem 3.12. Under the same assumptions of Theorem 3.10 for $\Psi \left(t\right)={t}^{\frac{1}{2}+\frac{1}{2b}}\varphi \left(t\right)$, the estimator fz corresponding to any learning algorithm converges to the regression function fρ with the following lower rate:

We obtain the following corollary as a consequence of Theorem 3.11, 3.12.

Corollary 3.3. For any learning algorithm under Hölder's source condition and the polynomial decay condition (13) for b > 1, the lower convergence rates can be described as

and

If the minimax lower rate coincides with the upper convergence rate for λ = λm. Then the choice of parameter is said to be optimal. For the parameter choice λ = Ψ−1(m−1/2), Theorem 3.3 and Theorem 3.8 share the upper convergence rate with the lower convergence rate of Theorem 3.11 in ${L}$2-norm. For the same parameter choice, Theorem 3.4 and Theorem 3.7 share the upper convergence rate with the lower convergence rate of Theorem 3.12 in RKHS-norm. Therefore, the choice of the parameter is optimal.

It is important to observe that we get the same convergence rates for b = 1.

### 3.4. Individual Lower Rates

In this section, we discuss the individual minimax lower rates that describe the behavior of the error for the class of probability measure ${{P}}_{\varphi ,b}$ as the sample size m grows.

Definition 3.4. A sequence of positive numbers an (n ∈ ℕ) is called the individual lower rate of convergence for the class of probability measure ${P}$, if

where ${A}$ denotes the set of all learning algorithms $l:\text{z}↦{f}_{\text{z}}^{l}$.

Theorem 3.13. Let z be i.i.d. samples drawn according to the probability measure ${{P}}_{\varphi ,b}$ where ϕ is the index function satisfying the conditions that $\varphi \left(t\right)/{t}^{{r}_{1}}$, ${t}^{{r}_{2}}/\varphi \left(t\right)$ are non-decreasing functions and dim(Y) = d < ∞. Then for every ε > 0, the following lower bound holds:

where c1 = 2r1 + 1 and c2 = 2r2 + 1.

We consider the class of probability measures such that the target function f${H}$ is parameterized by $\text{s}={\left({s}_{n}\right)}_{n=1}^{\infty }\in {\left\{-1,+1\right\}}^{\infty }$. Suppose for ε > 0,

$g=∑n=1∞snRεε+1αnbtn(ϕ(α/nb)ϕ(tn))n-(ε+1)/2en,$

where $\text{s}={\left({s}_{n}\right)}_{n=1}^{\infty }\in {\left\{-1,+1\right\}}^{\infty }$, tn's are the eigenvalues of the integral operator LK, en's are the eigenvectors of the integral operator LK and the orthonormal basis of RKHS ${H}$. Then the target function f${H}$ = ϕ(LK)g satisfies the general source condition. We assume that the conditional probability measure ρ(y|x) follows the normal distribution centered at f${H}$ and the marginal probability measure ρX = ν. Now we can derive the individual lower rates over the considered class of probability measures from the ideas of the literature [12, 30].

Theorem 3.14. Let z be i.i.d. samples drawn according to the probability measure ${{P}}_{\varphi ,b}$ where ϕ is the index function satisfying the conditions that $\varphi \left(t\right)/{t}^{{r}_{1}}$, ${t}^{{r}_{2}}/\varphi \left(t\right)$ are non-decreasing functions and dim(Y) = d < ∞. Then for every ε > 0, the following lower bound holds:

## 4. Conclusion

In our analysis we derive the upper and lower convergence rates over the wide class of probability measures considering general source condition in vector-valued setting. In particular, our minimax rates can be used for the scalar-valued functions and multi-task learning problems. The lower convergence rates coincide with the upper convergence rates for the optimal parameter choice based on smoothness parameters b, ϕ. We can also develop various parameter choice rules such as balancing principle [31], quasi-optimality principle [32], discrepancy principle [33] for the regularized solutions provided in our analysis.

## Author Contributions

All authors listed, have made substantial, direct and intellectual contribution to the work, and approved it for publication.

## Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

## Acknowledgments

The authors are grateful to the reviewers for their helpful comments and pointing out a subtle error that led to improve the quality of the paper.

## References

1. Cucker F, Smale, S. On the mathematical foundations of learning. Bull Am Math Soc. (2002) 39:1–49. doi: 10.1090/S0273-0979-01-00923-5

2. Evgeniou T, Pontil, M Poggio, T. Regularization networks and support vector machines. Adv Comput Math. (2000) 13:1–50. doi: 10.1023/A:1018946025316

3. Vapnik VN, Vapnik V. Statistical Learning Theory. New York, NY: Wiley (1998).

4. Bauer F, Pereverzev S, Rosasco L. On regularization algorithms in learning theory. J Complex. (2007) 23:52–72. doi: 10.1016/j.jco.2006.07.001

5. Engl HW, Hanke M, Neubauer A Regularization of Inverse Problems. Dordrecht: Kluwer Academic Publishers Group (1996).

6. Gerfo LL, Rosasco L, Odone F, De Vito E, Verri A. Spectral algorithms for supervised learning. Neural Comput. (2008) 20:1873–97. doi: 10.1162/neco.2008.05-07-517

7. Tikhonov AN, Arsenin VY. Solutions of Ill-Posed Problems. Washington, DC: W. H. Winston (1977).

8. Bousquet O, Boucheron S, Lugosi G. Introduction to statistical learning theory. In: Bousquet O, von Luxburg U, Ratsch G editors. Advanced Lectures on Machine Learning, Volume 3176 of Lecture Notes in Computer Science. Berlin; Heidelberg: Springer (2004). pp. 169–207.

9. Cucker F, Zhou DX. Learning Theory: An Approximation Theory Viewpoint. Cambridge, UK: Cambridge Monographs on Applied and Computational Mathematics, Cambridge University Press (2007).

10. Lu S, Pereverzev S. Regularization Theory for Ill-posed Problems: Selected Topics, Berlin: DeGruyter (2013).

11. Abhishake Sivananthan S. Multi-penalty regularization in learning theory. J Complex. (2016) 36:141–65. doi: 10.1016/j.jco.2016.05.003

12. Caponnetto A, De Vito E. Optimal rates for the regularized least-squares algorithm. Found Comput Math. (2007) 7:331–68. doi: 10.1007/s10208-006-0196-8

13. Smale S, Zhou DX. Estimating the approximation error in learning theory. Anal Appl. (2003) 1:17–41. doi: 10.1142/S0219530503000089

14. Smale S, Zhou DX. Shannon sampling and function reconstruction from point values. Bull Am Math Soc. (2004) 41:279–306. doi: 10.1090/S0273-0979-04-01025-0

15. Smale S, Zhou DX. Shannon sampling II: connections to learning theory. Appl Comput Harmon Anal. (2005) 19:285–302. doi: 10.1016/j.acha.2005.03.001

16. Smale S, Zhou DX. Learning theory estimates via integral operators and their approximations. Constr Approx. (2007) 26:153–72. doi: 10.1007/s00365-006-0659-y

17. Mathé P, Pereverzev SV. Geometry of linear ill-posed problems in variable Hilbert scales. Inverse Probl. (2003) 19:789–803. doi: 10.1088/0266-5611/19/3/319

18. Blanchard G, Mücke N. Optimal rates for regularization of statistical inverse learning problems. arXiv:1604.04054 (2016).

19. Mendelson S. On the performance of kernel classes. J Mach Learn Res. (2003) 4:759–71.

20. Zhang T. Effective dimension and generalization of kernel learning. In: Thrun S, Becker S, Obermayer K. editors. Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, (2003). pp. 454–61.

21. Akhiezer NI, Glazman IM. Theory of Linear Operators in Hilbert Space, Translated from the Russian and with a preface by Merlynd Nestell. New York, NY: Dover Publications Inc (1993).

22. Micchelli CA, Pontil M. On learning vector-valued functions. Neural Comput. (2005) 17:177–204. doi: 10.1162/0899766052530802

23. Aronszajn N. Theory of reproducing kernels. Trans Am Math Soc. (1950) 68:337–404. doi: 10.1090/S0002-9947-1950-0051437-7

24. Reed M, Simon B. Functional Analysis, Vol. 1, San Diego, CA: Academic Press (1980).

25. De Vito E, Rosasco L, Caponnetto A, De Giovannini U, Odone F. Learning from examples as an inverse problem. J Mach Learn Res. (2005) 6:883–904.

26. Pinelis IF, Sakhanenko AI. Remarks on inequalities for the probabilities of large deviations. Theory Prob Appl. (1985) 30:127–31. doi: 10.1137/1130013

27. Peller VV. Multiple operator integrals in perturbation theory. Bull Math Sci. (2016) 6:15–88. doi: 10.1007/s13373-015-0073-y

28. Boucheron S, Bousquet O, Lugosi G. Theory of classification: a survey of some recent advances. ESAIM: Prob Stat. (2005) 9:323–75. doi: 10.1051/ps:2005018

29. DeVore R, Kerkyacharian G, Picard D, Temlyakov V. Approximation methods for supervised learning. Found Comput Math. (2006) 6:3–58. doi: 10.1007/s10208-004-0158-6

30. Györfi L, Kohler M, Krzyzak A, Walk H. A Distribution-Free Theory of Nonparametric Regression. New York, NY: Springer Series in Statistics, Springer-Verlag (2002).

31. De Vito E, Pereverzyev S, Rosasco L. Adaptive kernel methods using the balancing principle. Found Comput Math. (2010) 10:455–79. doi: 10.1007/s10208-010-9064-2

32. Bauer F, Reiss M. Regularization independent of the noise level: an analysis of quasi-optimality. Inverse Prob. (2008) 24:055009. doi: 10.1088/0266-5611/24/5/055009

33. Lu S, Pereverzev SV, Tautenhahn U. A model function method in regularized total least squares. Appl Anal. (2010) 89:1693–703. doi: 10.1080/00036811.2010.492502

Keywords: learning theory, general source condition, vector-valued RKHS, error estimate, optimal rates

Mathematics Subject Classification 2010: 68T05, 68Q32

Citation: Rastogi A and Sampath S (2017) Optimal Rates for the Regularized Learning Algorithms under General Source Condition. Front. Appl. Math. Stat. 3:3. doi: 10.3389/fams.2017.00003

Received: 02 November 2016; Accepted: 09 March 2017;
Published: 27 March 2017.

Edited by:

Yiming Ying, University at Albany, SUNY, USA

Reviewed by:

Xin Guo, The Hong Kong Polytechnic University, Hong Kong
Ernesto De Vito, University of Genoa, Italy

Copyright © 2017 Rastogi and Sampath. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Abhishake Rastogi, abhishekrastogi2012@gmail.com