Definition 2.1. (Vector-valued reproducing kernel Hilbert space) For non-empty set X and the real Hilbert space (Y, 〈·, ·〉_Y), the Hilbert space (H, 〈·, ·〉_H) of functions from X to Y is called reproducing kernel Hilbert space if for any x ∈ X and y ∈ Y the linear functional which maps f ∈ H to 〈y, f(x)〉_Y is continuous.

By Riesz lemma [21], for every x ∈ X and y ∈ Y there exists a linear operator K_x : Y → H such that

Therefore, the adjoint operator Kx*:H→Y is given by Kx*f=f(x). Through the linear operator K_x:Y → H we define the linear operator K(x, t) : Y → Y,

From Proposition 2.1 [22], the linear operator K(x,t)∈L(Y) (the set of bounded linear operators on Y), K(x, t) = K(t, x)^* and K(x, x) is non-negative bounded linear operator. For any m ∈ ℕ, {x_i: 1 ≤ i ≤ m} ∈ X, {y_i: 1 ≤ i ≤ m} ∈ Y, we have that ∑i,j=1m〈yi,K(xi,xj)yj〉≥0. The operator valued function K:X×X→L(Y) is called the kernel.

There is one to one correspondence between the kernels and reproducing kernel Hilbert spaces [22, 23]. So a reproducing kernel Hilbert space H corresponding to a kernel K can be denoted as H_K and the norm in the space H can be denoted as ||·||_HK. In the following article, we suppress K by simply using H for reproducing kernel Hilbert space and ||·||_H for its norm.

Throughout the paper we assume the reproducing kernel Hilbert space H is separable such that

By the representation theorem [22], the solution of the penalized regularization problem (5) will be of the form:

Definition 2.2. let H be a separable Hilbert space and {ek}k=1∞ be an orthonormal basis of H. Then for any positive operator A∈L(H) we define Tr(A)=∑k=1∞〈Aek,ek〉. It is well-known that the number Tr(A) is independent of the choice of the orthonormal basis.

Definition 2.3. An operator A∈L(H) is called Hilbert-Schmidt operator if Tr(A^*A) < ∞. The family of all Hilbert-Schmidt operators is denoted by L2(H). For A∈L2(H), we define Tr(A)=∑k=1∞〈Aek,ek〉 for an orthonormal basis {ek}k=1∞ of H.

It is well-known that L2(H) is the separable Hilbert space with the inner product,

and its norm satisfies

where |A|=A*A and ||·||L(H) is the operator norm (For more details see [24]).

For the positive trace class operator KxKx*, we have

Given the ordered set x=(x1,…,xm)∈Xm, the sampling operator Sx:H→Ym is defined by S_x(f) = (f(x₁), …, f(x_m)) and its adjoint Sx*:Ym→H is given by Sx*y=1m∑i=1mKxiyi, ∀ y=(y1,…,ym)∈Ym.

The regularization scheme (5) can be expressed as

where ||y||m2=1m∑i=1m||yi||Y2.

We obtain the explicit expression of f_z,λ by taking the functional derivative of above expression over RKHS H.

Theorem 2.1. For the positive choice of λ, the functional (6) has unique minimizer:

Define f_λ as the minimizer of the optimization functional,

Using the fact E(f)=||LK1/2(f-fH)||H2+E(fH), we get the expression of f_λ,

where the integral operator LK:LρX2→LρX2 is a self-adjoint, non-negative, compact operator, defined as

The integral operator L_K can also be defined as a self-adjoint operator on H. We use the same notation L_K for both the operators defined on different domains. It is well-known that LK1/2 is an isometry from the space of square integrable functions to reproducing kernel Hilbert space.

In order to achieve the uniform convergence rates for learning algorithms we need some prior assumptions on the probability measure ρ. Following the notion of Bauer et al. [4] and Caponnetto and De Vito [12], we consider the class of probability measures Pϕ which satisfies the assumptions:

In addition, we consider the set of probability measures Pϕ,b which satisfies the conditions (i), (ii), (iii), (iv) and the eigenvalues t_n's of the integral operator L_K follow the polynomial decay: For fixed positive constants α, β and b > 1,

Under the polynomial decay of the eigenvalues the effective dimension N(λ), to measure the complexity of RKHS, can be estimated from Proposition 3 [12] as follows,

and without the polynomial decay condition (13), we have

We discuss the convergence issues for the learning algorithms (z → f_z ∈ H) in probabilistic sense by exponential tail inequalities such that

for all 0 < η ≤ 1 and ε(m) is a positive decreasing function of m. Using these probabilistic estimates we can obtain error estimates in expectation by integration of tail inequalities:

where ||f||ρ=||f||LρX2={∫X||f(x)||Y2dρX(x)}1/2 and Ez(ξ)=∫Zmξdρ(z1)…dρ(zm).

In General, we consider Tikhonov regularization in learning theory. Tikhonov regularization is briefly discussed in the literature [7, 9, 10, 25]. The error estimates for Tikhonov regularization are discussed theoretically under Hölder's source condition [12, 15, 16]. We establish the error estimates for Tikhonov regularization scheme under general source condition f_H ∈ Ω_ϕ,R for some continuous increasing index function ϕ and the polynomial decay of the eigenvalues of the integral operator L_K.

In order to estimate the error bounds, we consider the following inequality used in the papers [4, 12] which is based on the results of Pinelis and Sakhanenko [26].

Proposition 3.1. Let ξ be a random variable on the probability space (Ω,B,P) with values in real separable Hilbert space H. If there exist two constants Q and S satisfying

then for any 0 < η < 1 and for all m ∈ ℕ,

In particular, the inequality (15) holds if

We estimate the error bounds for the regularized estimators by measuring the effect of random sampling and the complexity of f_H. The quantities described in Proposition 3.2 express the probabilistic estimates of the perturbation measure due to random sampling. The expressions of Proposition 3.3 describe the complexity of the target function f_H which are usually referred to as the approximation errors. The approximation errors are independent of the samples z.

Proposition 3.2. Let z be i.i.d. samples drawn according to the probability measure ρ satisfying the assumptions (10), (11) and κ=supx∈XTr(Kx*Kx). Then for all 0 < η < 1, we have

and

with the confidence 1 − η.

The proof of the first expression is the content of the step 3.2 of Theorem 4 [12] while the proof of the second expression can be obtained from Theorem 2 in De Vito et al. [25].

Proposition 3.3. Suppose f_H ∈ Ω_ϕ,R. Then,

Under the source condition f_H ∈ Ω_{ϕ, R}, the proposition can be proved using the ideas of Theorem 10 [4].

Theorem 3.1. Let z be i.i.d. samples drawn according to the probability measure ρ∈Pϕ where ϕ is the index function satisfying the conditions that ϕ(t), t/ϕ(t) are non-decreasing functions. Then for all 0 < η < 1, with confidence 1 − η, for the regularized estimator f_z,λ (7) the following upper bound holds:

provided that

Proof. The error of regularized solution f_z,λ can be estimated in terms of the sample error and the approximation error as follows:

Now f_z,λ − f_λ can be expressed as

Then fλ=(LK+λI)-1LKfH implies

Therefore,

Employing RKHS-norm we get,

where I1=||(Sx*Sx+λI)-1(LK+λI)1/2||L(H), I2=||(LK+λI)-1/2(Sx*y-Sx*SxfH)||H and I3=||Sx*Sx-LK||L(H).

The estimates of I₂, I₃ can be obtained from Proposition 3.2 and the only task is to bound I₁. For this we consider

which implies

provided that ||(LK+λI)-1(LK-Sx*Sx)||L(H)<1. To verify this condition, we consider

Now using Proposition 3.2 we get with confidence 1 − η/2,

From the condition (21) we get with confidence 1 − η/2,

Consequently, using (25) in the inequality (24) we obtain with probability 1 − η/2,

From Proposition 3.2 we have with confidence 1 − η/2,

Again from the condition (21) we get with probability 1 − η/2,

Therefore, the inequality (23) together with (16), (20), (26), (27) provides the desired bound.

                                                                           □

The following theorem discuss the error estimates in L²-norm. The proof is similar to the above theorem.

Theorem 3.2. Let z be i.i.d. samples drawn according to the probability measure ρ∈Pϕ and f_z,λ is the regularized solution (7) corresponding to Tikhonov regularization. Then for all 0 < η < 1, with confidence 1 − η, the following upper bounds holds:

provided that

We derive the convergence rates of Tikhonov regularizer based on data-driven strategy of the parameter choice of λ for the class of probability measure Pϕ,b.

Theorem 3.3. Under the same assumptions of Theorem 3.2 and hypothesis (13), the convergence of the estimator f_{z, λ} (7) to the target function f_H can be described as:

Proof. (i) Let Ψ(t)=t12+12bϕ(t). Then it follows,

Under the parameter choice λ = Ψ⁻¹(m^−1/2) we have,

Therefore, for sufficiently large m,

Under the fact λ ≤ 1 from Theorem 3.2 and Equation (14) follows that with confidence 1 − η,

where C=2R+4κM+4βbΣ2/(b-1).

Now defining τ:=Clog(4η) gives

The estimate (29) can be reexpressed as

(ii) Suppose Θ(t)=t12bϕ(t). Then the condition (28) follows that

Hence under the parameter choice λ ∈ (0, 1], λ = Θ⁻¹(m^−1/2) we have

From Theorem 3.2 and Equation (14), it follows that with confidence 1 − η,

where C′:=R(κ+1)+M/2κ+4βbΣ2/(b-1).

Now defining τ:=C′log(4η) gives

The estimate (31) can be reexpressed as

Then from Equations (30) and (32) our conclusions follow.

                                         □

Theorem 3.4. Under the same assumptions of Theorem 3.1 and hypothesis (13) with the parameter choice λ ∈ (0, 1], λ = Ψ⁻¹(m^−1/2) where Ψ(t)=t12+12bϕ(t), the convergence of the estimator f_z,λ (7) to the target function f_H can be described as

and

The proof of the theorem follows the same steps as of Theorem 3.3 (i). We obtain the following corollary as a consequence of Theorem 3.3, 3.4.

Corollary 3.1. Under the same assumptions of Theorem 3.3, 3.4 for Tikhonov regularization with Hölder's source condition fH∈Ωϕ,R, ϕ(t)=tr, for all 0 < η < 1, with confidence 1 − η, for the parameter choice λ=m-b2br+b+1, we have

and for the parameter choice λ=m-b2br+1, we have

Bauer et al. [4] discussed the error estimates for general regularization schemes under general source condition. Here we study the convergence issues for general regularization schemes under general source condition and the polynomial decay of the eigenvalues of the integral operator L_K. We define the regularization in learning theory framework similar to considered for ill-posed inverse problems (See Section 3.1 [4]).

Definition 3.1. A family of functions gλ:[0,κ2]→ℝ, 0 < λ ≤ κ², is said to be the regularization if it satisfies the following conditions:

The properties of general regularization are satisfied by the large class of learning algorithms which are essentially all the linear regularization schemes. We refer to Section 2.2 [10] for brief discussion of the regularization schemes. Here we consider general regularized solution corresponding to the above regularization:

Here we are discussing the connection between the qualification of the regularization and general source condition [17].

Definition 3.2. The qualification p covers the index function ϕ if the function t→tpϕ(t) on t ∈ (0, κ²] is non-decreasing.

The following result is a restatement of Proposition 3 [17].

Proposition 3.4. Suppose ϕ is a non-decreasing index function and the qualification of the regularization g_λ covers ϕ. Then

Generally, the index function ϕ is not stable with respect to perturbation in the integral operator L_K. In practice, we are only accessible to the perturbed empirical operator Sx*Sx but the source condition can be expressed in terms of L_K only. So we want to control the difference ϕ(LK)-ϕ(Sx*Sx). In order to obtain the error estimates for general regularization, we further restrict the index functions to operator monotone functions which is defined as

Definition 3.3. A function ϕ₁:[0, d] → [0, ∞) is said to be operator monotone index function if ϕ₁(0) = 0 and for every non-negative pair of self-adjoint operators A, B such that ||A||, ||B|| ≤ d and A ≤ B we have ϕ₁(A) ≤ ϕ₁(B).

We consider the class of operator monotone index functions:

For the above class of operator monotone functions from Theorem 1 [4], given ϕ₁ ∈ F_μ there exists c_ϕ1 such that

Here we observe that the rate of convergence of ϕ1(Sx*Sx) to ϕ₁(L_K) is slower than the convergence rate of Sx*Sx to L_K. Therefore, we consider the following class of index functions:

The splitting of ϕ = ϕ₂ϕ₁ is not unique. So we can take ϕ₂ as a Lipschitz function with Lipschitz constant 1. Now using Corollary 1.2.2 [27] we get

General source condition f_H ∈ Ω_{ϕ, R} corresponding to index class functions F covers wide range of source conditions as Hölder's source condition ϕ(t) = t^r, logarithm source condition ϕ(t)=tplog-ν(1t). Following the analysis of Bauer et al. [4] we develop the error estimates of general regularization for the index class function F under the suitable priors on the probability measure ρ.

Theorem 3.5. Let z be i.i.d. samples drawn according to the probability measure ρ∈Pϕ. Suppose f_z,λ is the regularized solution (33) corresponding to general regularization and the qualification of the regularization covers ϕ. Then for all 0 < η < 1, with confidence 1 − η, the following upper bound holds: