Optimal rates for the regularized learning algorithms under general source condition

We consider the learning algorithms under general source condition with the polynomial decay of the eigenvalues of the integral operator in vector-valued function setting. We discuss the upper convergence rates of Tikhonov regularizer under general source condition corresponding to increasing monotone index function. The convergence issues are studied for general regularization schemes by using the concept of operator monotone index functions in minimax setting. Further we also address the minimum possible error for any learning algorithm.


Introduction
Learning theory [10,16,34] aims to learn the relation between the inputs and outputs based on finite random samples.We require some underlying space to search the relation function.From the experiences we have some idea about the underlying space which is called hypothesis space.Learning algorithms tries to infer the best estimator over the hypothesis space such that f (x) gives the maximum information of the output variable y for any unseen input x.The given samples {x i , y i } m i=1 are not exact in the sense that for underlying relation function f (x i ) = y i but f (x i ) ≈ y i .We assume that the uncertainty follows the probability distribution ρ on the sample space X × Y and the underlying function (called the regression function) for the probability distribution ρ is given by where ρ(y|x) is the conditional probability measure for given x.The problem of obtaining estimator from examples is ill-posed.Therefore we apply the regularization schemes [4,15,17,33] to stabilize the problem.Various regularization schemes are studied for inverse problems.In the context of learning theory [8,11,16,22,34], the square loss-regularization (Tikhonov regularization) is widely considered to obtain the regularized estimator [9,11,29,30,31,32].Rosasco et al. [17] introduced general regularization in the learning theory and provided the error bounds under Hölder's source condition [15].Bauer et al. [4] discussed the convergence issues for general regularization under general source condition [23] by removing the Lipschitz condition on the regularization considered in [17].Caponnetto et al. [9] discussed the squareloss regularization under the polynomial decay of the eigenvalues of the integral operator L K with Hölder's source condition.Here we are discussing the convergence issues of general regularization schemes under general source condition and the polynomial decay of the eigenvalues of the integral operator.We present the minimax upper convergence rates for Tikhonov regularization under general source condition Ω φ,R , for a monotone increasing index function φ.For general regularization the minimax rates are obtained using the operator monotone index function φ.The concept of effective dimension [24,35] is exploited to achieve the convergence rates.In the choice of regularization parameters, the effective dimension plays the important role.We also discuss the lower convergence rates for any learning algorithm under the smoothness conditions.We present the results in vector-values function setting.Therefore in particular they can be applied to multi-task learning problems.
The structure of the paper is as follows.In the second section, we introduce some basic assumptions and notations for supervised learning problems.In Section 3, we present the upper and lower convergence rates under the smoothness conditions in minimax setting.

Learning from examples: Notations and assumptions
In the learning theory framework [8,11,16,22,34], the sample space Z = X × Y consists of two spaces: The input space X (locally compact second countable Hausdorff space) and the output space (Y, •, • Y ) (the real Hilbert space).The input space X and the output space Y are related by some unknown probability distribution ρ on Z.The probability measure can be split as ρ(x, y) = ρ(y|x)ρ X (x), where ρ(y|x) is the conditional probability measure of y given x and ρ X is the marginal probability measure on X.The only available information is the random i.i.d.samples z = ((x 1 , y 1 ), . . ., (x m , y m )) drawn according to the probability measure ρ.Given the training set z, learning theory aims to develop an algorithm which provides an estimator f z : X → Y such that f z (x) predicts the output variable y for any given input x.The goodness of the estimator can be measured by the generalization error of a function f which can be defined as where V : Y × Y → R is the loss function.The minimizer of E(f ) for the square loss function V (f (x), y) = ||f (x) − y|| 2 Y is given by where f ρ is called the regression function.The regression function f ρ belongs to the space of square integrable functions provided that We search the minimizer of the generalization error over a hypothesis space H, where f H is called the target function.In case f ρ ∈ H, f H becomes the regression function f ρ .Because of inaccessibility of the probability distribution ρ, we minimize the regularized empirical esti-mate of the generalization error over the hypothesis space H, where λ is the positive regularization parameter.The regularization schemes [4,15,17,22,33] are used to incorporate various features in the solution such as boundedness, monotonicity and smoothness.In order to optimize the vector-valued regularization functional, one of the main problems is to choose the appropriate hypothesis space which is assumed to be a source to provide the estimator.
2.1 Reproducing kernel Hilbert space as a hypothesis space By Riesz lemma [2], for every x ∈ X and y ∈ Y there exists a linear operator Therefore the adjoint operator From Proposition 2.1 [25], the linear operator K(x, t) ∈ L(Y ) (the set of bounded linear operators on Y ), K(x, t) = K(t, x) * and K(x, x) is nonnegative bounded linear operator.For any m ∈ N, y i , K(x i , x j )y j ≥ 0. The operator valued function There is one to one correspondence between the kernels and reproducing kernel Hilbert spaces [3,25].So a reproducing kernel Hilbert space H corresponding to a kernel K can be denoted as H K and the norm in the space H can be denoted as || • || HK .In the following article, we suppress K by simply using H for reproducing kernel Hilbert space and || • || H for its norm.
Throughout the paper we assume the reproducing kernel Hilbert space H is separable such that By the representation theorem [25], the solution of the penalized regularization problem (5) will be of the form: Definition 2.2.let H be a separable Hilbert space and {e k } ∞ k=1 be an orthonormal basis of H. Then for any positive operator A ∈ L(H) we define T r(A) = ∞ k=1 Ae k , e k .It is well-known that the number T r(A) is independent of the choice of the orthonormal basis.
It is well-known that L 2 (H) is the separable Hilbert space with the inner product, and its norm satisfies where is the operator norm (For more details see [28]).For the positive trace class operator K x K * x , we have Given the ordered set x = (x 1 , . . ., x m ) ∈ X m , the sampling operator The regularization scheme (5) can be expressed as where We obtain the explicit expression of f z,λ by taking the functional derivative of above expression over RKHS H. Theorem 2.1.For the positive choice of λ, the functional (6) has unique minimizer: Define f λ as the minimizer of the optimization functional, Using the fact where the integral operator L K : L 2 ρX → L 2 ρX is a self-adjoint, non-negative, compact operator, defined as The integral operator L K can also be defined as a self-adjoint operator on H.We use the same notation L K for both the operators defined on different domains.It is well-known that L 1/2 K is an isometry from the space of square integrable functions to reproducing kernel Hilbert space.
In order to achieve the uniform convergence rates for learning algorithms we need some prior assumptions on the probability measure ρ.Following the notion of Bauer et al. [4] and Caponnetto et al. [9], we consider the class of probability measures P φ which satisfies the assumptions: (ii) The minimizer of the generalization error f H (4) over the hypothesis space H exists.
(iii) There exist some constants M, Σ such that for almost all x ∈ X, (iv) The target function f H belongs to the class Ω φ,R with where φ is a continuous increasing index function defined on the interval [0, κ 2 ] with the assumption φ(0) = 0.This condition is usually referred to as general source condition [23].
In addition, we consider the set of probability measures P φ,b which satisfies the conditions (i), (ii), (iii), (iv) and the eigenvalues t n 's of the integral operator L K follow the polynomial decay: For fixed positive constants α, β and b > 1, Under the polynomial decay of the eigenvalues the effective dimension N (λ), to measure the complexity of RKHS, can be estimated from Proposition 3 [9] as follows, and without the polynomial decay condition (13), we have We discuss the convergence issues for the learning algorithms (z → f z ∈ H) in probabilistic sense by exponential tail inequalities such that for all 0 < η ≤ 1 and ε(m) is a positive decreasing function of m.Using these probabilistic estimates we can obtain error estimates in expectation by integration of tail inequalities:

Convergence analysis
In this section, we analyze the convergence issues of the learning algorithms on reproducing kernel Hilbert space under the smoothness priors in the supervised learning framework.We discuss the upper and lower convergence rates for vector-valued estimators in the standard minimax setting.Therefore the estimates can be utilized particularly for scalar-valued functions and multi-task learning algorithms.

Upper rates for Tikhonov regularization
In General, we consider Tikhonov regularization in learning theory.Tikhonov regularization is briefly discussed in the literature [11,13,22,33].The error estimates for Tikhonov regularization are discussed theoretically under Hölder's source condition [9,31,32].We establish the error estimates for Tikhonov regularization scheme under general source condition f H ∈ Ω φ,R for some continuous increasing index function φ and the polynomial decay of the eigenvalues of the integral operator L K .
In order to estimate the error bounds, we consider the following inequality used in the papers [4,9] which is based on the results of Pinelis and Sakhanenko [27].Proposition 3.1.Let ξ be a random variable on the probability space (Ω, B, P ) with values in real separable Hilbert space H.If there exist two constants Q and S satisfying then for any 0 < η < 1 and for all m ∈ N, In particular, the inequality (15) holds if We estimate the error bounds for the regularized estimators by measuring the effect of random sampling and the complexity of f H .The quantities described in Proposition 3.2 express the probabilistic estimates of the perturbation measure due to random sampling.The expressions of Proposition 3.3 describe the complexity of the target function f H which are usually referred as the approximation errors.The approximation errors are independent of the samples z.Proposition 3.2.Let z be i.i.d.samples drawn according to the probability measure ρ satisfying the assumptions (10), (11) Then for all 0 < η < 1, with the confidence 1 − η, we have and Proof.To estimate the first expression, we consider the random variable Under the assumption (11) we get, , ∀n ≥ 2.
On applying Proposition 3.1 we conclude that The second expression can be estimated easily by considering the random variable ξ 2 (x) = K x K * x from (X, ρ X ) to L(H).The proof can also be found in De Vito et al. [13].
(ii) Under the assumption that φ(t) and t/φ(t) are nondecreasing functions, we have and To estimate the approximation error bounds, we consider Using the functional calculus we get, Then under the assumptions on φ described in (i), we obtain and under the assumptions on φ described in (ii), we have In the same manner with the assumptions on φ described in (ii), we get Hence we achieve the required estimates.
Theorem 3.1.Let z be i.i.d.samples drawn according to the probability measure ρ ∈ P φ where φ is the index function satisfying the conditions that φ(t), t/φ(t) are nondecreasing functions.Then for all 0 < η < 1, with confidence 1 − η, for the regularized estimator f z,λ (7) the following upper bound holds: Proof.The error of regularized solution f z,λ can be estimated in terms of the sample error and the approximation error as follows: Now f z,λ − f λ can be expressed as where The estimates of I 2 , I 3 can be obtained from Proposition 3.2 and the only task is to bound I 1 .For this we consider which implies provided that To verify this condition, we consider Now using Proposition 3.2 we get with confidence 1 − η/2, From the condition (21) we get with confidence 1 − η/2, Consequently, using (25) in the inequality ( 24) we obtain with probability 1 − η/2, From Proposition 3.2 we have with confidence 1 − η/2, Again from the condition (21) we get with probability 1 − η/2, Therefore, the inequality (23) together with ( 16), ( 20), ( 26), (27) provides the desired bound.
Theorem 3.2.Let z be i.i.d.samples drawn according to the probability measure ρ ∈ P φ and f z,λ is the regularized solution (7) corresponding to Tikhonov regularization.Then for all 0 < η < 1, with confidence 1 − η, the following upper bounds holds: (i) Under the assumption that φ(t), √ t/φ(t) are nondecreasing functions, (ii) Under the assumption that φ(t), t/φ(t) are nondecreasing functions, Proof.In order to establish the error bounds of where The estimates of I 2 and I 3 can be obtained from Proposition 3.2.To get the estimate for the sample error, we consider the following expression to bound which implies To analyze the second term we consider the expression, From Proposition 3.2 we obtain with probability 1 − η/2, Using the condition (28) we get with confidence 1 − η/2, Therefore eqn.(30) with (31) implies that with probability 1 − η/2, Using ( 16), ( 27), (32) in (29), we obtain with probability 1 − η, Using the approximation error bounds from Proposition 3.3, we get the required estimates.
Corollary 3.1.Under the same assumptions of Theorem 3.3, 3.4 for Tikhonov regularization with Hölder's source condition and for the parameter choice λ = m − b 2br+1 , we have

Upper rates for general regularization schemes
Bauer et al. [4] discussed the error estimates for general regularization schemes under general source condition.Here we study the convergence issues for general regularization schemes under general source condition and the polynomial decay of the eigenvalues of the integral operator L K .We define the regularization in learning theory framework similar to considered for ill-posed inverse problems (See Section 3.1 [4]).Definition 3.1.A family of functions g λ : [0, κ 2 ] → R, 0 < λ ≤ κ 2 , is said to be the regularization if it satisfies the following conditions: • ∃γ : sup |1 − g λ (σ)σ| ≤ γ.
• The maximal p satisfying the condition: is called the qualification of the regularization g λ , where γ p does not depend on λ.
The properties of general regularization are satisfied by the large class of learning algorithms which are essentially all the linear regularization schemes.We refer to Section 2.2 [22] for brief discussion of the regularization schemes.Here we consider general regularized solution corresponding to the above regularization: Here we are discussing the connection between the qualification of the regularization and general source condition [23].
Definition 3.2.The qualification p covers the index function φ if the function t → t p φ(t) on t ∈ (0, κ 2 ] is nondecreasing.
The following result is a restatement of Proposition 3 [23].

Proposition 3.4. Suppose φ is a nondecreasing index function and the qualification of the regularization
Generally, the index function φ is not stable with respect to perturbation in the integral operator L K .In practice, we are only accessible to the perturbed empirical operator S * x S x but the source condition can be expressed in terms of L K only.So we want to control the difference φ(L K ) − φ(S * x S x ).In order to obtain the error estimates for general regularization, we further restrict the index functions to operator monotone functions which is defined as ) is said to be operator monotone index function if φ 1 (0) = 0 and for every non-negative pair of self-adjoint operators A, B such that ||A||, ||B|| ≤ d and A ≤ B we have We consider the class of operator monotone index functions: For the above class of operator monotone functions from Theorem 1 [4], given φ 1 ∈ F µ there exists c φ1 such that ).Here we observe that the rate of convergence of φ 1 (S * x S x ) to φ 1 (L K ) is slower than the convergence rate of S * x S x to L K .Therefore we consider the following class of index functions: The splitting of φ = φ 2 φ 1 is not unique.So we can take φ 2 as a Lipschitz function with Lipschitz constant 1.Now using Corollary 1.2.2 [26] we get General source condition f H ∈ Ω φ,R corresponding to index class functions F covers wide range of source conditions as Hölder's source condition φ(t) = t r , logarithm source condition φ(t) = t p log −ν 1 t .Following the analysis of Bauer et al. [4] we develop the error estimates of general regularization for the index class function F under the suitable priors on the probability measure ρ.Theorem 3.5.Let z be i.i.d.samples drawn according to the probability measure ρ ∈ P φ .Suppose f z,λ is the regularized solution (38) corresponding to general regularization and the qualification of the regularization covers φ.Then for all 0 < η < 1, with confidence 1 − η, the following upper bound holds: Proof.We consider the error expression for general regularized solution (38), where r λ (σ) = 1 − g λ (σ)σ.Now the first term can be expressed as On applying RKHS-norm we get, where The estimate of I 2 can be obtained from the first estimate of Proposition 3.2 and from the second estimate of Proposition 3.2 with the condition (39) we obtain with probability 1 − η/2, which implies that with confidence 1 − η/2, From the properties of the regularization we have, Hence it follows, where ν 1 := B + √ BD.Therefore using ( 16), ( 42) and (44) in eqn.(41) we conclude that with probability 1 − η, Now we consider the second term, Employing RKHS-norm we get Here we used the fact that if the qualification of the regularization covers φ = φ 1 φ 2 , then the qualification also covers φ 1 and φ 2 both separately.From eqn. ( 17) and (39) we have with probability 1 − η/2, Therefore with probability 1 − η/2, Combining the bounds ( 45) and (47) we get the desired result.
Theorem 3.6.Let z be i.i.d.samples drawn according to the probability measure ρ ∈ P φ and f z,λ is the regularized solution (38) corresponding to general regularization.Then for all 0 < η < 1, with confidence 1 − η, the following upper bounds holds: (i) If the qualification of the regularization covers φ, Proof.Here we establish L 2 -norm estimate for the error expression: On applying L 2 -norm in the first term we get, where . The estimates of I 2 and I 5 can be obtained from Proposition 3.2 and Theorem 3.5 respectively.Now we consider Since φ(t) = √ t is operator monotone function.Therefore from eqn. (46) with probability 1 − η/2, we get Then using the properties of the regularization and eqn.(43) we conclude that with probability 1 − η/2, From eqn. (49) with ( 16), ( 42) and (50) we obtain with probability 1 − η, The second term can be expressed as Here two cases arises: Case 1.If the qualification of the regularization covers φ.Then we get with confidence 1 − η/2, Therefore using eqn.( 17) we obtain with probability 1 − η/2, Case 2. If the qualification of the regularization covers φ(t) √ t, we get with probability 1 − η/2, Combining the error estimates (51), ( 52) and (53) we get the desired results.
Corollary 3.2.Under the same assumptions of Theorem 3.7, 3.8 for Tikhonov regularization with Hölder's source condition and for the parameter choice λ = m − b 2br+1 , we have Remark 3.1.It is important to observe from Corollary 3.1, 3.2 that using the concept of operator monotonicity of index function we are able to achieve the same error estimates for general regularization as of Tikhonov regularization up to a constant multiple.
Remark 3.2.(Related work) Corollary 3.1 provides the order of convergence same as of Theorem 1 [9] for Tikhonov regularization under the Hölder's source condition f H ∈ Ω φ,R for φ(t) = t r 1 2 ≤ r ≤ 1 and the polynomial decay of the eigenvalues (13).Under the fact N (λ) ≤ κ 2 λ from Theorem 3.5, 3.6 we obtain the similar estimates as of Theorem 10 [4] for general regularization schemes without the polynomial decay condition of the eigenvalues (13).
Remark 3.3.For the real valued functions and multi-task algorithms (the output space Y ⊂ R m for some m ∈ N) we can obtain the error estimates from our analysis without imposing any condition on the conditional probability measure (11) for the bounded output space Y .Remark 3.4.We can address the convergence issues of binary classification problem [7] using our error estimates as similar to discussed in Section 3.3 [4] and Section 5 [32].

Lower rates for general learning algorithms
In this section, we discuss the estimates of minimum possible error over a subclass of the probability measures P φ,b parameterized by suitable functions f ∈ H. Throughout this section we assume that Y is finite-dimensional.
Let {v j } d j=1 be a basis of Y and f ∈ Ω φ,R .Then we parameterize the probability measure based on the function f , where )R and δ ξ denotes the Dirac measure with unit mass at ξ.It is easy to observe that the marginal distribution of ρ f over X is ν and the regression function for the probability measure ρ f is f (see Proposition 4 [9]).In addition to this, for the conditional probability measure ρ f (y|x) we have, We assume that the eigenvalues of the integral operator L K follow the polynomial decay (13) for the marginal probability measure ν.Then we conclude that the probability measure ρ f parameterized by f belongs to the class P φ,b .The concept of information theory such as the Kullback-Leibler information and Fano inequalities (Lemma 3.3 [12]) are the main ingredients in the analysis of lower bounds.In the literature [9,12], the closeness of probability measures is described through Kullback-Leibler information: Given two probability measures ρ 1 and ρ 2 , it is defined as where g is the density of ρ 1 with respect to ρ 2 , that is, ρ 1 (E) = E g(z)dρ 2 (z) for all measurable sets E.
Following the analysis of Caponnetto et al. [9] and DeVore et al. [12] we establish the lower rates of accuracy that can be attained by any learning algorithm.Theorem 3.9.Let z be i.i.d.samples drawn according to the probability measure ρ ∈ P φ,b under the hypothesis dim(Y ) = d < ∞.Then for any learning algorithm (z → f z ∈ H) there exists a probability measure ρ * ∈ P φ,b and f ρ * ∈ H such that for all 0 < ε < ε o , f z can be approximated as Proof.To estimate the lower rates of learning algorithms, we generate N ε -functions belonging to Ω φ,R for given ε > 0 such that (58) holds.Then we construct the probability measures ρ i ∈ P φ,b from (54), parameterized by these functions f i 's (1 ≤ i ≤ N ε ).On applying Lemma 3.3 [12], we obtain the lower convergence rates using Kullback-Leibler information.
For given ε > 0, we define εσ n e n √ ℓt n φ(t n ) , where σ = (σ 1 , . . ., σ ℓ ) ∈ {−1, +1} ℓ , t n 's are the eigenvalues of the integral operator L K , e n 's are the eigenvectors of the integral operator L K and the orthonormal basis of RKHS H.Under the decay condition on the eigenvalues α ≤ n b t n , we get where ψ(t) = √ tφ(t).
The functions f i = φ(L K )g i satisfy the source condition . For ℓ > 16, we can choose ε o such that for every 0 < ε < ε o there exists a ℓ ε satisfying this condition.
For 1 ≤ i, j ≤ N ε , we have where c = 16c ′ /15dL 2 .On applying Lemma 3.3 [12] with the probability measures ρ m fi , 1 ≤ i ≤ N ε for the disjoint sets From eqn. (64) for the probability measure ρ * such that p = ρ m * (A c i ) follows the result.
Theorem 3.12.Under the same assumptions of Theorem 3.11 for Ψ(t) = t We obtain the following corollary as a consequence of Theorem 3.10, 3.12.
Corollary 3.3.For any learning algorithm under Hölder's source condition f ρ ∈ Ω φ,R , φ(t) = t r and the polynomial decay condition ( 13) for b > 1, the lower convergence rates can be described as If the minimax lower rate coincides with the upper convergence rate for λ = λ m .Then the choice of parameter is said to be optimal.For the parameter choice λ = Ψ −1 (m −1/2 ), Theorem 3.3 and Theorem 3.8 share the upper convergence rate with the lower convergence rate of Theorem 3.10 in L 2 -norm.For the same parameter choice, Theorem 3.4 and Theorem 3.7 share the upper convergence rate with the lower convergence rate of Theorem 3.12 in RKHS-norm.Therefore the choice of the parameter is optimal.
It is important to observe that we get the same convergence rates for b = 1.

Individual lower rates
In this section, we discuss the individual minimax lower rates that describe the behavior of the error for the class of probability measure P φ,b as the sample size m grows.
where A denotes the set of all learning algorithms l : z → f l z .
In order to derive the individual lower rates we recall the following lemma (Lemma 3.2 [18]).The proofs of these lower rates are motivated from the ideas of the literature [9,18].Proof.We consider the class of probability measures such that the target function f H is parameterized by s = (s n ) ∞ n=1 ∈ {−1, +1} ∞ , the conditional probability measure ρ(y|x) follows the normal distribution centered at f H and the marginal probability measure ρ X = ν.We derive the minimax lower rates over the considered class of probability measures which is the subset of P φ,b .
For given ε > 0, where s = (s n ) ∞ n=1 ∈ {−1, +1} ∞ , t n 's are the eigenvalues of the integral operator L K , e n 's are the eigenvectors of the integral operator L K and the orthonormal basis of RKHS H. Then we have, Hence f H = φ(L K )g ∈ Ω φ,R .The mean of the probability measure f ρ = f H satisfies the moment condition (11): and H = (h i ) m i=1 .We consider, where f sgn (x)= 1, x ≥ 0; −1, x < 0, and E x (ξ) = X m ξdρ X (x 1 ) . . .dρ X (x m ).Now we discuss the lower rates over the subclass of probability measure P φ,b parameterized by {−1, +1}valued independent random variables S = (S n ) n∈N with mean zero.P rob(D(y ′ ) = S n |x) .

1 2 + 1 2b
φ(t), the estimator f z corresponding to any learning algorithm converges to the regression function f ρ with the following lower rate:

Definition 3 . 4 .
A sequence of positive numbers a n (n ∈ N) is called the individual lower rate of convergence for the class of probability measure P, if inf

L 2 ν 2 n 2 Y 1 2 n
(X) ≥ E S E x ||e n (x i )|| K xi K * xi e n , e n H 1/2 Definition 2.1.(Vector-valued reproducing kernel Hilbert space) For non-empty set X and the real Hilbert space (Y, •, • Y ), the Hilbert space (H, •, • H ) of functions from X to Y is called reproducing kernel Hilbert space if for any x ∈ X and y ∈ Y the linear functional which maps f ∈ H to y, f (x) Y is continuous.