Kernel-Based Analysis of Massive Data

Abstract

Dealing with massive data is a challenging task for machine learning. An important aspect of machine learning is function approximation. In the context of massive data, some of the commonly used tools for this purpose are sparsity, divide-and-conquer, and distributed learning. In this paper, we develop a very general theory of approximation by networks, which we have called eignets, to achieve local, stratified approximation. The very massive nature of the data allows us to use these eignets to solve inverse problems, such as finding a good approximation to the probability law that governs the data and finding the local smoothness of the target function near different points in the domain. In fact, we develop a wavelet-like representation using our eignets. Our theory is applicable to approximation on a general locally compact metric measure space. Special examples include approximation by periodic basis functions on the torus, zonal function networks on a Euclidean sphere (including smooth ReLU networks), Gaussian networks, and approximation on manifolds. We construct pre-fabricated networks so that no data-based training is required for the approximation.

1. Introduction

Rapid advances in technology have led to the availability and need to analyze a massive data. The problem arises in almost every area of life from medical science to homeland security to finance. An immediate problem in dealing with a massive data set is that it is not possible to store it in a computer memory; we therefore have to deal with the data piecemeal to keep access to an external memory to a minimum. The other challenge is to devise efficient numerical algorithms to overcome difficulties, for example, in using the customary optimization problems in machine learning. On the other hand, the very availability of a massive data set should lead also to opportunities to solve some problems heretofore considered unmanageable. For example, deep learning often requires a large amount of training data, which, in turn, helps us to figure out the granularity in the data. Apart from deep learning, distributed learning is also a popular way of dealing with big data. A good survey with the taxonomy for dealing with massive data was recently conducted by Zhou et al. [1].

As pointed out in Cucker and Smale [2], Cucker and Zhou [3], and Girosi and Poggio [4], the main task in machine learning can be viewed as one of approximation of functions based on noisy values of the target function, sampled at points that are themselves sampled from an unknown distribution. It is therefore natural to seek approximation theory techniques to solve the problem. However, most of the classical approximation theory results are either not constructive or study function approximation only on known domains. In this century, there is a new paradigm to consider function approximation on data-defined manifolds; a good introduction to the subject is in the special issue [5] of Applied and Computational Harmonic Analysis, edited by Chui and Donoho. In this theory, one assumes the manifold hypothesis, i.e., that the data is sampled from a probability distribution μ^* supported on a smooth, compact, and connected Riemannian manifold; for simplicity, even that μ^* is the Riemannian volume measure for the manifold, normalized to be a probability measure. Following (e.g., [6–10]), one constructs first a “graph Laplacian” from the data and finds its eigen decomposition. It is proved in the abovementioned papers that as the size of the data tends to infinity, the graph Laplacian converges to the Laplace-Beltrami operator on the manifold, and the eigenvalues (eigenvectors) converge to the corresponding quantities on the manifold. A great deal of work is devoted to studying the geometry of this unknown manifold (e.g., [11, 12]) based on the so-called heat kernel. The theory of function approximation on such manifolds is also well-developed (e.g., [13–17]).

A bottleneck in this theory is the computation of the eigendecomposition of a matrix, which is necessarily huge in the case of big data. Kernel-based methods have been used also in connection with approximation on manifolds (e.g., [18–22]). The kernels used in this method are constructed typically as a radial basis function (RBF) in the ambient space, and the methods are traditional machine learning methods involving optimization. As mentioned earlier, massive data poses a big challenge for the solution of these optimization problems. The theoretical results in this connection assume a Mercer's expansion in terms of the Laplacian eigenfunctions for the kernel, satisfying certain conditions. In this paper, we develop a general theory including several RBF kernels in use in different contexts (examples are discussed in section 2). Rather than using optimization-based techniques, we will provide a direct construction of the approximation based on what we have called eignets. An eignet is defined directly using the eigendecomposition on the manifold. We thus focus directly on the properties of Mercer expansion in an abstract and unified manner that enables us to construct local approximations suitable for working with massive data without using optimization.

It is also possible that the manifold hypothesis does not hold, and there is a recent work [23] by Fefferman et al. proposing an algorithm to test this hypothesis. On the other hand, our theory for function approximation does not necessarily use the full strength of Riemannian geometry. In this paper, we have therefore decided to work with a general locally compact metric measure space, isolating those properties which are needed for our analysis and substituting some that are not applicable in the current setting.

Our motivation comes from some recent works on distributed learning by Zhou et al. [24–26] as well as our own work on deep learning [27, 28]. For example, in Lin et al. [26], the approximation is done on the Euclidean sphere using a localized kernel introduced in Mhaskar [29], where the massive data is divided into smaller parts, each dense on the sphere, and the resulting polynomial approximations are added to get the final result. In Chui et al. [24], the approximation takes place on a cube, and exploits any known sparsity in the representation of the target function in terms of spline functions. In Mhaskar and Poggio [28] and Mhaskar [27], we have argued that from a function approximation point of view, the observed superiority of deep networks over shallow ones results from the ability of deep networks to exploit any compositional structure in the target function. For example, in image analysis, one may divide the image into smaller patches, which are then combined in a hierarchical manner, resulting in a tree structure [30]. By putting a shallow network at each node to learn those aspects of the target function that depend upon the pixels seen up to that level, one can avoid the curse of dimensionality. In some sense, this is a divide-and-conquer strategy, not so much on the data set itself but on the dimension of the input space.

The highlights of this paper are the following.

• In order to avoid an explicit, data-dependent eigendecomposition, we introduce the notion of an eignet, which generalizes several radial basis function and zonal function networks. We construct pre-fabricated eignets, whose linear combinations can be constructed just by using the noisy values of the target function as the coefficients, to yield the desired approximation.

• Our theory generalizes the results in a number of examples used commonly in machine learning, some of which we will describe in section 2.

• The use of optimization methods, such as empirical risk minimization has an intrinsic difficulty, namely, the minimizer of this risk may have no connection with the approximation error. There are also other problems, such as local minima, saddle points, speed of convergence, etc. that need to be taken into account, and the massive nature of the data makes this an even more challenging task. Our results do not depend upon any kind of optimization in order to determine the necessary approximation.

• We developed a theory for local approximation using eignets so that only a relatively small amount of data is used in order to approximate the target function in any ball of the space, the data being sub-sampled using a distribution supported on a neighborhood of that ball. The accuracy of approximation adjusts itself automatically depending upon the local smoothness of the target function on the ball.

• In normal machine learning algorithms, it is customary to assume a prior on the target function called smoothness class in approximation theory parlance. Our theory demonstrates clearly how a massive data can actually help to solve the inverse problem to determine the local smoothness of the target function using a wavelet-like representation based solely on the data.

• Our results allow one to solve the inverse problem of estimating the probability density from which the data is chosen. In contrast to the statistical approaches that we are aware of, there is no limitation on how accurate the approximation can be asymptotically in terms of the number of samples; the accuracy is determined entirely by the smoothness of the density function.

• All our estimates are given in terms of probability of the error being small rather than the expected value of some loss function being small.

This paper is abstract, theoretical, and technical. In section 2, we present a number of examples that are generalized by our set-up. The abstract set-up, together with the necessary definitions and assumptions, are discussed in section 3. The main results are stated in section 4 and proved in section 8. The proofs require a great deal of preparation, which is presented in sections 5–7. The results in these sections are not all new. Many of them are new only in some nuance. For example, we have proven in section 7 the quadrature formulas required in the construction of our pre-fabricated networks in a probabilistic setting, and we have also substituted an estimate on the gradients by certain Lipschitz condition, which makes sense without the differentiability structure on the manifold as we had done in our previous works. Our Theorem 7.1 generalizes most of our previous results in this direction with the exception of [31, Theorem 2.3]. We have striven to give as many proofs as possible, partly for the sake of completion and partly because the results were not stated earlier in exactly the same form as needed here. In Appendix A, we give a short proof of the fact that the Gaussian upper bound for the heat kernel holds for arbitrary smooth, compact, connected manifolds. We could not find a reference for this fact. In Appendix B, we state the main probability theory estimates that are used ubiquitously in the paper.

2. Motivating Examples

In this paper, we aim to develop a unifying theory applicable to a variety of kernels and domains. In this section, we describe some examples which have motivated the abstract theory to be presented in the rest of the paper. In the following examples, q ≥ 1 is a fixed integer.

Example 2.1. Let 𝕋^q = ℝ^q/(2πℤ^q) be the q-dimensional torus. The distance between points x = (x₁, ⋯, x_q) and y = (y₁, ⋯, y_q) is defined by . The trigonometric monomial system {exp(ik · ○) : k ∈ ℤ^q} is orthonormal with respect to the Lebesgue measure normalized to be a probability measure on 𝕋^q. We recall that the periodization of a function f :ℝ^q → ℝ is defined formally by . When f is integrable then the Fourier transform of f at k ∈ ℤ^q is the same as the k-th Fourier coefficient of f^○. This Fourier coefficient will be denoted by . A periodic basis function network has the form , where G is a periodic function called the activation function. The examples of the activation functions in which we are interested in this paper include:

• Periodization of the Gaussian.

• Periodization of the Hardy multiquadric¹.

Example 2.2. If , there exists a unique such that x = cos(θ). Therefore, [−1, 1]^q can be thought of as a quotient space of 𝕋^q where all points of the form ε ⊙ θ = {(ε₁θ₁, ⋯, ε_qθ_q)}, , are identified. Any function on [−1, 1]^q can then by lifted to 𝕋^q, and this lifting preserves all the smoothness properties of the function. Our set-up below includes [−1, 1]^q, where the distance and the measure are defined via the mapping to the torus, and suitably weighted Jacobi polynomials are considered to be the orthonormalized family of functions. In particular, if G is a periodic activation function, x = cos(θ), y = cos(ϕ), then the function is an activation function on [−1, 1]^q with an expansion , where T_k's are tensor product, orthonormalized, Chebyshev polynomials. Furthermore, b_k's have the same asymptotic behavior as (k)'s.      □

Example 2.3. Let be the unit sphere in ℝ^q+1. The dimension of 𝕊^q as a manifold is q. We assume the geodesic distance ρ on 𝕊^q and the volume measure μ^* are normalized to be a probability measure. We refer the reader to Müller [33] for details, describing here only the essentials to get a “what-it-is-all-about” introduction. The set of (equivalence classes) of restrictions of polynomials in q + 1 variables with total degree < n to 𝕊^q are called spherical polynomials of degree < n. The set of restrictions of homogeneous harmonic polynomials of degree ℓ to 𝕊^q is denoted by ℍ_ℓ with dimension d_ℓ. There is an orthonormal basis for each ℍ_ℓ that satisfies an addition formula

where ω_q−1 is the volume of 𝕊^q−1, and p_ℓ is the degree ℓ ultraspherical polynomial so that the family {p_ℓ} is orthonormalized with respect to the weight (1 − x²)^(q−2)/2 on (−1, 1). A zonal function on the sphere has the form x ↦ G(x · y), where the activation function G:[−1, 1] → ℝ has a formal expansion of the form

In particular, formally, . The examples of the activation functions in which we are interested in this paper include

• It is shown in Müller [33, Lemma 18] that

• It is shown in Mhaskar et al. [34, Lemma 5.1] that

• The smooth ReLU function . The function G has an analytic extension to the strip ℝ + (−π, π)i of the complex plane. So, Bernstein approximation theorem [35, Theorem 5.4.2] can be used to show that

Example 2.4. Let 𝕏 be a smooth, compact, connected Riemannian manifold (without boundary), ρ be the geodesic distance on 𝕏, μ^* be the Riemannian volume measure normalized to be a probability measure, {λ_k} be the sequence of eigenvalues of the (negative) Laplace-Beltrami operator on 𝕏, and ϕ_k be the eigenfunction corresponding to the eigenvalue λ_k; in particular, ϕ₀ ≡ 1. This example, of course, includes Examples 2.1–2.3. An eignet in this context has the form , where the activation function G has a formal expansion of the form . One interesting example is the heat kernel:

     □

Example 2.5. Let 𝕏 = ℝ^q, ρ be the ℓ^∞ norm on 𝕏, μ^* be the Lebesgue measure. For any multi-integer , the (multivariate) Hermite function ϕ_k is defined via the generating function

The system {ϕ_k} is orthonormal with respect to μ^*, and satisfies

where Δ is the Laplacian operator. As a consequence of the so called Mehler identity, one obtains [36] that

A Gaussian network is a network of the form , where it is convenient to think of .      □

3. The Set-Up and Definitions

3.1. Data Spaces

Let 𝕏 be a connected, locally compact metric space with metric ρ. For r > 0, x ∈ 𝕏, we denote

If K ⊆ 𝕏 and x ∈ 𝕏, we write as usual . It is convenient to denote the set

{x ∈ 𝕏; ρ(K, x) ≤ r} by 𝔹(K, r). The diameter of K is defined by .

For a Borel measure ν on 𝕏 (signed or positive), we denote by |ν| its total variation measure defined for Borel subsets K ⊂ 𝕏 by

where the supremum is over all countable measurable partitions of K. In the sequel, the term measure will mean a signed or positive, complete, sigma-finite, Borel measure. Terms, such as measurable will mean Borel measurable. If f:𝕏 → ℝ is measurable, K ⊂ 𝕏 is measurable, and ν is a measure, we define²

The symbol L^p(ν, K) denotes the set of all measurable functions f for which ‖f‖_{p, ν, K} < ∞, with the usual convention that two functions are considered equal if they are equal |ν|-almost everywhere on K. The set C₀(K) denotes the set of all uniformly continuous functions on K vanishing at ∞. In the case when K = 𝕏, we will omit the mention of K, unless it is necessary to mention it to avoid confusion.

We fix a non-decreasing sequence , with λ₀ = 0 and λ_k ↑ ∞ as k → ∞. We also fix a positive sigma-finite Borel measure μ^* on 𝕏, and a system of orthonormal functions , such that ϕ₀(x) > 0 for all x ∈ 𝕏. We define

It is convenient to write Π_n = {0} if n ≤ 0 and Π_∞ = ⋃_n>0Π_n. It will be assumed in the sequel that Π_∞ is dense in C₀ (and, thus, in every L^p, 1 ≤ p < ∞). We will often refer to the elements of Π_∞ as diffusion polynomials in keeping with [13].

Definition 3.1. We will say that a sequence {a_n} (or a functionF :[0, ∞) → ℝ) isfast decreasingif(respectively, ) for everyS > 0. A sequence {a_n} haspolynomial growthif there existc₁, c₂ > 0 such thatfor alln ≥ 1, and similarly for functions.

Definition 3.2. The space 𝕏 (more precisely, the tuple) is called adata spaceif each of the following conditions is satisfied.

• For eachx ∈ 𝕏, r > 0, 𝔹(x, r) is compact.

• (Ball measure condition) There existq ≥ 1 and κ > 0 with the following property: for eachx ∈ 𝕏, r > 0,

  (In particular, μ^*({y ∈ 𝕏: ρ(x, y) = r}) = 0.)

• (Gaussian upper bound) There exist κ₁, κ₂ > 0 such that for allx, y ∈ 𝕏, 0 < t ≤ 1,

• (Essential compactness) For everyn ≥ 1, there exists a compact set 𝕂_n ⊂ 𝕏 such that the functionn ↦ diam(𝕂_n) has polynomial growth, while the functions

  and

  are both fast decreasing. (Necessarily, has polynomial growth as well.)

Remark 3.1. We assume without loss of generality that 𝕂_n ⊆ 𝕂_m for all n < m and that .      □

Remark 3.2. If 𝕏 is compact, then the first condition as well as the essential compactness condition are automatically satisfied. We may take 𝕂_n = 𝕏 for all n. In this case, we will assume tacitly that μ^* is a probability measure, and ϕ₀ ≡ 1.      □

Example 3.1. (Manifold case) This example points out that our notion of data space generalizes the set-ups in Examples 2.1–2.4. Let 𝕏 be a smooth, compact, connected Riemannian manifold (without boundary), ρ be the geodesic distance on 𝕏, μ^* be the Riemannian volume measure normalized to be a probability measure, {λ_k} be the sequence of eigenvalues of the (negative) Laplace-Beltrami operator on 𝕏, and ϕ_k be the eigenfunction corresponding to the eigenvalue λ_k; in particular, ϕ₀ ≡ 1. If the condition (3.2) is satisfied, then is a data space. Of course, the assumption of essential compactness is satisfied trivially (see Appendix B for the Gaussian upper bound).      □

Example 3.2. (Hermite case) We illustrate how Example 2.5 is included in our definition of a data space. Accordingly, we assume the set-up as in that example. For a > 0, let . With , the system is a data space. When a = 1, we will omit its mention from the notation in this context. The first two conditions are obvious. The Gaussian upper bound follows by the multivariate Mehler identity [37, Equation 4.27]. The assumption of essential compactness is satisfied with 𝕂_n = 𝔹(0, cn) for a suitable constant c (cf. [38, Chapter 6]).      □

In the rest of this paper, we assume 𝕏 to be a data space. Different theorems will require some additional assumptions, two of which we now enumerate. Not every theorem will need all of these; we will state explicitly which theorem uses which assumptions, apart from 𝕏 being a data space.

The first of these deals with the product of two diffusion polynomials. We do not know of any situation where it is not satisfied but are not able to prove it in general.

Definition 3.3. (Product assumption) There existsA^* ≥ 1 and a familysuch that for everyS > 0,

We say that anstrong product assumptionis satisfied if, instead of (3.4), we have for everyn > 0 andP, Q ∈ Π_n, .

Example 3.3. In the setting of Example 3.2, if P, Q ∈ Π_n, then PQ = Rϕ₀ for some R ∈ Π_2n. So, the product assumption holds trivially. The strong product assumption does not hold. However, if P, Q ∈ Π_n, then . The manifold case is discussed below in Remark 3.3.      □

Remark 3.3. One of the referees of our paper has pointed out three recent references [39–41], on the subject of the product assumption. The first two of these deal with the manifold case (Example 3.1). The paper [41] extends the results in Lu et al. [40] to the case when the functions ϕ_k are eigenfunctions of a more general elliptic operator. Since the results in these two papers are similar qualitatively, we will comment on Lu et al. [40] and Steinerberger [39].

In this remark only, let . Let λ_k, λ_j < n. In Steinerberger [39], Steinerberger relates E_An(2, ϕ_kϕ_j) [see (3.6) below for definition] with

While this gives some insight into the product assumption, the results are inconclusive about the product assumption as stated. Also, it is hard to verify whether the conditions mentioned in the paper are satisfied for a given manifold.

In Lu et al. [40], it is shown that for any ϵ, δ > 0, there exists a subspace V of dimension such that for all ϕ_k, ϕ_j ∈ Π_n, . The subspace V does not have to be Π_An for any A. Since the dimension of span{ϕ_kϕ_j} is O(n²), the result is meaningful only if 0 < δ < 1 and ϵ ≥ n^1−1/δ.

In Geller and Pesenson [42, Theorem 6.1], it is shown that the strong product assumption (and, thus, also the product assumption) holds in the manifold case when the manifold is a compact homogeneous manifold. We have extended this theorem in Filbir and Mhaskar [17, Theorem A.1] for the case of eigenfunctions of general elliptic partial differential operators on arbitrary compact, smooth manifolds provided that the coefficient functions in the operator satisfy some technical conditions.      □

In our results in section 4, we will need the following condition, which serves the purpose of gradient in many of our earlier theorems on manifolds.

Definition 3.4. We say that the system Ξ satisfiesBernstein-Lipschitz conditionif for everyn > 0, there existsB_n > 0 such that

Remark 3.4. Both in the manifold case and the Hermite case, B_n = cn for some constant c > 0. A proof in the Hermite case can be found in Mhaskar [43] and in the manifold case in Filbir and Mhaskar [44].      □

3.2. Smoothness Classes

We define next the smoothness classes of interest here.

Definition 3.5. A functionw:𝕏 → ℝ will be called aweight functioniffor allk. Ifwis a weight function, we define

We will omit the mention ofwifw ≡ 1 on 𝕏.

We find it convenient to denote by X^p the space ; i.e., X^p = L^p(𝕏) if 1 ≤ p < ∞ and .

Definition 3.6. Let 1 ≤ p ≤ ∞, γ > 0, andwbe a weight function.

(a) Forf ∈ L^p(𝕏), we define

and note that

The spaceW_γ,p,wcomprises allffor which ‖f‖_Wγ,p,w < ∞.

(b) We write. IfBis a ball in 𝕏, comprises functions in, which are supported onB.

(c) Ifx₀ ∈ 𝕏, the spaceW_γ,p,w(x₀) comprises functionsfsuch that there existsr > 0 with the property that, for every, ϕf ∈ W_γ,p,w.

Remark 3.5. In both the manifold case and the Hermite case, characterizations of the smoothness classes W_γ,p are available in terms of constructive properties of the functions, such as the number of derivatives, estimates on certain moduli of smoothness or K-functionals, etc. In particular, the class C^∞ coincides with the class of infinitely differentiable functions vanishing at infinity.      □

We can now state another assumption that will be needed in studying local approximation.

Definition 3.7. (Partition of unity) For everyr > 0, there exists a countable familyof functions inC^∞with the following properties:

• Eachis supported on 𝔹(x_k, r) for somex_k ∈ 𝕏.

• For everyandx ∈ 𝕏, 0 ≤ ψ_{k, r}(x) ≤ 1.

• For everyx ∈ 𝕏, there exists a finite subsetsuch that

We note some obvious observations about the partition of unity without the simple proof.

Proposition 3.1. Letr > 0, be a partition of unity.

(a)Necessarily,is supported on 𝔹(x, 3r).

(b)Forx ∈ 𝕏, .

The constant conventionIn the sequel, c, c₁, ⋯ will denote generic positive constants depending only on the fixed quantities under discussion, such as Ξ, q, κ, κ₁, κ₂, the various smoothness parameters, and the filters to be introduced. Their value may be different at different occurrences, even within a single formula. The notationA ~ Bmeansc₁A ≤ B ≤ c₂A.      □

We end this section by defining a kernel that plays a central role in this theory.

Let H :[0, ∞) → ℝ be a compactly supported function. In the sequel, we define

If S ≥ 1 is an integer, and H is S times continuously differentiable, we introduce the notation

The following proposition recalls an important property of these kernels. Proposition 3.2 is proven in Maggioni and Mhaskar [13] and more recently in much greater generality in Mhaskar [45, Theorem 4.3].

Proposition 3.2. LetS > qbe an integer,H :ℝ → ℝ be an even,Stimes continuously differentiable, compactly supported function. Then, for everyx, y ∈ 𝕏, N > 0,

In the sequel, let h :ℝ → [0, 1] be a fixed, infinitely differentiable, even function, non-increasing on [0, ∞), with h(t) = 1 if |t| ≤ 1/2 and h(t) = 0 if t ≥ 1. If ν is any measure with a bounded total variation on 𝕏, we define

We will omit the mention of h in the notations; e.g., write Φ_n(x, y) = Φ_n(h; x, y), and the mention of ν if ν = μ^*. In particular,

where for , we write

.

3.3. Measures

In this section, we describe the terminology involving measures.

Definition 3.8. Letd ≥ 0. A measurewill be calledd–regularif

The infimum of all constantscthat work in (3.15) will be denoted by |||ν|||_{R, d}, and the class of alld-regular measures will be denoted by.

For example, μ^* itself is in R₀ with [cf. (3.2)]. More generally, if w ∈ C₀(𝕏) then the measure wdμ^* is R₀ with .

Definition 3.9. (a) A sequence {ν_n} of measures on 𝕏 is called anadmissible quadrature measure sequenceif the sequence {|ν_n|(𝕏)}has polynomial growth and

(b) A sequence {ν_n} of measures on 𝕏 is called anadmissible product quadrature measure sequenceif the sequence {|ν_n|(𝕏)}has polynomial growth and

(c) By abuse of terminology, we will say that a measure ν_nis anadmissible quadrature measure(respectively, anadmissible product quadrature measure) of ordernif(with constants independent ofn) and (3.16) [respectively, (3.17)] holds.

In the case when 𝕏 is compact, a well-known theorem called Tchakaloff's theorem [46, Exercise 2.5.8, p. 100] shows the existence of admissible product quadrature measures (even finitely supported probability measures). However, in order to construct such measures, it is much easier to prove the existence of admissible quadrature measures, as we will do in Theorem 7.1, and then use one of the product assumptions to derive admissible product quadrature measures.

Example 3.4. In the manifold case, let the strong product assumption hold as in Remark 3.3. If n ≥ 1 and is a finite subset satisfying the assumptions of Theorem 7.1, then the theorem asserts the existence of an admissible quadrature measure supported on . If {ν_n} is an admissible quadrature measure sequence, then is an admissible product quadrature measure sequence. In particular, there exist finitely supported admissible product quadrature measures of order n for every n ≥ 1.      □

Example 3.5. We consider the Hermite case as in Example 3.2. For every a > 0 and n ≥ 1, Theorem 7.1 applied with the system Ξ_a yields admissible quadrature measures of order n supported on finite subsets of ℝ^q (in fact, of [−cn, cn]^q for an appropriate c). In particular, an admissible quadrature measure of order for is an admissible product quadrature measure of order n for Ξ = Ξ₁.      □

3.4. Eignets

The notion of an eignet defined below is a generalization of the various kernels described in the examples in section 2.

Definition 3.10. A functionb:[0, ∞) → (0, ∞) is called asmooth maskifbis non-increasing, and there existsB^* = B^*(b) ≥ 1 such that the mappingt ↦ b(B^*t)/b(t) is fast decreasing. A functionG:𝕏 × 𝕏 → ℝ is called asmooth kernelif there exists a measurable functionW = W(G) :𝕏 → ℝ such that we have a formal expansion (with a smooth maskb)

Ifm ≥ 1 is an integer, aneignetwithmneurons is a function of the formfory_k ∈ 𝕏.

Example 3.6. In the manifold case, the notion of eignet includes all the examples stated in section 2 with W ≡ 1, except for the example of smooth ReLU function described in Example 2.3. In the Hermite case, (2.2) shows that the kernel defined on ℝ^q × ℝ^q is a smooth kernel, with λ_k = |k|₁, ϕ_k as in Example 2.5, and . The function W here is .      □

Remark 3.6. It is possible to relax the conditions on the mask in Definition 3.10. Firstly, the condition that b should be non-increasing is made only to simplify our proofs. It is not difficult to modify them without this assumption. Secondly, let b₀ :[0, ∞) → ℝ satisfy |b₀(t)| ≤ b₁(t) for a smooth mask b₁ as stipulated in that definition. The function b₂ = b + 2b₁ is then a smooth mask and so is b₁. Let , j = 0, 1, 2. Then G₀(x, y) = G₂(x, y) − 2G₁(x, y). Therefore, all of the results in sections 4 and 8 can be applied once with G₂ and once with G₁ to obtain a corresponding result for G₀ with different constants. For this reason, we will simplify our presentation by assuming the apparently restrictive conditions stipulated in Definition 3.10. In particular, this includes the example of the smooth ReLU network described in Example 2.3.      □

Definition 3.11. Let ν be a measure on 𝕏 (signed or having bounded variation), andG ∈ C₀(𝕏 × 𝕏). We define

and

Remark 3.7. Typically, we will use an approximate product quadrature measure sequence in place of the measure ν, where each of the measures in the sequence is finitely supported, to construct a sequence of networks. In the case when 𝕏 is compact, Tchakaloff's theorem shows that there exists an approximate product quadrature measure of order m supported on points. Using this measure in place of ν, one obtains a pre-fabricated eignet 𝔾_n(ν) with neurons. However, this is not an actual construction. In the presence of the product assumption, Theorem 7.1 leads to the pre-fabricated networks 𝔾_n in a constructive manner with the number of neurons as stipulated in that theorem.      □

4. Main Results

In this section, we assume the Bernstein-Lipschitz condition (Definition 3.4) in all the theorems. We note that the measure μ^* may not be a probability measure. Therefore, we take the help of an auxiliary function f₀ to define a probability measure as follows. Let f₀ ∈ C₀(𝕏), f₀ ≥ 0 for all x ∈ 𝕏, and be a probability measure. Necessarily, ν^* is 0-regular, and . We assume noisy data of the form (y, ϵ), with a joint probability distribution τ defined for Borel subsets of 𝕏 × Ω for some measure space Ω, and with ν^* being the marginal distribution of y with respect to τ. Let be a random variable following the law τ, and denote

It is easy to verify using Fubini's theorem that if is integrable with respect to τ, then, for any x ∈ 𝕏,

Let Y be a random sample from τ, and {ν_n} be an admissible product quadrature sequence in the sense of Definition 3.9. We define [cf. (3.20)]

where B^* is as in Definition 3.10.

Remark 4.1. We note that the networks 𝔾_n are prefabricated independently of the data. The network therefore has only |Y| terms depending upon the data.      □

Our first theorem describes local function recovery using local sampling. We may interpret it in the spirit of distributed learning as in Chui et al. [24] and Lin et al. [26], where we are taking a linear combination of pre-fabricated networks 𝔾_n using the function values themselves as the coefficients. The networks 𝔾_n have essentially the same localization property as the kernels Φ_n (cf. Theorem 8.2).

Theorem 4.1. Letx₀ ∈ 𝕏 and r > 0. We assume the partition of unity and find a function ψ ∈ C^∞supported on 𝔹(x₀, 3r), which is equal to 1 on 𝔹(x₀, r), , and letf₀ = ψ/𝔪, . We assume the rest of the set-up as described. Iff₀f ∈ W_{γ, ∞}, then for 0 < δ < 1, and,

Remark 4.2. If {y₁, ⋯, y_M} is a random sample from some probability measure supported on 𝕏, , and we construct a sub-sample using the distribution that associates the mass f₀(y_j)/s with each y_j, then the probability of selecting points outside of the support of f₀ is 0. This leads to a sub-sample Y. If , then the Chernoff bound, Proposition B.1(b), can be used to show that |Y| is large, as stipulated in Theorem 4.1.               □

Next, we state two inverse theorems. Our first theorem obtains accuracy on the estimation of the density f₀ using eignets instead of positive kernels.

Theorem 4.2. With the set-up as in Theorem 8.3, let γ > 0, f₀ ∈ W_{γ, ∞}, and

Then, with,

Remark 4.3. Unlike density estimation using positive kernels, there is no inherent limit on the accuracy predicted by (4.5) on the estimation of f₀.      □

The following theorem gives a complete characterization of the local smoothness classes using eignets. In particular, Part (b) of the following theorem gives a solution to the inverse problem of determining what smoothness class the target function belongs to near each point of 𝕏. In theory, this leads to a data-based detection of singularities and sparsity analogous to what is assumed in Chui et al. [24] but in a much more general setting.

Theorem 4.3. Let f₀ ∈ C₀(𝕏), f₀(x) ≥ 0 for all x ∈ 𝕏, and be a probability measure, τ, , and let f be as described above. We assume the partition of unity and the product assumption. Let S ≥ q + 2, 0 < γ ≤ S, x₀ ∈ 𝕏, 0 < δ < 1. For each j ≥ 0, suppose that Y_j is a random sample from τ with . Then with τ-probability ≥ 1 − δ,

(a) Iff₀f ∈ W_γ,∞(x₀) then there exists a ball 𝔹 centered atx₀such that

(b) If there exists a ball 𝔹 centered atx₀for which (4.6) holds, thenf₀f ∈ W_{γ, ∞,ϕ0}(x₀).

5. Preparatory Results

We prove a lower bound on μ^*(𝔹(x, r)) for x ∈ 𝕏 and 0 < r ≤ 1 (cf. [47]).

Proposition 5.1. We have

In order to prove the proposition, we recall a lemma, proved in Mhaskar [14, Proposition 5.1].

Lemma 5.1. Let ν ∈ R_d, N > 0. Ifg₁:[0, ∞) → [0, ∞) is a non-increasing function, then, for anyN > 0, r > 0, x ∈ 𝕏,

Proof of Proposition 5.1.

Let x ∈ 𝕏, r > 0 be fixed in this proof, although the constants will not depend upon these. In this proof, we write

The Gaussian upper bound (3.3) shows that for t > 0,

Using Lemma 5.1 with d = 0, dν = dμ^*, , , we obtain for :

Therefore, denoting in this proof only that κ₀ = ‖ϕ₀‖_∞, we obtain that

We now choose t ~ r² so that to obtain (5.1) for r ≤ c₄. The estimate is clear for c₄ < r ≤ 1.      □

Next, we prove some results about the system {ϕ_k}.

Lemma 5.2. Forn ≥ 1, we have

and

In particular, the functionn ↦ dim(Π_n) has polynomial growth.

Proof. The Gaussian upper bound with x = y implies that

The estimate (5.6) follows from a Tauberian theorem [44, Proposition 4.1]. The essential compactness now shows that for any R > 0,

In particular,

            □

Next, we prove some properties of the operators σ_n and diffusion polynomials. The following proposition follows easily from Lemma 5.1 and Proposition 3.2. (cf. [14, 48]).

Proposition 5.2. Let S, H be as in Proposition 3.2, d > 0, , andx ∈ 𝕏.

(a)Ifr ≥ 1/N, then

(b) We have

and

The following lemma is well-known; a proof is given in Mhaskar [15, Lemma 5.3].

Lemma 5.3. Let (Ω₁, ν), (Ω₂, τ) be sigma–finite measure spaces, Ψ : Ω₁ × Ω₂ → ℝ be ν × τ–integrable,

and formally, for τ–measurable functionsf : Ω₂ → ℝ,

Let 1 ≤ p ≤ ∞. IfthenT(f, x) is defined for ν–almost allx ∈ Ω₁, and

Theorem 5.1. Let n > 0. If P ∈ Π_n/2, then σ_n(P) = P. Also, for any p with 1 ≤ p ≤ ∞,

If 1 ≤ p ≤ ∞, and f ∈ L^p (𝕏), then

Proof. The fact that σ_n(P) = P for all P ∈ Π_n/2 is verified easily using the fact that h(t) = 1 for 0 ≤ t ≤ 1/2. Using (5.9) with μ^* in place of |ν| and 0 in place of d, we see that

The estimate (5.14) follows using Lemma 5.3. The estimate (5.15) is now routine to prove.      □

Proposition 5.3. Forn ≥ 1, P ∈ Π_n, 1 ≤ p ≤ ∞, andS > 0, we have

Proof. In this proof, all constants will depend upon S. Using Schwarz inequality and essential compactness, it is easy to deduce that

Therefore, a use of Lemma 5.3 shows that

We use P in place of f to obtain (5.16).      □

Proposition 5.4. Letn ≥ 1, P ∈ Π_n, 0 < p < r ≤ ∞. Then

Proof. The first part of (5.18) is proved in Mhaskar [15, Lemma 5.4]. In that paper, the measure μ^* is assumed to be a probability measure, but this assumption was not used in this proof. The second estimate follows easily from Proposition 5.3.           □

Lemma 5.4. LetR, n > 0, P₁, P₂ ∈ Π_n, 1 ≤ p, r, s ≤ ∞. If the product assumption holds, then

Proof. In view of essential compactness, Proposition 5.4 implies that for any P ∈ Π_n, 1 ≤ r ≤ ∞, . Therefore, using Schwarz inequality, Parseval identity, and Lemma 5.2, we conclude that

Now, the product assumption implies that for p = 1, ∞, and λ_k, λ_j < n, there exists such that for any R > 0,

where c is the constant appearing in (5.20). The convexity inequality

shows that (5.21) is valid for all p, 1 ≤ p ≤ ∞. So, using (5.20), we conclude that

     □

6. Local Approximation by Diffusion Polynomials

In the sequel, we write g(t) = h(t) − h(2t), and

We note that

It is clear from Theorem 5.1 that for any p, 1 ≤ p ≤ ∞,

with convergence in the sense of L^p.

Theorem 6.1. Let 1 ≤ p ≤ ∞, γ > 0, f ∈ X^p, x₀ ∈ 𝕏. We assume the partition of unity and the product assumption.

(a) If 𝔹 is a ball centered atx₀, then

(b) If there exists a ballBcentered atx₀such that

thenf ∈ W_{γ, p,ϕ0}(x₀).

(c) If f ∈ W_{γ, p}(x₀), then there exists a ball 𝔹 centered atx₀such that (6.5) holds.

Remark 6.1. In the manifold case (Example 3.1), ϕ₀ ≡ 1. So, the statements (b) and (c) in Theorem 6.1 provide necessary and sufficient conditions for f ∈ W_{γ, p}(x₀) in terms of the local rate of convergence of the globally defined operator σ_n(f) and the growth of the local norms of the operators τ_j, respectively In the Hermite case (Example 3.2), it is shown in Mhaskar [49] that f ∈ W_{γ, p,ϕ0} if and only if f ∈ W_{γ, p}. Therefore, the statements (b) and (c) in Theorem 6.1 provide similar necessary and sufficient conditions for f ∈ W_{γ, p}(x₀) in this case as well.      □

The proof of Theorem 6.1 is routine, but we sketch a proof for the sake of completeness.

Proof of Theorem 6.1.

Part (a) is easy to prove using the definitions.

In the rest of this proof, we fix S > γ+q+2. To prove part (b), let ϕ ∈ C^∞ be supported on 𝔹. Then there exists such that

Further, Lemma 5.4 yields a sequence such that

Hence,

Thus, fϕ ∈ W_{γ, p,ϕ0} for every ϕ ∈ C^∞ supported on 𝔹, and part (b) is proved.

To prove part (c), we observe that there exists r > 0 such that for any , fϕ ∈ W_{γ, p}. Using partition of unity [cf. Proposition 3.1(a)], we find such that ψ(x) = 1 for all x ∈ 𝔹(x₀, 2r), and we let 𝔹 = 𝔹(x₀, r). In view of Proposition 3.2, for all x ∈ B and y ∈ 𝕏\𝔹(x₀, 2r). Hence,

Recalling that ψ(x) = 1 for x ∈ B and 𝕊−q≥γ+2, we deduce that

This proves part (c).      □

Let {Ψ_n:𝕏×𝕏→𝕏} be a family of kernels (not necessarily symmetric). With a slight abuse of notation, we define when possible, for any measure ν with bounded total variation on 𝕏,

and

As usual, we will omit the mention of ν when ν = μ^*.

Corollary 6.1. Let the assumptions of Theorem 6.1 hold, and {Ψ_n:𝕏 × 𝕏 → 𝕏} be a sequence of kernels (not necessarily symmetric) with the property that both of the following functions of n are decreasing rapidly.

(a) If Bis a ball centered atx₀, then

(b)If there exists a ballBcentered atx₀such that

thenf ∈ W_{γ, p,ϕ0}(x₀).

(c)Iff ∈ W_{γ, p}(x₀), then there exists a ballBcentered atx₀such that (6.13) holds.

Proof. In view of Lemma 5.3, the assumption about the functions in (6.11) implies that ‖σ(Ψ_n; f)−σ_n(f)‖_p is decreasing rapidly.      □

7. Quadrature Formula

The purpose of this section is to prove the existence of admissible quadrature measures in the general set-up as in this paper. The ideas are mostly developed already in our earlier works [17, 36, 43, 44, 50, 51] but always require an estimate on the gradient of diffusion polynomials. Here, we use the Bernstein-Lipschitz condition (Definition 3.4) instead.

If , we denote

If K is compact, ϵ > 0, a subset is ϵ-distinguishable if ρ(x, y) ≥ ϵ for every , x ≠ y. The cardinality the maximal ϵ-distinguishable subset of K will be denoted by H_ϵ(K).

Remark 7.1. If is a maximal -distinguishable subset of , x ≠ y, then it is easy to deduce that

In particular, by replacing by , we can always assume that

Theorem 7.1. We assume the Bernstein-Lipschitz condition. Let n > 0, be a finite subset, ϵ > 0.

(a)There exists a constant c(ϵ) with the following property: if , then there exist non-negative numbers W_k satisfying

such that for everyP ∈ Π_n,

(b)Let the assumptions of part (a) be satisfied with ϵ = 1/2. There exist real numbers w₁, ⋯, w_M such that |w_k| ≤ 2W_k, k = 1, ⋯, M, in particular,

and

(c)Let δ > 0, be a random sample from the probability law given by

and ϵ_n = min(1/n, 1/B_2n). If

then the statements (a) and (b) hold with -probability exceeding 1−δ.

In order to prove Theorem 7.1, we first recall the following theorem [52, Theorem 5.1], applied to our context. The statement of Mhaskar [52, Theorem 5.1] seems to require that μ^* is a probability measure, but this fact is not required in the proof. It is required only that μ^*(𝔹(x, r)) ≥ cr^q for 0 < r ≤ 1.

Theorem 7.2. Let τ be a positive measure supported on a compact subset of 𝕏, ϵ > 0, be a maximal ϵ-distinguishable subset ofsupp(τ), and . There then exists a subset and a partition of 𝕂 with each of the following properties.

• (volume property) For , Y_y ⊆ 𝔹(y, 18ϵ), , and.

• (density property) , .

• (intersection property) Let 𝕂₁ ⊆ 𝕂 be a compact subset. Then

Proof of Theorem 7.1 (a), (b).

We observe first that it is enough to prove this theorem for sufficiently large values of n. In view of Proposition 5.3, we may choose n large enough so that for any P ∈ Π_n,

In this proof, we will write so that . We use Theorem 7.2 with τ to be the measure associating the mass 1 with each element of , and δ in place of ϵ. If is a maximal δ-distinguished subset of , then we denote in this proof, and observe that . We obtain a partition {Y_y} of 𝕂 as in Theorem 7.2. The volume property implies that each Y_y contains at least one element of . We construct a subset of by choosing exactly one element of for each y. We may then re-index so that, without loss of generality, for some N ≤ M, and re-index {Y_y} as {Y_k}, so that z_k ∈ Y_k, k = 1, ⋯, N. To summarize, we have a subset , and a partition of 𝕂⊃𝕂_2n such that each Y_k ⊂ 𝔹(z_k, 36δ) and . In particular (cf. (7.7)), for any P ∈ Π_n,

We now let , k = 1, ⋯, N, and W_k = 0, k = N+1, ⋯, M.

The next step is to prove that if δ ≤ c(ϵ)min(1/n, 1/B_2n), then

In this part of the proof, the constants denoted by c₁, c₂, ⋯ will retain their value until (7.9) is proved. Let y ∈ 𝕏. We let r≥δ to be chosen later, and write in this proof, , and for j = 0, 1, ⋯, . Since r≥δ, and each Y_k ⊂ 𝔹(z_k, 36δ), there are at most elements in . Using the Bernstein-Lipschitz condition and the fact that , we deduce that

Next, since , we see that the number of elements in each is ~ (^2jr/δ)q. Using Proposition 3.2 and the fact that S > q, we deduce that if r ≥ 1/n, then

Since S>q, we may choose r~_ϵn such that , and we then require δ ≤ min(r, c₇(ϵ)/B_2n) so that, in (7.10), . Then (7.10) and (7.11) lead to (7.9). The proof of (7.9) being completed, we resume the constant convention as usual.

Next, we observe that for any P ∈ Π_n,

We therefore conclude, using (7.9), that

Together with (7.8), this leads to (7.4). From the definition of , k = 1, ⋯, N, , and . Since W_k = 0 if k≥N+1, we have now proven (7.3), and we have thus completed the proof of part (a).

Having proved part (a), the proof of part (b) is by now a routine application of the Hahn-Banach theorem [cf. [17, 44, 50, 51]]. We apply part (a) with ϵ = 1/2. Continuing the notation in the proof of part (a), we then have

We now equip ℝ^N with the norm and consider the sampling operator given by , let V be the range of this operator, and define a linear functional x^* on V by . The estimate (7.12) shows that the norm of this functional is ≤ 2. The Hahn-Banach theorem yields a norm-preserving extension 𝕏^* of x^* to ℝ^N, which, in turn, can be identified with a vector . We set w_k = 0 if k ≥ N + 1. Formula (7.6) then expresses the fact that 𝕏^* is an extension of x^*. The preservation of norms shows that |w_k| ≤ 2W_k if k = 1, ⋯, N, and it is clear that for k = N+1, ⋯, M, |w_k| = 0 = W_k. This completes the proof of part (b).      □

Part (c) of Theorem 7.1 follows immediately from the first two parts and the following lemma.

Lemma 7.1. Let ν^* be a probability measure on 𝕏, 𝕂 ⊂ supp(ν^*) be a compact set. Let ϵ, δ ∈ (0, 1], be a maximal ϵ/2-distinguished subset of 𝕂, and. If

and {z₁, ⋯, z_M} be random samples from the probability law ν^*then

Proof. If δ(𝕂, {z₁, ⋯, z_M})>ϵ, then there exists at least one such that 𝔹(x, ϵ/2)∩{z₁, ⋯, z_M} = ∅. For every , . We consider the random variable z_j to be equal to 1 if z_j ∈ 𝔹(x, ϵ/2) and 0 otherwise. Using (B.2) with t = 1, we see that

Since ,

We set the right-hand side above to δ and solve for M to prove the lemma.      □

8. Proofs of the Results in Section 4

We assume the set-up as in section 4. Our first goal is to prove the following theorem.

Theorem 8.1. Let τ, ν^*, , f be as described section 4. We assume the Bernstein-Lipschitz condition. Let 0 < δ <1. We assume further that for all y ∈ 𝕏, ϵ ∈ Ω. There exist constants c₁, c₂, such that if , and {(y₁, ϵ₁), ⋯, (y_M, ϵ_M)} is a random sample from τ, then

In order to prove this theorem, we record an observation. The following lemma is an immediate corollary of the Bernstein-Lipschitz condition and Proposition 5.3.

Lemma 8.1. Let the Bernstein-Lipschitz condition be satisfied. Then for everyn > 0 and ϵ > 0, there exists a finite set such that and for anyP ∈ Π_n,

Proof of Proposition 8.1.

Let x ∈ 𝕏. We consider the random variables

Then in view of (4.2), for every j. Further, Proposition 3.2 shows that for each j, . Using (5.10) with ν^* in place of ν, N = n, d = 0, we see that for each j,

Therefore, Bernstein concentration inequality (B.1) implies that for any t ∈ (0, 1),

We now note that Z_j, are all in Π_n. Taking a finite set as in Lemma 8.1, so that , we deduce that

Then (8.3) leads to

We set the right-hand side above equal to and solve for t to obtain (8.1) (with different values of c, c₁, c₂).      □

Before starting to prove results regarding eignets, we first record the continuity and smoothness of a “smooth kernel” G as defined in Definition 3.10.

Proposition 8.1. IfGis a smooth kernel, then (x, y) ↦ W(y)G(x, y) is in. Further, for anyp, 1 ≤ p ≤ ∞, and Λ ≥ 1,

In particular, for everyx, y ∈ 𝕏, W(○)G(x, ○) andW(y)G(○, y) are inC^∞.

Proof. Let b be the smooth mask corresponding to G. For any S ≥ 1, b(n) ≤ cn^−sb(n/B^*) ≤ cn^−sb(0). Thus, b itself is decreasing rapidly. Next, let r > 0. Then remembering that B^* ≥ 1 and b is non-increasing, we obtain that for S > 0, b(B^*Λu) ≤ c(Λu)^−s−r−1b(Λu), and

In this proof, let , so that s(t) ≤ ct^q, t ≥ 1. If Λ ≥ 1, then, integrating by parts, we deduce (remembering that b is non-increasing) that for any x ∈ 𝕏,

Using Schwarz inequality, we conclude that

In particular, since b is fast decreasing, W(○)G(x, ○) ∈ C₀(𝕏) (and in fact, W(y)G(x, y) ∈ C₀(𝕏 × 𝕏)) and (8.5) holds with p = ∞. Next, for any j ≥ 0, essential compactness implies that

So, there exists r≥q such that

Hence, for any x ∈ 𝕏,

This shows that

In view of the convexity inequality,

(8.8) and (8.10) lead to

In turn, this implies that WG(x, ○) ∈ L^p for all x ∈ 𝕏, and (8.5) holds.      □

A fundamental fact that relates the kernels Φ_n and the pre-fabricated eignets 𝔾_n's is the following theorem.

Theorem 8.2. Let G be a smooth kernel and {ν_n} be an admissible product quadrature measure sequence. Then, for 1 ≤ p ≤ ∞,

is fast decreasing. In particular, for everyS > 0

Proof. Let x ∈ 𝕏. In this proof, we define P_n = P_{n, x} by , z ∈ 𝕏, and note that . In view of Proposition 8.1, the expansion in (3.18) converges in , so that term-by-term integration can be made to deduce that for y ∈ 𝕏,

By definition, , and, hence, each of the summands in the last expression above is equal to 0. Therefore, recalling that h(λ_k/n) = 0 if λ_k > n, we obtain

Since , and is an admissible product quadrature measure of order B^*n, this implies that

Therefore, for y ∈ 𝕏,

Using Proposition 8.1 (used with Λ = B^*n) and the fact that has polynomial growth, we deduce that

In view of Proposition 5.4 and Proposition 5.2, we see that for any z ∈ 𝕏,

We now conclude from (8.14) that

Since {b(B^*n)/b(n)} is fast decreasing, this completes the proof.        □

The theorems in section 4 all follow from the following basic theorem.

Theorem 8.3. We assume the strong product assumption and the Bernstein-Lipschitz condition. With the set-up just described, we have

In particular, forf ∈ 𝕏^∞(𝕏), Then

Proof. Theorems 8.1 and Theorem 8.2 together lead to (8.15). Since , the estimate 8.91 follows from Theorem 5.1 used with p = ∞.      □

Proof of Theorem 4.1.

We observe that with the choice of f₀ as in this theorem, . Using 𝔪δ in place of δ, we obtain Theorem 4.1 directly from Theorem 8.3 by some simple calculations.      □

Proof of Theorem 4.2.

This follows directly from Theorem 8.3 by choosing .           □

Proof of Theorem 4.3.

In view of Theorem 8.3, our assumptions imply that for each j ≥ 0,

Consequently, with probability ≥ 1 − δ, we have for each j ≥ 1,

Hence, the theorem follows from Theorem 6.1.      □

References

• 1.

  ZhouLPanSWangJVasilakosAV. Machine learning on big data: opportunities and challenges. Neurocomputing. (2017) 237:350–61. 10.1016/j.neucom.2017.01.026

  • CrossRef
  • Google Scholar
  • View reference in article

• 2.

  CuckerFSmaleS. On the mathematical foundations of learning. Bull Am Math Soc. (2002) 39:1–49. 10.1090/S0273-0979-01-00923-5

  • CrossRef
  • Google Scholar

• 3.

  CuckerFZhouDX. Learning Theory: An Approximation Theory Viewpoint, Vol. 24. Cambridge: Cambridge University Press (2007).

  • Google Scholar
  • View reference in article

• 4.

  GirosiFPoggioT. Networks and the best approximation property. Biol Cybernet. (1990) 63:169–76. 10.1007/BF00195855

  • CrossRef
  • Google Scholar
  • View reference in article

• 5.

  ChuiCKDonohoDL. Special issue: diffusion maps and wavelets. Appl Comput Harm Anal. (2006) 21:1–2. 10.1016/j.acha.2006.05.005

  • CrossRef
  • Google Scholar
  • View reference in article

• 6.

  BelkinMNiyogiP. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. (2003) 15:1373–96. 10.1162/089976603321780317

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 7.

  BelkinMNiyogiP. Towards a theoretical foundation for Laplacian-based manifold methods. J Comput Syst Sci. (2008) 74:1289–308. 10.1016/j.jcss.2007.08.006

  • CrossRef
  • Google Scholar
  • View reference in article

• 8.

  BelkinMNiyogiP. Semi-supervised learning on Riemannian manifolds. Mach Learn. (2004) 56:209–39. 10.1023/B:MACH.0000033120.25363.1e

  • CrossRef
  • Google Scholar
  • View reference in article

• 9.

  LafonSS. Diffusion maps and geometric harmonics (Ph.D. thesis), Yale University, New Haven, CT, United States (2004).

  • Google Scholar
  • View reference in article

• 10.

  SingerA. From graph to manifold Laplacian: the convergence rate. Appl Comput Harm Anal. (2006) 21:128–34. 10.1016/j.acha.2006.03.004

  • CrossRef
  • Google Scholar
  • View reference in article

• 11.

  JonesPWMaggioniMSchulR. Universal local parametrizations via heat kernels and eigenfunctions of the Laplacian. Ann Acad Sci Fenn Math. (2010) 35:131–74. 10.5186/aasfm.2010.3508

  • CrossRef
  • Google Scholar
  • View reference in article

• 12.

  LiaoWMaggioniM. Adaptive geometric multiscale approximations for intrinsically low-dimensional data. arXiv. (2016) 1611.01179.

  • Google Scholar
  • View reference in article

• 13.

  MaggioniMMhaskarHN. Diffusion polynomial frames on metric measure spaces. Appl Comput Harm Anal. (2008) 24:329–53. 10.1016/j.acha.2007.07.001

  • CrossRef
  • Google Scholar
  • View reference in article

• 14.

  MhaskarHN. Eignets for function approximation on manifolds. Appl Comput Harm Anal. (2010) 29:63–87. 10.1016/j.acha.2009.08.006

  • CrossRef
  • Google Scholar
  • View reference in article

• 15.

  MhaskarHN. A generalized diffusion frame for parsimonious representation of functions on data defined manifolds. Neural Netw. (2011) 24:345–59. 10.1016/j.neunet.2010.12.007

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 16.

  EhlerMFilbirFMhaskarHN. Locally learning biomedical data using diffusion frames. J Comput Biol. (2012) 19:1251–64. 10.1089/cmb.2012.0187

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 17.

  FilbirFMhaskarHN. Marcinkiewicz-Zygmund measures on manifolds. J Complexity. (2011) 27:568–96. 10.1016/j.jco.2011.03.002

  • CrossRef
  • Google Scholar
  • View reference in article

• 18.

  RosascoLBelkinMVitoED. On learning with integral operators. J Mach Learn Res. (2010) 11:905–34.

  • Google Scholar
  • View reference in article

• 19.

  RudiACarratinoLRosascoL. Falkon: an optimal large scale kernel method. arXiv. (2017) 1705.10958. Available online at: http://jmlr.org/papers/v11/rosasco10a.html.

  • Google Scholar
  • View reference in article

• 20.

  LuSPereverzevSV. Regularization Theory for Ill-Posed Problems. Berlin: de Gruyter (2013).

  • Pubmed Abstract
  • Google Scholar
  • View reference in article

• 21.

  MhaskarHPereverzyevSVSemenovVYSemenovaEV. Data based construction of kernels for semi-supervised learning with less labels. Front Appl Math Stat. (2019) 5:21. 10.3389/fams.2019.00021

  • CrossRef
  • Google Scholar
  • View reference in article

• 22.

  PereverzyevSVTkachenkoP. Regularization by the linear functional strategy with multiple kernels. Front Appl Math Stat. (2017) 3:1. 10.3389/fams.2017.00001

  • CrossRef
  • Google Scholar
  • View reference in article

• 23.

  FeffermanCMitterSNarayananH. Testing the manifold hypothesis. J Am Math Soc. (2016) 29:983–1049. 10.1090/jams/852

  • CrossRef
  • Google Scholar
  • View reference in article

• 24.

  ChuiCKLinS-BZhangBZhouDX. Realization of spatial sparseness by deep relu nets with massive data. arXiv. (2019) 1912.07464.

  • Google Scholar
  • View reference in article

• 25.

  GuoZCLinSBZhouDX. Learning theory of distributed spectral algorithms. Inverse Probl. (2017) 33:074009. 10.1088/1361-6420/aa72b2

  • CrossRef
  • Google Scholar
  • View reference in article

• 26.

  LinSBWangYGZhouDX. Distributed filtered hyperinterpolation for noisy data on the sphere. arXiv. (2019) 1910.02434.

  • Google Scholar
  • View reference in article

• 27.

  MhaskarHNPoggioT. Deep vs. shallow networks: an approximation theory perspective. Anal Appl. (2016) 14:829–48. 10.1142/S0219530516400042

  • CrossRef
  • Google Scholar
  • View reference in article

• 28.

  MhaskarHPoggioT. Function approximation by deep networks. arXiv. (2019) 1905.12882.

  • Google Scholar
  • View reference in article

• 29.

  MhaskarHN. On the representation of smooth functions on the sphere using finitely many bits. Appl Comput Harm Anal. (2005) 18:215–33. 10.1016/j.acha.2004.11.004

  • CrossRef
  • Google Scholar
  • View reference in article

• 30.

  SmaleSRosascoLBouvrieJCaponnettoAPoggioT. Mathematics of the neural response. Foundat Comput Math. (2010) 10:67–91. 10.1007/s10208-009-9049-1

  • CrossRef
  • Google Scholar
  • View reference in article

• 31.

  MhaskarHN. On the representation of band limited functions using finitely many bits. J Complexity. (2002) 18:449–78. 10.1006/jcom.2001.0637

  • CrossRef
  • Google Scholar
  • View reference in article

• 32.

  HardyRL. Theory and applications of the multiquadric-biharmonic method 20 years of discovery 1968–1988. Comput Math Appl. (1990) 19:163–208. 10.1016/0898-1221(90)90272-L

  • CrossRef
  • Google Scholar
  • View reference in article

• 33.

  MüllerA. Spherical Harmonics, Vol. 17. Berlin: Springer (2006).

  • Google Scholar

• 34.

  MhaskarHNNarcowichFJWardJD. Approximation properties of zonal function networks using scattered data on the sphere. Adv Comput Math. (1999) 11:121–37. 10.1023/A:1018967708053

  • CrossRef
  • Google Scholar
  • View reference in article

• 35.

  TimanAF. Theory of Approximation of Functions of a Real Variable: International Series of Monographs on Pure and Applied Mathematics, Vol. 34. New York, NY: Dover Publications (2014).

  • Google Scholar
  • View reference in article

• 36.

  ChuiCKMhaskarHN. A unified method for super-resolution recovery and real exponential-sum separation. Appl Comput Harmon Anal. (2019) 46:431–51. 10.1016/j.acha.2017.12.007

  • CrossRef
  • Google Scholar
  • View reference in article

• 37.

  ChuiCKMhaskarHN. A Fourier-invariant method for locating point-masses and computing their attributes. Appl Comput Harmon Anal. (2018) 45:436–52. 10.1016/j.acha.2017.08.010

  • CrossRef
  • Google Scholar
  • View reference in article

• 38.

  MhaskarHN. Introduction to the Theory of Weighted Polynomial Approximation, Vol. 56. Singapore: World Scientific Singapore (1996).

  • Google Scholar
  • View reference in article

• 39.

  SteinerbergerS. On the spectral resolution of products of laplacian eigenfunctions. arXiv. (2017) 1711.09826.

  • Google Scholar
  • View reference in article

• 40.

  LuJSoggeCDSteinerbergerS. Approximating pointwise products of laplacian eigenfunctions. J Funct Anal. (2019) 277:3271–82. 10.1016/j.jfa.2019.05.025

  • CrossRef
  • Google Scholar
  • View reference in article

• 41.

  LuJSteinerbergerS. On pointwise products of elliptic eigenfunctions. arXiv. (2018) 1810.01024.

  • Google Scholar
  • View reference in article

• 42.

  GellerDPesensonIZ. Band-limited localized Parseval frames and Besov spaces on compact homogeneous manifolds. J Geometr Anal. (2011) 21:334–71. 10.1007/s12220-010-9150-3

  • CrossRef
  • Google Scholar

• 43.

  MhaskarHN. Local approximation using Hermite functions. In: N. K. Govil, R. Mohapatra, M. A. Qazi, G. Schmeisser eds. Progress in Approximation Theory and Applicable Complex Analysis. Cham: Springer (2017). p. 341–62. 10.1007/978-3-319-49242-1_16

  • CrossRef
  • Google Scholar
  • View reference in article

• 44.

  FilbirFMhaskarHN. A quadrature formula for diffusion polynomials corresponding to a generalized heat kernel. J Fourier Anal Appl. (2010) 16:629–57. 10.1007/s00041-010-9119-4

  • CrossRef
  • Google Scholar
  • View reference in article

• 45.

  MhaskarHN. A unified framework for harmonic analysis of functions on directed graphs and changing data. Appl Comput Harm Anal. (2018) 44:611–44. 10.1016/j.acha.2016.06.007

  • CrossRef
  • Google Scholar
  • View reference in article

• 46.

  RivlinTJ. The Chebyshev Polynomials. New York, NY: John Wiley and Sons (1974).

  • Google Scholar
  • View reference in article

• 47.

  GrigorlyanA. Heat kernels on metric measure spaces with regular volume growth. Handb Geometr Anal. (2010) 2. Available online at: https://www.math.uni-bielefeld.de/~grigor/hga.pdf.

  • Google Scholar

• 48.

  MhaskarHN. Approximate quadrature measures on data-defined spaces. In: Dick J, Kuo FY, Wozniakowski H, editors. Festschrift for the 80th Birthday of Ian Sloan. Berlin: Springer (2017). p. 931–62. 10.1007/978-3-319-72456-0_41

  • CrossRef
  • Google Scholar
  • View reference in article

• 49.

  MhaskarHN. On the degree of approximation in multivariate weighted approximation. In: M. D. Buhman, and D. H. Mache, eds. Advanced Problems in Constructive Approximation. Basel: Birkhäuser (2003). p. 129–41. 10.1007/978-3-0348-7600-1_10

  • CrossRef
  • Google Scholar
  • View reference in article

• 50.

  MhaskarHN. Approximation theory and neural networks. In: Proceedings of the International Workshop in Wavelet Analysis and Applications. Delhi (1999). p. 247–89.

  • Pubmed Abstract
  • Google Scholar
  • View reference in article

• 51.

  MhaskarHNNarcowichFJWardJD. Spherical Marcinkiewicz-Zygmund inequalities and positive quadrature. Math Comput. (2001) 70:1113–30. 10.1090/S0025-5718-00-01240-0

  • CrossRef
  • Google Scholar
  • View reference in article

• 52.

  MhaskarHN. Dimension independent bounds for general shallow networks. Neural Netw. (2020) 123:142–52. 10.1016/j.neunet.2019.11.006

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 53.

  HörmanderL. The spectral function of an elliptic operator. Acta Math. (1968) 121:193–218. 10.1007/BF02391913

  • CrossRef
  • Google Scholar
  • View reference in article

• 54.

  ShubinMA. Pseudodifferential Operators and Spectral Theory. Berlin: Springer (1987).

  • Google Scholar
  • View reference in article

• 55.

  Grigor'yanA. Gaussian upper bounds for the heat kernel on arbitrary manifolds. J Diff Geom. (1997) 45:33–52. 10.4310/jdg/1214459753

  • CrossRef
  • Google Scholar
  • View reference in article

• 56.

  BoucheronSLugosiGMassartP. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford: Oxford University Press (2013).

  • Google Scholar
  • View reference in article

• 57.

  HagerupTRübC. A guided tour of Chernoff bounds. Inform Process Lett. (1990) 33:305–8. 10.1016/0020-0190(90)90214-I

  • CrossRef
  • Google Scholar
  • View reference in article