On the Construction of Sparse Matrices From Expander Graphs

1. Introduction

Sparse binary matrices, say A ∈ {0, 1}^n×N, with n ≪ N are widely used in applications including graph sketching [2, 3], network tomography [4, 5], data streaming [6, 7], breaking privacy of databases via aggregate queries [8], compressed imaging of intensity patterns [9], and more generally combinatorial compressed sensing [10–15], linear sketching [7], and group testing [16, 17]. In all these areas we are interested in the case where n ≪ N, in which case A is used as an efficient encoder of sparse signals x ∈ ℝ^N with sparsity s ≪ n, where they are known to preserve ℓ¹ distance of sparse vectors [18]. Conditions that guarantee that a given encoder, A, also referred to as a sensing matrix in compressed sensing, typically include the the nullspace, coherence, and the restricted isometry conditions, see [19] and references there in. The goal is for A to satisfy one or more of these conditions with the minimum possible n, the number of measurements. For uniform guarantees over all A, it has been established that n has to be Ω(s²), but that with high probability on the draw of random A, n can be ${O}(slogN/n)$ for A with entries drawn from a sub-gaussian distribution, see [19] for a review of such results. Matrices with entries drawn from a Bernoulli distribution fall in the family of sub-gaussian but these are dense as opposed the the sparse binary matrices considered here. For computational advantages, such as faster application and smaller storage, it is advantageous to use sparse A in application [1, 14, 18].

Herein we consider the n achievable when A is an adjacency matrix of an expander graph [18], expander graph will be defined in the next section. Since the construction of such matrices can be construed as either a linear algebra problem or equivalently a graph theory one (in this manuscript we will focus more on the linear algebra discourse). There has been significant research on expander graphs in pure mathematics and theoretical computer science, see [20] and references therein. Both deterministic and probabilistic constructions of expander graphs have been suggested. The best known deterministic constructions achieve $n={O}(s^{1+\alpha })$ for some α > 0 [21]. On the other hand random constructions, first proven in Bassalygo and Pinsker [22], achieve the optimal $n={O}(slog(N/s))$, precisely $n={O}(sd)$, with $d={O}(log(N/s))$, where d is the left degree of the expander graph but also the number of ones in each column of A, to be defined in the next section. However, to the best of our knowledge, it was [1] that proposed a probabilistic construction that is not only optimal but also more suitable to making quantitative statements where such matrices are applied.

This work follows the probabilistic construction proposed in Bah and Tanner[1] but with careful computation of the bounds, is able to achieve $n={O}(slog(N/s))$ with $d={O}\left(\frac{log(N/s)}{logs}\right)$. We retain the complexity of n but got a smaller complexity for d, which is novel. Related results with a similar d were derived in Indyk and Razenshteyn [23] and Bah et al.[24] but for structure sparse signals in the framework of model-based compressing sensing or sketching. In that framework, one has second order information about x beyond simple sparsity, which is first order information about x. It is thus expected and established that it is possible to get a small n and hence a smaller d. Arguably, such a small complexity for d justifies in hindsight fixing d to a small number in simulations with such A as in Bah and Tanner [1], Bah et al. [24], and Mendoza-Smith and Tanner [14], just to mention a few.

The results derived here are asymptotic, though finite dimensional bounds follow directly. We focus on for what ratios of the problem dimensions (s, n, N) does these results hold. There is almost a standard way of interrogating such a question, i.e., phase transitions, probably introduced to the compressed sensing literature by Donoho and Tanner [25]. In other words, we derive sampling theorems numerically depicted by phase transition plots about problem size spaces for which our construction holds. This is similar to what was done in Bah and Tanner [1] but for comparison purposes we include phase transition plots from probabilistic constructions by Buhrman et al. [26] and Berinde [27]. The plots show improvement over these earlier works. Furthermore, we show implications of our results for compressed sensing by using our results with the phase transition framework to compare the performance of selected combinatorial compressed sensing algorithms as is done in Bah and Tanner [1] and Mendoza-Smith and Tanner [14].

The manuscript is organized as follows. Section 1 gives the introduction; while section 2 sets the notation and defines some useful terms. The main results are stated in section 3 and the details of the construction is given in section 4. This is followed by a discussion in section 5 about our results, comparing them to existing results and using the results to compare the performance of some combinatorial compressed sensing algorithms. In section 6 we state the remaining proofs of theorems, lemmas, corollaries, and propositions used in this manuscript. After this section is the conclusion in section 7. We include an Appendix in Supplementary Materials, where we summarized key relevant materials from Bah and Tanner [1], and showed the derivation of some bounds used in the proofs.

2. Preliminaries

2.1. Notation

Scalars will be denoted by lowercase letters (e.g., k), vectors by lowercase boldface letters (e.g., x), sets by uppercase calligraphic letters (e.g., ${S}$) and matrices by uppercase boldface letters (e.g., A). The cardinality of a set ${S}$ is denoted by $\left|{S}\right|$ and [N]: = {1, …, N}. Given ${S}\subseteq \left[N\right]$, its complement is denoted by ${{S}}^c:=\left[N\right]\backslash {S}$ and ${\text{x}}_{{S}}$ is the restriction of x ∈ ℝ^N to ${S}$, i.e., ${({\text{x}}_{{S}})}_i={\text{x}}_i$ if $i\in {S}$ and 0 otherwise. For a matrix A, the restriction of A to the columns indexed by ${S}$ is denoted by ${\text{A}}_{{S}}$. For a graph, $\Gamma ({S})$ denotes the set of neighbors of ${S}$, that is the nodes that are connected to the nodes in ${S}$, and e_ij = (x_i, y_j) represents an edge connecting node x_i to node y_j. The ℓ_p norm of a vector x ∈ ℝ^N is defined as $\Vert \text{x}{\Vert }_p:={\left(\sum _{i=1}^N{x}_i^p\right)}^{1/p}$.

2.2. Definitions

Below we give formal definitions that will be used in this manuscript.

Definition 2.1 (ℓ_p-norm restricted isometry property). A matrix A satisfies the ℓ_p-norm restricted isometry property (RIP-p) of order s and constant δ_s < 1 if it satisfies the following inequality.

$\begin{array}{ll}(1-{\delta }_s)\Vert \text{x}{\Vert }_p^p\le \Vert \text{A}\text{x}{\Vert }_p^p\le (1+{\delta }_s)\Vert \text{x}{\Vert }_p^p,\forall s\mathrm{\text{–sparse }}\text{x}.& (1)\end{array}$

The most popular case is RIP-2 and was first proposed in Cand‘es et al. [28]. Typically when RIP is mentioned without qualification, it means RIP₂. In the discourse of this work though RIP-1 is the most relevant. The RIP says that A is a near-isometry and it is a sufficient condition to guarantee exact sparse recovery in the noiseless setting (i.e., y = Ax); or recovery up to some error bound, also referred to as optimality condition, in the noisy setting (i.e., y = Ax + e, where e is the bounded noise vector). we define optimality condition more precisely below.

Definition 2.2 (Optimality condition). Given y = Ax + e and $\hat{\text{x}}=\Delta (\text{A}\text{x}+\text{e})$ for a reconstruction algorithm Δ, the optimal error guarantee is

$\begin{array}{ll}\Vert \hat{\text{x}}-\text{x}{\Vert }_p\le C_1{\sigma }_s{(\text{x})}_q+C_2\Vert \text{x}{\Vert }_p,& (2)\end{array}$

where C₁, C₂ > 0 depend only on the RIP constant (RIC), i.e., δ_s, and not the problem size, 1 ≤ q ≤ p ≤ 2, and σ_s(x)_q denote the error of the best s-term approximation in the ℓ_q-norm, that is

$\begin{array}{ll}{\sigma }_s{(\text{x})}_q:={\displaystyle \underset{s-\mathrm{\text{sparse }}\text{z}}{min}}\Vert \text{z}-\text{x}{\Vert }_q.& (3)\end{array}$

Equation (2) is also referred to as the ℓ_p/ℓ_q optimality condition (or error guarantee). Ideally, we would like ℓ₂/ℓ₂, but the best provable is ℓ₂/ℓ₁ [28], weaker than this is the ℓ₁/ℓ₁ [18], which is what is possible with the A considered in this work.

To aid translation between the terminology of graph theory and linear algebra we define the set of neighbors in both notation.

Definition 2.3 (Definition 1.4 in Bah and Tanner [1]). Consider a bipartite graph $G=(\left[N\right],\left[n\right],{E})$ where ${E}$ is the set of edges and e_ij = (x_i, y_j) is the edge that connects vertex x_i to vertex y_j. For a given set of left vertices ${S}\subset \left[N\right]$ its set of neighbors is $\Gamma ({S})=\left\{y_j|x_i\in {S}\mathrm{\text{ and }}{e}_{ij}\in {E}\right\}$. In terms of the adjacency matrix, A, of $G=(\left[N\right],\left[n\right],{E})$ the set of neighbors of ${\text{A}}_{{S}}$ for $\left|{S}\right|=s$, denoted by A_s, is the set of rows with at least one non-zero.

Definition 2.4 (Expander graph). Let $G=(\left[N\right],\left[n\right],{E})$ be a left-regular bipartite graph with N left vertexes, n right vertexes, a set of edges ${E}$ and left degree d. If, for any ϵ ∈ (0, 1/2) and any ${S}\subset \left[N\right]$ of size $\left|{S}\right|\le s$, we have that $\left|\Gamma ({S})\right|\ge (1-ϵ)d\left|{S}\right|$, then G is referred to as a (s, d, ϵ)-expander graph.

The ϵ is referred to as the expansion coefficient of the graph. A (s, d, ϵ)-expander graph, also called an unbalanced expander graph [18] or a lossless expander graph [29], is a highly connected bipartite graph. We denote the ensemble of n × N binary matrices with d ones per column by ${B}(N,n;d)$, or just ${B}$ to simplify notation. We also will denote the ensemble of n × N adjacency matrices of (s, d, ϵ)-expander graphs as 𝔼(N, n; s, d, ϵ) or simply 𝔼.

3. Results

Before stating the main results of the paper, we summarise here how we improve upon existing results in Bah and Tanner [1], which is generally true of standard random construction of such matrices. Essentially in the random constructions we want one minus the probability in (5) to go to zero as the problem size (s, n, N) → ∞. In Theorem 4.1 (which was Theorem 1.6 in Bah and Tanner [1]) for this to happen we require $d={O}(log(N/s))$ based on the bound (7). In this work we derived a tighter bound than (7), which is (12), that requires d to be smaller than the standard bound by a factor of log(s).

The main result of this work is formalized in Theorem 3.1, which is an asymptotic result, where the dimensions grow while their ratios remain bounded. This is also referred to as the proportional growth asymptotics [30, 31]. We point out that it has been observed in practice that applying these matrices (including the simulations we do in this manuscript), which necessitate having finite dimensions, produces results that agree with the theory. However, the theoretical analysis used in this work cannot be applied to the finite setting.

Theorem 3.1. Consider $ϵ\in (0,\frac{1}{2})$ and let d, s, n, N ∈ ℕ, a random draw of an n×N matrix A from ${B}$, i.e., for each column of A uniformly assign ones in d out of n positions, as (s, n, N) → ∞ while s/n ∈ (0, 1) and n/N ∈ (0, 1), with probability approaching 1 exponentially, the matrix A ∈ 𝔼 with

$\begin{array}{ll}d={O}\left(\frac{log\left(N/s\right)}{ϵlogs}\right),\text{ and }n={O}\left(\frac{slog\left(N/s\right)}{{ϵ}^2}\right).& (4)\end{array}$

The proof of this theorem is found in section 6.1. It is worth emphasizing that the complexity of d is novel and it is the main contribution of this work.

Furthermore, in the proportional growth asymptotics, i.e., as (s, n, N) → ∞ while s/n → ρ and n/N → δ with ρ, δ ∈ (0, 1), for completeness, we derived a phase transition function (curve) in δρ-space below which Theorem 3.1 holds, but above which Theorem 3.1 fails to hold. This is formalized in the following lemma.

Lemma 3.1. Fix $ϵ\in (0,\frac{1}{2})$ and let d, s, n, N ∈ ℕ, as (s, n, N) → ∞ while s/n → ρ ∈ (0, 1) and n/N → δ ∈ (0, 1) then for ρ < (1−γ)ρ_BT(δ; d, ϵ) and γ > 0, a random draw of A from ${B}$ implies A ∈ 𝔼 with probability approaching 1 exponentially.

The proof of this lemma is given in section 6.2. The phase transition function ρ_BT(δ; d, ϵ) turned out to be significantly higher that those derived from existing probabilistic constructions, hence our results are significant improvement over earlier works. This will be graphically demonstrated with some numerical simulations in section 5.

4. Construction

The standard probabilistic construction is for each column of A to uniformly assign ones in d out of n positions; while the standard approach to derive the probability bounds is to randomly selected s columns of A indexed by ${S}$ and compute the probability that |A_s| < (1 − ϵ)ds, then do a union bound over all sets ${S}$ of size s. Our work in Bah and Tanner [1] computed smaller bounds than previous works based on a dyadic splitting of ${S}$ and derived the following bound. We changed the notation and format of Theorem 1.6 in Bah and Tanner [1] slightly to be consistent with the notation and format in this manuscript.

Theorem 4.1 (Theorem 1.6, Bah and Tanner [1]). Consider d, s, n, N ∈ ℕ, fix ${S}$ with $\left|{S}\right|\le s$, let an n×N matrix A be drawn from ${B}$, then

$\begin{array}{ll}Prob(|A_s|\le a_s)<p_n(s,d)·e^{\left[n·{\Psi }_n(a_s,\dots ,a_1)\right]}& (5)\end{array}$

with a₁: = d, and the functions defined as

$\begin{array}{ll}{p}_n(s,d)=\frac{2}{25\sqrt{2\pi s^3{d}^3}},and& (6)\end{array}$

$\begin{array}{ll}{\Psi }_n(a_s,\dots ,a_1)=\frac{1}{n}\left[3slog(5d)+{\displaystyle \sum _{i\in \Omega }}\frac{s}{2i}{\psi }_i\right],\Omega ={\left\{2^j\right\}}_{j=0}^{{log}_2(s)-1},& (7)\end{array}$

where ψ_i is given by the following expression

$\begin{array}{ll}\left(n-a_i\right)·{H}\left(\frac{a_{2i}-a_i}{n-a_i}\right)+a_i·{H}\left(\frac{a_{2i}-a_i}{a_i}\right)-n·{H}\left(\frac{a_i}{n}\right)& (8)\end{array}$

and ${H}(·)$ is the Shannon entropy in base e logarithms, and the index set $\Omega ={\left\{2^j\right\}}_{j=0}^{{log}_2(s)-1}$.

a) If no restriction is imposed on a_s then the a_i for i > 1 take on their expected value â_i given by

$\begin{array}{ll}{â}_{2i}={â}_i\left(2-\frac{{â}_i}{n}\right),fori\in {\left\{2^j\right\}}_{j=0}^{{log}_2(s)-1}.& (9)\end{array}$

b) If a_s is restricted to be less than â_s, then the a_i for i > 1 are the unique solutions to the following polynomial system

$\begin{array}{ll}{a}_{2i}^3-2a_i{a}_{2i}^2+2a_i^2{a}_{2i}-a_i^2{a}_{4i}=0,fori\in {\left\{2^j\right\}}_{j=0}^{{log}_2(s)-2},& (10)\end{array}$

with a_2i ≥ a_i for each i.

In this work, based on the same approach as in Bah and Tanner [1], we derive new expressions for the p_n(s, d) and Ψ_n(a_s, …, a₁) in Theorem 4.1, i.e., (6) and (7) respectively, and provide a simpler bound for the improved expression of Ψ_n(a_s, …, a₁).

Lemma 4.1. Theorem 4.1 holds with the functions

$\begin{array}{ll}{p}_n(s,d)=2^{-3}{s}^{9/2}{e}^{1/4},& (11)\end{array}$

$\begin{array}{l}{\Psi }_n(a_s,\dots ,a_1)=\frac{1}{n}[\frac{3\log 2}{2}{\log }_2^{2}s+({\log }_{2}s-\frac{3}{2})\log a_s\hfill \\ +{\displaystyle \sum }_{i\in \Omega }\frac{s}{2i}{\psi }_i],for \Omega ={\left\{2^j\right\}}_{j=0}^{{\log }_2(s)-1}.\hfill & (12)\end{array}$

The proof of the lemma is given in section 6.3. Asymptotically, the argument of the exponential term in the bound of the probability in (5) of Theorem 4.1, i.e., Ψ_n(a_s, …, a₁) in (12) is more important than the polynomial p_n(s, d) in (11) since the exponential factor will dominate the polynomial factor. The significance of the lemma is that Ψ_n(a_s, …, a₁) in (12) is smaller than Ψ_n(a_s, …, a₁) in (7) since $\frac{3log2}{2}{log}_2^{2}s+({log}_{2}s-\frac{3}{2})log{a}_s$ in (12) is asymptotically smaller than 3slog(5d) in (7), because the former is ${O}(\mathrm{\text{polylog}}s)$ while the latter is ${O}(s)$, since we consider $a_s={O}(s)$.

Recall that we are interested in computing Prob(|A_s| ≤ a_s) when a_s = (1 − ϵ)ds. This means having to solve the polynomial equation (10) to compute as small a bound of Ψ_n((1 − ϵ)ds, …, d) as possible. We derive an asymptotic solution to (10) for a_s = (1 − ϵ)ds and use that solution to get the following bounds.

Theorem 4.2. Consider d, s, n, N ∈ ℕ, fix ${S}$ with $\left|{S}\right|\le s$, for η > 0, β ≥ 1, and $ϵ\in (0,\frac{1}{2})$, let an n × N matrix A be drawn from ${B}$, then

$\begin{array}{ll}Prob\left(\left|A_s\right|\le (1-ϵ)ds\right)<p_n(s,d,ϵ)·exp\left[n·{\Psi }_n(s,d,ϵ)\right]& (13)\end{array}$

where

$\begin{array}{ll}{p}_n(s,d,ϵ)=\frac{\sqrt[4]{e}·s^{{log}_2(1-ϵ)+3}}{\sqrt{2^6{(1-ϵ)}^3{d}^3}},& (14)\end{array}$

$\begin{array}{l}{\Psi }_n(s,d,ϵ)\le -\frac{1}{2n}[\eta {\beta }^{-1}(\beta -1)(1-ϵ)ds{\log }_2(s/2)\hfill \\ -(5\log 2){\log }_2^{2}s-2\log d{\log }_{2}s].\hfill & (15)\end{array}$

The proof of this theorem is also found in section 6.5. Since Theorem 4.2 holds for a fixed ${S}$ of size at most s, if we want this to hold for all ${S}$ of size at most s, we do a union bound over all ${S}$ of size at most s. This leads to the following probability bound.

Theorem 4.3. Consider d, s, n, N ∈ ℕ, and all ${S}$ with $\left|{S}\right|\le s$, for τ > 0, and $ϵ\in (0,\frac{1}{2})$, let an n × N matrix A be drawn from ${B}$, then

$\begin{array}{ll}Prob(\left|A_s\right|\le (1-ϵ)ds)<p_N(s,d,ϵ)·e^{\left[N·{\Psi }_N(s,d,ϵ)\right]},& (16)\end{array}$

where

$\begin{array}{ll}{p}_N(s,d,ϵ)=\frac{5·\sqrt[4]{e}·s^{{log}_2(1-ϵ)+5/2}}{\sqrt{2^{10}{(1-ϵ)}^3{d}^3\pi (1-s/N)}},& (17)\end{array}$

$\begin{array}{l}{\Psi }_N(s,d,ϵ)\le -\frac{s}{N}log(\frac{s}{N})+\frac{s}{N}\\ -\frac{\tau (1-ϵ)d}{2log2}\frac{s}{N}log(\frac{s}{2})+o(N).& (18)\end{array}$

Proof: Applying the union bound over all ${S}$ of size at most s to (13) leads to the following.

$\begin{array}{ll}\text{Prob}\left(\left|A_s\right|\le (1-ϵ)ds\right)<\left(\genfrac{}{}{0.0pt}{}{N}{s}\right)p_n(s,d,ϵ)e^{\left[n·{\Psi }_n(s,d,ϵ)\right]}.& (19)\end{array}$

Then we used the upper bound of (143) to bound the combinatorial term $(\genfrac{}{}{0.0pt}{}{N}{s})$ in (19). After some algebraic manipulations, we separated the the polynomial term, given in (17), from the exponential terms whose exponent is

$\begin{array}{ll}{\Psi }_N(s,d,ϵ):={H}\left(\frac{s}{N}\right)+\frac{n}{N}{\Psi }_n(s,d,ϵ).& (20)\end{array}$

We upper bound Ψ_N(s, d, ϵ) in (18) by upper bounding ${H}\left(\frac{s}{N}\right)$ with $-\frac{s}{N}log\left(\frac{s}{N}\right)+\frac{s}{N}$ and the upper bound of Ψ_n(s, d, ϵ) in (15). The o(N) decays to zeros with N and its a result of dividing the polylogarithmic terms of s in (15), and τ = ηβ⁻¹(β − 1) in (15). This concludes the proof.□

The next corollary easily follows from Theorem 4.3 and it is equivalent to Theorem 3.1. Its statement is that if the conditions therein holds, then the probability that the cardinality of the set of neighbors of any ${S}$ with $\left|{S}\right|\le s$ is less than (1 − ϵ)ds goes to zero as dimensions of A grows. On the other hand, the probability that the cardinality of the set of neighbors of any ${S}$ with $\left|{S}\right|\le s$ is greater than (1 − ϵ)ds goes to one as dimensions of A grows. Implying that A is the adjacency matrix of a (s, d, ϵ)-expander graph.

Corollary 4.1. Given d, s, n, N ∈ ℕ, and $ϵ\in (0,\frac{1}{2})$, for $d\ge \frac{c_{d}log(N/s)}{ϵlogs}$ and $n\ge \frac{c_{n}slog(N/s)}{{ϵ}^2}$, with c_d, c_n > 0. Let an n × N matrix A be drawn from ${B}$, in the proportional growth asymptotics

$\begin{array}{ll}Prob\left(\left|A_s\right|\le (1-ϵ)ds\right)\to 0.& (21)\end{array}$

Proof: It suffice to focus on the exponent of (16), more precisely on the bound of Ψ_N(s, d, ϵ) in (18), i.e.,

$\begin{array}{ll}-\frac{s}{N}log\left(\frac{s}{N}\right)+\frac{s}{N}-\frac{\tau \left(1-ϵ\right)d}{2log2}\frac{s}{N}log\left(\frac{s}{2}\right)+o\left(N\right).& (22)\end{array}$

We can ignore the o(N) term as this goes to zero as N grows, and show that the remaining sum is negative. The remaining sum is

$\begin{array}{l}-\frac{s}{N}log\left(\frac{s}{N}\right)+\frac{s}{N}-\frac{\tau \left(1-ϵ\right)d}{2log2}\frac{s}{N}log\left(\frac{s}{2}\right)\\ =\frac{s}{N}\left[-log\left(\frac{s}{N}\right)+1-\frac{\tau \left(1-ϵ\right)d}{2log2}log\left(\frac{s}{2}\right)\right].& (23)\end{array}$

Hence, we can further focus on the sum in the square brackets, and find conditions on d that will make it negative. We require

$\begin{array}{l}-log\left(\frac{s}{N}\right)+1-\frac{\tau \left(1-ϵ\right)d}{2log2}log\left(\frac{s}{2}\right)<0,\\ \Rightarrow d>\frac{2log2\left(log\left(N/s\right)+1\right)}{\tau \left(1-ϵ\right)log\left(s/2\right)}& (24)\end{array}$

$\begin{array}{ll}\iff d\ge \frac{c_{d}log(N/s)}{ϵlogs},\exists c_d>0.& (25)\end{array}$

Recall τ = ηβ⁻¹(β−1) and β is a function of ϵ, with β(ϵ) ≈ 1+ϵ. Therefore, τ is a function of ϵ, and $\tau (ϵ)={O}(ϵ)$, hence there exists a c_d > 0 for (25) to hold.

With regards to the complexity of n, we go back to the right hand side (RHS) of (24) and we substitute $\frac{C_{d}log(N/s)}{ϵlog(s/2)}$ with C_d > 0 for d in the RHS of (24) to get the following.

$\begin{array}{ll}-log\left(\frac{s}{N}\right)+1-\frac{C_d\tau (1-ϵ)log(N/s)}{2ϵlog2}<0.& (26)\end{array}$

Now we assume $n=\frac{C_{n}slog(N/s)}{{ϵ}^2}$ with C_n > 0 for n and substitute this in (26) to get the following.

$\begin{array}{l}-log(\frac{s}{N})+1-\frac{C_d\tau (1-ϵ)ϵn}{2C_{n}slog2}<0,\\ \Rightarrow n>\frac{2C_{n}slog2(log(N/s)+1)}{C_d\tau ϵ(1-ϵ)}& (27)\end{array}$

$\begin{array}{ll}\iff n\ge \frac{c_{n}slog(N/s)}{{ϵ}^2},\exists c_n>0.& (28)\end{array}$

Again since $\tau (ϵ)={O}(ϵ)$, hence there exists a c_n > 0 for (28) to hold. The bound of n in (28) agrees with our earlier assumption, thus concluding the proof.□

5. Discussion

5.1. Comparison to Other Constructions

In addition to being the first probabilistic construction of adjacency matrices of expander graphs with such a small degree, quantitatively our results compares favorably to existing probabilistic constructions.We use the standard tool of phase transitions to compare our construction to the construction proposed in Berinde [27] and those proposed in Buhrmanet al. [26]. The phase transition curve ρ_BT(δ; d, ϵ) we derived in Lemma 3.1 is the ρ that solves the following equation.

$\begin{array}{l}-\rho log(\delta \rho )+\rho -\frac{\tau (1-ϵ)d\rho log(\delta \rho )}{2log2}-\frac{\tau ϵ(1-ϵ)d}{2c_{n}log2}\\ +\frac{\tau (1-ϵ)d\rho }{2}=0,& (29)\end{array}$

where c_n > 0 is as in (28). Equation (29) comes from taking the limit, in the proportional growth asymptotics, of the bound in (18), setting that to zero and simplifying. Similarly, for any ${S}$ with $\left|{S}\right|\le s$, Berinde [27] derived the following bound on the set of neigbors of ${S}$, i.e., A_s.

$\begin{array}{ll}\text{Prob}\left(\left|A_s\right|\le \left(1-ϵ\right)ds\right)<(\genfrac{}{}{0.0pt}{}{N}{s})(\genfrac{}{}{0.0pt}{}{ds}{ϵds}){\left(\frac{ds}{n}\right)}^{ϵds}.& (30)\end{array}$

We then express the bound in (30) as the product of a polynomial term and an exponential term as we did for the right hand side of (16). A bound of the exponent is carefully derived as in the derivations above. We set the limit, in the proportional growth asymptotics, of this bound to zero and simplify to get the following.

$\begin{array}{ll}(ϵd-1)log\rho -log\delta +(1+ϵd)-ϵdlog(ϵ/d)=0.& (31)\end{array}$

We refer to the ρ that solves (31) as the phase transition for the construction proposed by Berinde [27] and denote this ρ (the phase transition function) as ρ_BI(δ; d, ϵ). Another probabilistic construction was proposed by Burhman et al. [26]. In conforminty with the notation used in this manuscript their bound is equivalent to the following, also stated in a similar form by Indyk and Razenshteyn [23].

$\begin{array}{ll}\text{Prob}\left(\left|A_s\right|\le (1-ϵ)ds\right)<\left(\begin{array}{c}N\\ s\end{array}\right){\left(\frac{\nu ϵn}{ds}\right)}^{-ϵds},& (32)\end{array}$

where ν > 0. We again express the bound in (32) as the product of a polynomial term and an exponential term. A bound of the exponent is carefully derived as in the derivations above. We set the limit, in the proportional growth asymptotics, of this bound to zero and simplify to get the following.

$\begin{array}{ll}(ϵd-1)log\rho -log\delta +1-ϵdlog(\nu ϵ/d)=0.& (33)\end{array}$

Similarly, we refer to the ρ that solves (33) as the phase transition for the construction proposed by Burhman et al. [26] and denote this ρ as ρ_BM(δ; d, ϵ). We compute numerical solutions to (29), (31), and (33) to derive the phase transitions, for the existence (probabilistic construction) of expanders, ρ_BT(δ; d, ϵ), ρ_BI(δ; d, ϵ), and ρ_BM(δ; d, ϵ) respectively. These are plotted in the left panel of Figure 1. It is clear that our construction has a much higher phase transition than the others, as expected from our previous results, see Figure 9 in Bah and Tanner [1]. Recall that the phase transition curves in these plots depict construction of adjacency matrices of (s, d, ϵ)-expanders with high probability for ratios of s, n, and N (since ρ: = s/n, and δ: = n/N) below the curve; and the failure to construct adjacency matrices of (s, d, ϵ)-expanders with high probability for ratios of s, n, and N above the curve. Essentially, the larger the area under the curve the better.

FIGURE 1

Figure 1. Plots of phase transitions for probabilistic construction of expanders, with fixed d = 2⁵, ϵ = 1/6, and δ ∈ [10⁻⁶, 1] on a logarithmically spaced grid of 100 points. (Left) A comparison of ρ_BT(δ; d, ϵ), ρ_BI(δ; d, ϵ), and ρ_BM(δ; d, ϵ), where c_n = 2. (Right) A comparison of ρ_BT(δ; d, ϵ) denoted as ρ_BT2(δ; d, ϵ) to our previous ρ_BT(δ; d, ϵ) denoted as ρ_BT1(δ; d, ϵ) in Bah and Tanner [1] with different values of N, (i.e., 2¹⁰, 2¹², and 2²⁰) and c_n = 2/3.

Remark 5.1. It is easy to see that ρ_BI(δ; d, ϵ) is a special case of ρ_BM(δ; d, ϵ) since the two phase transitions will coincide, or equivalently (31) and (33) will be the same, when ν = e⁻¹. One could argue that Berinde's derivation in [27] suffers from over counting.

Given that this work is an improvement of our work in Bah and Tanner [1] in terms of simplicity in computing ρ_BT(δ; d, ϵ), for completeness we compare our new phase transition ρ_BT(δ; d, ϵ) denoted as ρ_BT2(δ; d, ϵ) to our previous ρ_BT(δ; d, ϵ) denoted as ρ_BT1(δ; d, ϵ) in the right panel of Figure 1. Each computation of ρ_BT1(δ; d, ϵ) requires the specification of N, which is not needed in the computation of ρ_BT2(δ; d, ϵ), hence the simplification. However, the simplification led to a lower phase transition as expected, which is confirmed by the plots in the right panel of Figure 1.

Remark 5.2. These simulations also inform us about the size of c_n. See from the plots of ρ_BT(δ; d, ϵ) and ρ_BT2(δ; d, ϵ) that the smaller the value of c_n the higher the phase transition but since ρ_BT2(δ; d, ϵ) has to be a lower bound of ρ_BT1(δ; d, ϵ), for values of c_n much smaller than 2/3, the lower bound will fail to hold. This informed the choice of c_n = 2 in the plot of ρ_BT(δ; d, ϵ) in the left panel of Figure 1.

5.2. Implications for Combinatorial Compressed Sensing

When the sensing matrices are restricted to the sparse binary matrices considered in this manuscript, compressed sensing is usually referred to as combinatorial compressed sensing a term introduced in Berinde et al. [18] and used extensively in Mendoza-Smith and Tanner [14] and Mendoza-Smith et al. [15]. In this setting, compressed sensing is more-or-less equivalent to linear sketching. The implications of our results on combinatorial compressed sensing are two-fold. One is on the ℓ₁-norm RIP, we donate as RIP-1; while the second is in the comparison of performance of recovery algorithms for combinatorial compressed sensing.

5.2.1. RIP-1

As can be seen from (2), the recovery errors in compressed sensing depend on the RIC, i.e., δ_s. The following lemma deduced from Theorem 1 of Berinde et al.[18] shows that a scaled A drawn from 𝔼 have RIP with δ_s = 2ϵ.

Lemma 5.1. Consider $ϵ\in (0,\frac{1}{2})$ and let A be drawn from 𝔼, then Φ = A/d satisfies the following RIP-1 condition

$\begin{array}{ll}(1-2ϵ)\Vert \text{x}{\Vert }_1\le \Vert \Phi \text{x}{\Vert }_1\le \Vert \text{x}{\Vert }_1,\forall s–sparse\text{ x}.& (34)\end{array}$

The interested reader is referred to the proof of Theorem 1 in Berinde et al. [18] for the proof of this lemma. Key to the holding of Lemma 5.1 is the existence of (s, d, ϵ)-expander graphs, hence one can draw corollaries from our results on this.

Corollary 5.1. Consider $ϵ\in (0,\frac{1}{2})$ and let d, s, n, N ∈ ℕ. In the proportional growth asymptotics with a random draw of an n × N matrix A from ${B}$, the matrix Φ: = A/d has RIP-1 with probability approaching 1 exponentially, if

$\begin{array}{ll}d={O}\left(\frac{log(N/s)}{ϵlogs}\right),\text{ and }n={O}\left(\frac{slog(N/s)}{{ϵ}^2}\right).& (35)\end{array}$

Proof: Note that the upper bound of (34) holds trivially for any Φ = A/d where A has d ones per column, i.e., $\text{A}\in {B}$. But for the lower bound of (34) to hold for any Φ = A/d, we need A to be an (s, d, ϵ)-expander matrix, i.e., A ∈ 𝔼. Note that the event |A_s| ≥ (1 − ϵ)ds is equal to the event ||A${S}$x||₁ ≥ (1 − 2ϵ)d||x|₁, which is equivalent to $\Vert {\Phi }_{{S}}\text{x}{\Vert }_1\ge (1-2ϵ)\Vert \text{x}{|}_1$, for a fixed ${S}$, with

$\left|{S}\right|\le s$. For A to be in 𝔼, we need expansion for all sets ${S}$, with $\left|{S}\right|\le s$, i.e., A∈𝔼. The key thing to remember is that

$\begin{array}{l}\text{Prob}\left(\Vert \Phi \text{X}{\Vert }_1\ge (1-2є)\Vert \text{X}{|}_1\right)\\ =\text{Prob}\left(\left|A_s\right|\ge (1-є)ds\right),\text{ for all }\{{S} : |{S}|\le s\}.& (36)\end{array}$

The probability in (36) going to 1 exponentially in the proportional growth asymptotics, i.e., the existence of A∈𝔼 with parameters as given in (35), is what is stated in Theorem 3.1. Therefore, the rest of the proof follows from the proof of Theorem 3.1, hence concluding the proof of the corollary□.

Notably, Lemma 5.1 holds with Φ having much smaller number of non-zeros per column due to our construction. More over, we can derive sampling theorems for which Lemma 5.1 holds as thus.

Corollary 5.2. Fix $ϵ\in (0,\frac{1}{2})$ and let d, s, n, N ∈ ℕ. In the proportional growth asymptotics, for any ρ < (1−γ)ρ_BT(δ; d, ϵ) and γ > 0, a random draw of A from ${B}$ implies Φ: = A/d has RIP-1 with probability approaching 1 exponentially.

Proof. The proof of this corollary follows from the proof of Corollary 5.1 above, and it is related to the proof of Lemma 3.1 as the proof of Corollary 5.1 is to the proof of Theorem 3.1. The details of the proof is thus skipped.□

5.2.2. Performance of Algorithms

We wish to compare the performance of selected combinatorial compressed sensing algorithms in terms of the possible problem sizes (s, n, N) that these algorithms can reconstruct sparse/compressible signals/vectors up to their respective error guarantees. The comparison is typically done in the framework of phase transitions, which depict a boundary curve where ratios of problems sizes above this curve are recovered with probability approaching 0 exponentially; while problems sizes below the curve are recovered with probability approaching 1 exponentially. The list of combinatorial compressed sensing algorithms includes Expander Matching Pursuit (EMP) [32], Sparse Matching Pursuit [33], Sequential Sparse Matching Pursuit (SSMP) [13], Left Degree Dependent Signal Recovery (LDDSR) [11], Expander Recovery (ER) [12], Expander Iterative Hard-Thresholding (EIHT) [19, Section 13.4], and Expander ℓ₀-decoding (ELD) with both serial and parallel versions [14]. For reason similar to those used in Bah and Tanner [1] and Mendoza-Smith and Tanner [14], we selected out of this list four of the algorithms: (i) SSMP, (ii) ER, (ii) EIHT, (iv) ELD. Descriptions of these algorithms is skipped here but the interested reader is referred to the original papers or their summarized details in Bah and Tanner [1] and Mendoza-Smith and Tanner [14]. We were also curious as to how ℓ₁-minimization's performance compares to these selected combinatorial compressed sensing algorithms, since ℓ₁-minimization (ℓ₁-min) can be used to solve the combinatorial problem solved by these algorithms, see [18, Theorem 3].

The phase transitions are based on conditions on the RIC of the sensing matrices used. Consequent to Lemma 5.1, this becomes conditions on the expansion coefficient (i.e., ϵ) of the underlying (s, d, ϵ)-expander graphs of the sparse sensing matrices used. Where this condition on ϵ is not explicitly given it is easily deducible from the recovery guarantees given for each algorithms. The conditions are summarized in the table below.

The theoretical values are what will be found in the reference given in the table; while the computational values are what we used in our numerical experiments to compute the phase transition curves of the algorithms. The value for e was set to be 10⁻¹⁵, to make the ϵ_k as large as possible under the given condition. With these values we computed the phase transitions in Figure 2.

FIGURE 2

Figure 2. Phase transitions plots of algorithms with fixed d = 2⁵, ϵ as in the fourth column of Table 1 with e = 10⁻¹⁵, c_n = 2 and δ ∈ [10⁻⁶, 1] on a logarithmically spaced grid of 100 points. (Left) k = 3s for ρ_SSMP(δ; d, ϵ). (Right) k = ⌈(2+e)s⌉ for ρ_SSMP(δ; d, ϵ).

TABLE 1

Table 1. Recovery conditions on ϵ.

The two figures are the same except for the different sparsity value used. The performance of the algorithms in this framework are thus ranked as follows: ELD, ER, ℓ₁-min, EIHT, and SSMP.

Remark 5.3. We point out that there are many way to compare performance of algorithms, this is just one way. For instance, we can compare runtime complexities or actual computational runtimes as in Mendoza-Smith and Tanner [14]; phase transitions of different probabilities, here the probability of recovery is 1 but this could be set to something else, like 1/2 in the simulations in Mendoza-Smith and Tanner [14]; one could also compare number of iterations and iteration cost as was also done in Mendoza-Smith and Tanner [14]. We further point out that such comaprisons could be conducted with real-world data simulations, as was done in Mendoza-Smith and Tanner [14].

6. Proofs of Theorems, Lemmas, and Propositions

6.1. Theorem 3.1

The proof of the theorem follows trivially from Corollary 4.1. Based on (21) of Corollary 4.1 we deduce that A ∈ 𝔼 with probability approaching 1 exponentially with $d\ge \frac{c_{d}log(N/s)}{ϵlogs}$ and $n\ge \frac{c_{n}slog(N/s)}{{ϵ}^2}$, with c_d, c_n > 0, hence concluding the proof.□

6.2. Lemma 3.1

The phase transition curve ρ_BT(δ; d, ϵ) is based the bound of the exponent of (16), which is

$\begin{array}{ll}-\frac{s}{N}log\left(\frac{s}{N}\right)+\frac{s}{N}-\frac{\tau (1-ϵ)d}{2log2}\frac{s}{N}log\left(\frac{s}{2}\right)+o(N).& (37)\end{array}$

In the propotional growth asymptotics (s, n, N) → ∞ while s/n → ρ ∈ (0, 1) and n/N → δ ∈ (0, 1). This implies that o(N) → 0 and (37) becomes

$\begin{array}{l}-\rho log(\delta \rho )+\rho -\frac{\tau (1-ϵ)d\rho log(\delta \rho )}{2log2}\\ -\frac{\tau ϵ(1-ϵ)d}{2c_{n}log2}+\frac{\tau (1-ϵ)d\rho }{2},& (38)\end{array}$

where c_n > 0 is as in (28). If (38) is negative then as the problem size grows we have

$\begin{array}{ll}\text{Prob}\left(\left|A_s\right|\le \left(1-ϵ\right)ds\right)\to 0.& (39)\end{array}$

Therefore, setting (38) to zero and solving for ρ gives us a critical ρ below which (38) is negative and positive above it. The critical ρ is the phase transition ρ, i.e., ρ_BT(δ; d, ϵ), where below ρ_BT(δ; d, ϵ) is parameterized by the γ in the lemma. This concludes the proof.□

6.3. Lemma 4.1

By the dyadic splitting proposed in Bah and Tanner [1], we let $A_s=A_{\lceil \frac{s}{2}\rceil }^1\cup A_{\lfloor \frac{s}{2}\rfloor }^2$ such that $\left|A_s\right|=|A_{\lceil \frac{s}{2}\rceil }^1\cup A_{\lfloor \frac{s}{2}\rfloor }^2|$ and therefore

$\begin{array}{ll}\text{Prob}(\left|A_s\right|\le a_s)=\text{Prob}\left(|A_{\lceil \frac{s}{2}\rceil }^1\cup A_{\lfloor \frac{s}{2}\rfloor }^2|\le a_s\right)& (40)\end{array}$

$\begin{array}{ll}={\displaystyle \sum _{l_s=d}^{a_s}}\text{Prob}(|A_{\lceil \frac{s}{2}\rceil }^1\cup A_{\lfloor \frac{s}{2}\rfloor }^2|=l_s)& (41)\end{array}$

In (41) we sum over all possible events, i.e., all possible sizes of l_s. In line with the splitting technique, we simplify the probability to the product of the probabilities of the cardinalities of $\left|A_{\lceil \frac{s}{2}\rceil }^1\right|$ and $\left|A_{\lfloor \frac{s}{2}\rfloor }^2\right|$ and their intersection. Using the definition of P_n(·) in Lemma 8.2 (Appendix), thus leads to the following.

$\begin{array}{l}\text{Prob}(|A_s|\le a_s)\\ ={\displaystyle \sum _{l_s=2d}^{a_s}}{\displaystyle \sum _{l_{\lceil \frac{s}{2}\rceil }^1=d}^{a_{\lceil \frac{s}{2}\rceil }}}{\displaystyle \sum _{l_{\lfloor \frac{s}{2}\rfloor }^2=d}^{a_{\lfloor \frac{s}{2}\rfloor }}}{P}_n\left(l_s,l_{\lceil \frac{s}{2}\rceil }^1,l_{\lfloor \frac{s}{2}\rfloor }^2\right)\\ \times \text{Prob}\left(\left|A_{\lceil \frac{s}{2}\rceil }^1\right|=l_{\lceil \frac{s}{2}\rceil }^1\right)\text{Prob}\left(\left|A_{\lfloor \frac{s}{2}\rfloor }^2\right|=l_{\lfloor \frac{s}{2}\rfloor }^2\right).& (42)\end{array}$

In a slight abuse of notation we write ${\displaystyle \sum _{\underset{{ }_{j\in [m]}}{{ }_{l^j}}}}$ to denote applying the sum m times. We also drop the limits of the summation indices henceforth. Now we use Lemma 8.1 in Appendix to simplify (42) as follows.

$\begin{array}{l}\underset{{ }_{j_1\in [q_0]}}{\sum _{l_{Q_0}^{j_1}}}\underset{{ }_{j_2\in [q_1]}}{\sum _{l_{Q_1}^{j_2}}}\underset{{ }_{j_3\in [r_1]}}{\sum _{l_{R_1}^{j_3}}}{\text{P}}_n(l_{Q_0}^{j_1},l_{\lceil \frac{Q_0}{2}\rceil }^{2j_1-1},l_{\lfloor \frac{Q_0}{2}\rfloor }^{2j_1}) \times \\ \begin{array}{c} {\displaystyle \underset{j_2=1}{\overset{q_1}{{{\displaystyle \prod }}^{\text{}}}}}\text{Prob}\left(\right|A_{Q_1}^{j_2}|=l_{Q_1}^{j_2})\underset{q_1+r_1}{\overset{j_3=q_1+1}{{{\displaystyle \prod }}^{\text{}}}}\text{Prob}\left(\right|A_{R_1}^{j_3}|=l_{R_1}^{j_3})\end{array}& (43)\end{array}$

Now we proceed with the splitting—note (43) stopped only at the first level. At the next level, the second, we will have q₂ sets with Q₂ columns and r₂ sets with R₂ columns which leads to the following expression.

$\begin{array}{l}\begin{array}{l}{\displaystyle \underset{{ }_{j_1\in [q_0]}}{\sum _{l_{Q_0}^{j_1}}}}{\displaystyle \underset{{ }_{j_2\in [q_1]}}{\sum _{l_{Q_1}^{j_2}}}}{\displaystyle \underset{{ }_{j_3\in [r_1]}}{\sum _{l_{R_1}^{j_3}}}}{\text{P}}_n(l_{Q_0}^{j_1},l_{\lceil \frac{Q_0}{2}\rceil }^{2j_1-1},l_{\lfloor \frac{Q_0}{2}\rfloor }^{2j_1})\\ [{\displaystyle \underset{{ }_{j_4\in [q_2]}}{\sum _{l_{Q_2}^{j_4}}}}{\displaystyle \underset{{ }_{j_5\in [r_2]}}{\sum _{l_{R_2}^{j_5}}}}{\text{P}}_n(l_{Q_1}^{j_2},l_{\lceil \frac{Q_1}{2}\rceil }^{2j_2-1},l_{\lfloor \frac{Q_1}{2}\rfloor }^{2j_2}) \times {\text{P}}_n(l_{R_1}^{j_3},l_{\lceil \frac{R_1}{2}\rceil }^{2j_3-1},l_{\lfloor \frac{R_1}{2}\rfloor }^{2j_3})\end{array}\hfill \\ \begin{array}{c}\begin{array}{l} {\displaystyle \underset{j_4=1}{\overset{q_2}{{{{\displaystyle \prod }}^{\text{}}}^{\text{}}}}}\text{Prob}(|A_{Q_2}^{j_4}|=l_{Q_2}^{j_4}){\displaystyle \underset{j_5=q_2+1}{\overset{q_2+r_2}{{{{\displaystyle \prod }}^{\text{}}}^{\text{}}}}}\text{Prob}(|A_{R_2}^{j_5}|=l_{R_2}^{j_5})].\\ \end{array}\end{array}\hfill & (44)\end{array}$

We continue this splitting of each instance of Prob(·) for ⌈log₂s⌉−1 levels until reaching sets with single columns where, by construction, the probability that the single column has d non-zeros is one. Note that at this point we drop the subscripts j_i, as they are no longer needed. This process gives a complicated product of nested sums of P_n(·) which we express as

$\begin{array}{l}{\displaystyle \sum _{l_{Q_0}}}{\displaystyle \sum _{l_{Q_1}}}{\displaystyle \sum _{l_{R_1}}}{\text{P}}_n\left(l_{Q_0},l_{\lceil \frac{Q_0}{2}\rceil },l_{\lfloor \frac{Q_0}{2}\rfloor }\right)\\ [ {\displaystyle \sum _{l_{Q_2}}}{\displaystyle \sum _{l_{R_2}}}{\text{P}}_n\left(l_{Q_1},l_{\lceil \frac{Q_1}{2}\rceil },l_{\lfloor \frac{Q_1}{2}\rfloor }\right){\text{P}}_n\left(l_{R_1},l_{\lceil \frac{R_1}{2}\rceil },l_{\lfloor \frac{R_1}{2}\rfloor }\right) \\ [\times \cdots [{\displaystyle \sum _{l_{Q_{\lceil {log}_{2}s\rceil -1}}}}{\text{P}}_n\left(l_4,l_2,l_2\right){\text{P}}_n\left(l_3,l_2,d\right){\text{P}}_n\left(l_2,d,d\right) ]\cdots ].& (45)\end{array}$

Using the expression for P_n(·) in (133) of Lemma 8.2 (Appendix) we bound (45) by bounding each P_n(·) as in (134) with a product of a polynomial, π(·), and an exponential with exponent ψ_n(·).

$\begin{array}{l}{\displaystyle \sum _{l_{Q_0}}}{\displaystyle \sum _{l_{Q_1}}}{\displaystyle \sum _{l_{R_1}}}\pi \left(l_{Q_0},l_{\lceil \frac{Q_0}{2}\rceil },l_{\lfloor \frac{Q_0}{2}\rfloor }\right)e^{{\psi }_n\left(l_{Q_0},l_{\lceil \frac{Q_0}{2}\rceil },l_{\lfloor \frac{Q_0}{2}\rfloor }\right)}\times \\ [ {\displaystyle \sum _{l_{Q_2}}}{\displaystyle \sum _{l_{R_2}}}\pi \left(l_{Q_1},l_{\lceil \frac{Q_1}{2}\rceil },l_{\lfloor \frac{Q_1}{2}\rfloor }\right)\times \\ e^{{\psi }_n\left(l_{Q_1},l_{\lceil \frac{Q_1}{2}\rceil },l_{\lfloor \frac{Q_1}{2}\rfloor }\right)}\pi \left(l_{R_1},l_{\lceil \frac{R_1}{2}\rceil },l_{\lfloor \frac{R_1}{2}\rfloor }\right)·e^{{\psi }_n\left(l_{R_1},l_{\lceil \frac{R_1}{2}\rceil },l_{\lfloor \frac{R_1}{2}\rfloor }\right)}\times \\ [ \cdots \times [{\displaystyle \sum _{l_{Q_{\lceil {log}_{2}s\rceil -1}}}}\pi \left(l_4,l_2,l_2\right)\times e^{{\psi }_n\left(l_4,l_2,l_2\right)}\pi \left(l_3,l_2,d\right)\times \\ e^{{\psi }_n\left(l_3,l_2,d\right)}\pi \left(l_2,d,d\right)·e^{{\psi }_n\left(l_2,d,d\right)} ]\cdots ].& (46)\end{array}$

Using Lemma 8.4 in Appendix we maximize the ψ_n(·) and hence the exponentials in (46). We maximize each ψ_n(·) by choosing l_(·) to be a_(·). Then (46) will be upper bounded by the following.

$\begin{array}{l}{\displaystyle \sum _{l_{Q_0}}}{\displaystyle \sum _{l_{Q_1}}}{\displaystyle \sum _{l_{R_1}}}\pi \left(l_{Q_0},l_{\lceil \frac{Q_0}{2}\rceil },l_{\lfloor \frac{Q_0}{2}\rfloor }\right)e^{{\psi }_n\left(a_{Q_0},a_{\lceil \frac{Q_0}{2}\rceil },a_{\lfloor \frac{Q_0}{2}\rfloor }\right)}\times \\ [ {\displaystyle \sum _{l_{Q_2}}}{\displaystyle \sum _{l_{R_2}}}\pi \left(l_{Q_1},l_{\lceil \frac{Q_1}{2}\rceil },l_{\lfloor \frac{Q_1}{2}\rfloor }\right)\times e^{{\psi }_n\left(a_{Q_1},a_{\lceil \frac{Q_1}{2}\rceil },a_{\lfloor \frac{Q_1}{2}\rfloor }\right)}\\ \pi \left(l_{R_1},l_{\lceil \frac{R_1}{2}\rceil },l_{\lfloor \frac{R_1}{2}\rfloor }\right)·e^{{\psi }_n\left(a_{R_1},a_{\lceil \frac{R_1}{2}\rceil },a_{\lfloor \frac{R_1}{2}\rfloor }\right)} [\dots \times \\ [{\displaystyle \sum _{l_{Q_{\lceil {log}_{2}s\rceil -1}}}}\pi \left(l_4,l_2,l_2\right)\times e^{{\psi }_n\left(a_4,a_2,a_2\right)}\pi \left(l_3,l_2,d\right)e^{{\psi }_n\left(a_3,a_2,d\right)}\\ \pi \left(l_2,d,d\right)·e^{{\psi }_n\left(a_2,d,d\right)} ]\dots ].& (47)\end{array}$

We then factor the product of exponentials. This product becomes an exponential where exponent is the summation of the ψ_n(·), we will denote this exponent as ${\widetilde{\text{Ψ}}}_n(a_s,\dots ,a_2,d)$. Then (47) simplifies to the following.

$\begin{array}{l}{e}^{{\tilde{\Psi }}_n\left(a_s,\dots ,a_2,d\right)}·{\displaystyle \sum _{l_{Q_0}}}{\displaystyle \sum _{l_{Q_1}}}{\displaystyle \sum _{l_{R_1}}}\pi \left(l_{Q_0},l_{\lceil \frac{Q_0}{2}\rceil },l_{\lfloor \frac{Q_0}{2}\rfloor }\right)\times \\ [ {\displaystyle \sum _{l_{Q_2}}}{\displaystyle \sum _{l_{R_2}}}\pi \left(l_{Q_1},l_{\lceil \frac{Q_1}{2}\rceil },l_{\lfloor \frac{Q_1}{2}\rfloor }\right)\times \pi \left(l_{R_1},l_{\lceil \frac{R_1}{2}\rceil },l_{\lfloor \frac{R_1}{2}\rfloor }\right)\times [ \\ \dots \times \left[ {\displaystyle \sum _{l_{Q_{\lceil {log}_{2}s\rceil -1}}}}\pi \left(l_4,l_2,l_2\right)\pi \left(l_3,l_2,d\right)\pi \left(l_2,d,d\right)\right] \dots ] & (48)\end{array}$

We denote the factor multiplying the exponential term by Π(l_s, …, l₂, d), therefore we have the following bound.

$\begin{array}{ll}\text{Prob}(\left|A_s\right|\le a_s)\le \Pi (l_s,\dots ,l_2,d)·e^{({\tilde{\Psi }}_n(a_s,\dots ,a_2,d))},& (49)\end{array}$

where ${\widetilde{\text{Ψ}}}_n(a_s,\dots ,a_2,d)$ is exactly Ψ_n(a_s, …, a₂, d) given by (78) in [1, Proof of Theorem 1.6]. Consequently, we state the bound of ${\widetilde{\text{Ψ}}}_n(a_s,\dots ,a_2,d)$ and skip the proof, which is as thus.

$\begin{array}{ll}{\tilde{\Psi }}_n(a_s,\dots ,a_2,d)\le {\displaystyle \sum _{i\in \Omega }}\frac{s}{2i}{\psi }_i,\text{ for }\Omega ={\left\{2^j\right\}}_{j=0}^{{log}_2(s)-1},& (50)\end{array}$

where ψ_i is given by (8). The upper bound of Π(l_s, …, l₂, d) is given by the following proposition.

Proposition 6.1. Given l_s ≤ a_s, l_⌈s/2⌉ ≤ a_⌈s/2⌉, ⋯ , l₂ ≤ a₂, we have

$\begin{array}{ll}\Pi (l_s,\dots ,l_2,d)\le 2^{-3}·e^{\frac{1}{4}}·s^{\frac{3}{2}{log}_{2}s+\frac{9}{2}}·{(a_s)}^{{log}_{2}s-\frac{3}{2}}.& (51)\end{array}$

The proof of the proposition is found in section 6.4. Taking log of right hand side of (51) and then exponentiating the results yields

$\begin{array}{l}\Pi \left(l_s,\dots ,l_2,d\right)\\ \le \exp [\frac{1}{4}-3\text{log }2+\left(\frac{9}{2}+\frac{3}{2}{\text{log }}_{2}s\right)\text{log }s\\ +\left({\text{log }}_{2}s-\frac{3}{2}\right)\text{log}{a}_s]& (52)\end{array}$

$\begin{array}{ll}=2^{-3}{s}^{9/2}{e}^{1/4}·e^{\left[\frac{3}{2}{log}_{2}slogs+\left({log}_{2}s-\frac{3}{2}\right)log{a}_s\right]}& (53)\end{array}$

$\begin{array}{ll}=2^{-3}{s}^{9/2}{e}^{1/4}·e^{\left[\frac{3log2}{2}{log}_2^{2}s+\left({log}_{2}s-\frac{3}{2}\right)log{a}_s\right]}.& (54)\end{array}$

Combining (54) and (50) gives the following bound for (49)

$\begin{array}{l}\text{Prob}(|A_s|\le a_s)\le 2^{-3}{s}^{9/2}{e}^{1/4}·\exp [\frac{\log 8}{2}{\log }_2^{2}s\hfill \\ +({\log }_{2}s-\frac{3}{2})\log a_s+{\tilde{\Psi }}_n(a_s,\dots ,d)]\hfill & (55)\end{array}$

It follows therefore that $p_n(s,d)=2^{-3}{s}^{9/2}{e}^{1/4}$ as in (11) and the exponent in (55) is n · Ψ_n(a_s, …, a₁), which implies (12). This concludes the proof of the lemma.□

6.4. Proposition 6.1

By definition, from (48), we have

$\begin{array}{l}\prod (l_s,\dots ,l_2,d):={\displaystyle \sum _{l_{Q_0}}{\displaystyle \sum _{l_{Q_1}}{\displaystyle \sum _{l_{R_1}}\pi \left(l_{Q_0},l_{\lceil \frac{Q_0}{2}\rceil },l_{\lfloor \frac{Q_0}{2}\rfloor }\right)}}}\times \\ [{\displaystyle \sum _{l_{Q_2}}{\displaystyle \sum _{l_{R_2}}\pi \left(l_{Q_1},l_{\lceil \frac{Q_1}{2}\rceil },l_{\lfloor \frac{Q_1}{2}\rfloor }\right)}}\times \pi \left(l_{R_1},l_{\lceil \frac{R_1}{2}\rceil },l_{\lfloor \frac{R_1}{2}\rfloor }\right)·[\cdots \\ \times \left[{\displaystyle \sum _{l_{Q_{\lceil {\log }_{2}s\rceil -1}}}\pi \left(l_4,l_2,l_2\right)\pi \left(l_3,l_2,d\right)\pi \left(l_2,d,d\right)}\right]\cdots ].& (56)\end{array}$

From (138) we see that π(·) is maximized when all the three arguments are the same and using Corollary 8.1 we take largest possible arguments that are equal in the range of the summation. Before we write out the resulting bound for Π(l_s, …, l₂, d), we simplify notation by denoting π(x, x, x) as π(x), and noting that Q_{⌈log2s⌉−1} = 2. Therefore, the bound becomes the following.

$\begin{array}{l}\Pi \left(l_s,\dots ,l_2,d\right)\le {\displaystyle {{\displaystyle \sum }}^{}_{l_{Q_0}}}{\displaystyle {{\displaystyle \sum }}^{}_{l_{Q_1}}}{\displaystyle {{\displaystyle \sum }}^{}_{l_{R_1}}}\pi \left(l_{Q_0}\right)[{\displaystyle {{\displaystyle \sum }}^{}_{l_{Q_2}}}{\displaystyle {{\displaystyle \sum }}^{}_{l_{R_2}}}\pi \left(l_{Q_1}\right)\hfill \\ \begin{array}{l}\times \pi \left(l_{R_1}\right)\left[\cdots \left[{\displaystyle {{\displaystyle \sum }}^{}_{l_2}}\pi \left(l_4\right)\pi \left(l_3\right)\pi \left(l_2\right)\right]\cdots \right].\hfill \end{array}\hfill & (57)\end{array}$

Properly aligning the π(·) with their relevant summations simplifies the right hand side (RHS) of (57) to the following.

$\begin{array}{ll}\begin{array}{l}\begin{array}{l}\left[{\displaystyle {{\displaystyle \sum }}^{}_{l_{Q_0}}}\pi \left(l_{Q_0}\right)\right]\cdot \left[{\displaystyle {{\displaystyle \sum }}^{}_{l_{Q_1}}}\pi \left(l_{Q_1}\right)\right]·\left[{\displaystyle {{\displaystyle \sum }}^{}_{l_{R_1}}}\pi \left(l_{R_1}\right)\right]\times \cdots \\ \times \left[{\displaystyle {{\displaystyle \sum }}^{}_{l_4}}\pi \left(l_4\right)\right]·\left[{\displaystyle {{\displaystyle \sum }}^{}_{l_3}}\pi \left(l_3\right)\right]·\left[{\displaystyle {{\displaystyle \sum }}^{}_{l_2}}\pi \left(l_2\right)\right].\end{array}\hfill \end{array}& (58)\end{array}$

From (138) we have π (y) defined as

$\begin{array}{ll}\pi (y,y,y)={\left(\frac{5}{4}\right)}^2\sqrt{\frac{2\pi y\left(n-y\right)}{n}}<{\left(\frac{5}{4}\right)}^2\sqrt{2\pi y}.& (59)\end{array}$

We use the RHS of (59) to upper bound each term in (58), leading to the following bound.

$\begin{array}{l}\left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2{{\displaystyle \sum }}^{}_{l_{Q_1}}\sqrt{l_{Q_0}}\right]\cdot \left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2{{\displaystyle \sum }}^{}_{l_{Q_1}}\sqrt{l_{Q_1}}\right].\hfill \\ \left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2{{\displaystyle \sum }}^{}_{l_{R_1}}\sqrt{l_{R_1}}\right]\times \cdots \times \left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2{{\displaystyle \sum }}^{}_{l_4}\sqrt{l_4}\right]\times \hfill \\ \left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2{{\displaystyle \sum }}^{}_{l_3}\sqrt{l_3}\right]\cdot \left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2{{\displaystyle \sum }}^{}_{l_2}\sqrt{l_2}\right].\hfill & (60)\end{array}$

For each Q_i and R_i, i = 1, …, ⌈log₂ s⌉−2, which means we have ⌈log₂ s⌉ − 2 pairs plus one Q₀, hence (60) simplifies to the following.

$\begin{array}{l}{\left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2\right]}^{2\lceil {log}_{2}s\rceil -3}\left[{\displaystyle \sum _{l_{Q_0}}}\sqrt{l_{Q_0}}\right]·\left[{\displaystyle \sum _{l_{Q_1}}}\sqrt{l_{Q_1}}\right]\times \cdots \\ \times \left[{\displaystyle \sum _{l_3}}\sqrt{l_3}\right]·\left[{\displaystyle \sum _{l_2}}\sqrt{l_2}\right]& (61)\end{array}$

$\begin{array}{l}\le {\left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2\right]}^{2\lceil {log}_{2}s\rceil -3}·(q_0\sqrt{a_{Q_0}})·(q_1\sqrt{a_{Q_1}})\cdots \\ \times (q_{\lceil {log}_{2}s\rceil -2}\sqrt{a_3})·(r_{\lceil {log}_{2}s\rceil -2}\sqrt{a_2})& (62)\end{array}$

$\begin{array}{l}={\left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2\right]}^{2\lceil {log}_{2}s\rceil -3}·(q_0{q}_1{r}_1\cdots q_{\lceil {log}_{2}s\rceil -2}{r}_{\lceil {log}_{2}s\rceil -2})·\\ {(a_{Q_0}{a}_{Q_1}{a}_{R_1}\cdots a_3{a}_2)}^{1/2}.& (63)\end{array}$

From (61) to (62) we upper each sum by taking the largest possible value of l_(·), which is a_(·), and multiplied it with the total number terms in the summation given by Lemma 8.1 in Appendix. We did upper bound the following two terms of (63).

$\begin{array}{ll}{q}_0{q}_1{r}_1{q}_2{r}_2{q}_3{r}_3\cdots q_{\lceil {log}_{2}s\rceil -2}{r}_{\lceil {log}_{2}s\rceil -2}\le s^{{log}_{2}s-1},& (64)\end{array}$

$\begin{array}{l}{(a_{Q_0}{a}_{Q_1}{a}_{R_1}{a}_{Q_2}{a}_{R_2}{a}_{Q_3}{a}_{R_3}\cdots a_3{a}_2)}^{1/2}\le 2^{-1}·e^{\frac{1}{4}}·{(a_s)}^{{log}_{2}s-\frac{3}{2}}\times \\ s^{\frac{1}{2}{log}_{2}s+\frac{3}{2}}& (65)\end{array}$

Details of the derivation of the bounds (64) and (65) is in the Appendix. Using these bounds from (63) we have the following upper bound for Π(l_s, …, l₂, d).

$\begin{array}{l}\Pi (l_s,\dots ,l_2,d)\le {\left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2\right]}^{2\lceil {log}_{2}s\rceil -3}·\left(s^{{log}_{2}s-1}\right)\times \\ \left(2^{-1}·e^{\frac{1}{4}}·{(a_s)}^{{log}_{2}s-\frac{3}{2}}·s^{\frac{1}{2}{log}_{2}s+\frac{3}{2}}\right)& (66)\end{array}$

$\begin{array}{ll}\le 2^{-3}·e^{\frac{1}{4}}·{(a_s)}^{{log}_{2}s-\frac{3}{2}}·s^{\frac{3}{2}{log}_{2}s+\frac{9}{2}}.& (67)\end{array}$

From (66) to (67) we used the following upper bound.

$\begin{array}{l}{\left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2\right]}^{2\lceil {log}_{2}s\rceil -3}\le {\left[\sqrt{2\pi }{\left(\frac{5}{4}\right)}^2\right]}^{2({log}_{2}s+1)-3}\\ \le 4^{2{log}_{2}s-1}=2^{4{log}_{2}s-2}=2^{-2}{s}^4.& (68)\end{array}$

The bound (67) coincides with (51), hence concluding the proof.□

6.5. Theorem 4.2

The following lemma is a key input in this proof.

Lemma 6.1. Let 0 < α ≤ 1, and ε_n > 0 such that ε_n → 0 as n → ∞. Then for a_s < â_s,

$\begin{array}{ll}{a}_{2i}=2a_i+c{a}_i^2,\text{ for }c=-\beta n^{-1},& (69)\end{array}$

where

$\begin{array}{ll}\beta =\frac{1+\sqrt{1-4{(1-{\epsilon }_n)}^2(1-e^{-\alpha d})e^{-\alpha d}}}{2(1-{\epsilon }_n)(1-e^{-\alpha d})}.& (70)\end{array}$

The proof of the lemma is found in section 6.6. Recall from Theorem 4.1 that

$\begin{array}{l}{\Psi }_n(a_s,\dots ,a_1)=\frac{1}{n}[\frac{3\log 2}{2}{\log }_2^{2}s+({\log }_{2}s-\frac{3}{2})\log a_s+\hfill \\ {\displaystyle {\displaystyle \sum }_{i\in \Omega }}\frac{s}{2i}{\psi }_i], for\Omega ={\left\{2^j\right\}}_{j=0}^{{\log }_2(s)-1},\hfill & (71)\end{array}$

where ψ_i is given by the following expression

$\begin{array}{ll}(n-a_i)·{H}\left(\frac{a_{2i}-a_i}{n-a_i}\right)+a_i·{H}\left(\frac{a_{2i}-a_i}{a_i}\right)-n·{H}\left(\frac{a_i}{n}\right)& (72)\end{array}$

We use Lemma 6.1 to upper bound ψ_i in (72) away from zero from above as n → 0. We formalize this bound in the following proposition.

Proposition 6.2. Let η > 0 and β > 1 as defined in Lemma 6.1. Then

$\begin{array}{ll}{\psi }_i\le -a_i\eta \left(\beta -1\right){\beta }^{-1}{\left(1-\beta \frac{a_i}{n}\right)}^{-1}.& (73)\end{array}$

The proof of Proposition 6.2 is found in section 6.7. Using the bound of ψ_i in Proposition 6.2, we upper Ψ(a_s, …, a₁) as follows.

$\begin{array}{l}\Psi \left(a_s,\dots ,a_1\right)\le \frac{1}{n}[\frac{3\log 2}{2}{\log }_2^{2}s+\left({\log }_{2}s-\frac{3}{2}\right)\log a_s\hfill \\ -{\displaystyle \sum _{i\in \Omega }\frac{s}{2i}}·\frac{a_i\eta (\beta -1)}{\beta \left(1-\beta \frac{a_i}{n}\right)}],\hfill & (74)\end{array}$

$\begin{array}{l}\le -\frac{\eta (\beta -1)}{\beta }{\displaystyle \sum _{i\in \Omega }\left(\frac{s}{2i}\cdot \frac{a_i}{n}\right)}+\frac{1}{n}[\frac{3\log 2}{2}{\log }_2^{2}s+\hfill \\ \left({\log }_{2}s-\frac{3}{2}\right)\log a_s].\hfill & (75)\end{array}$

Then setting a_s = (1 − ϵ)ds and substituting in (75), the factor multiplying $\frac{1}{n}$ becomes

$\begin{array}{ll}\frac{3log2}{2}{log}_2^{2}s+\left({log}_{2}s-\frac{3}{2}\right)log\left[(1-ϵ)ds\right]& (76)\end{array}$

$\begin{array}{l}=\frac{3log2}{2}{log}_2^{2}s+log(1-ϵ){log}_{2}s+logd{log}_{2}s+\\ {log}_{2}slogs-\frac{3}{2}log\left[(1-ϵ)d\right]-\frac{3}{2}logs& (77)\end{array}$

$\begin{array}{l}=\frac{5log2}{2}{log}_2^{2}s+({log}_2(1-ϵ)-3/2)logs+logd{log}_{2}s+\\ log\left[{(1-ϵ)}^{-3/2}{d}^{-3/2}\right]& (78)\end{array}$

$\begin{array}{l}=\frac{5log2}{2}{log}_2^{2}s+logd{log}_{2}s+log\left[{(1-ϵ)}^{-3/2}{d}^{-3/2}\right]\\ +log{s}^{{log}_2(1-ϵ)-3/2}.& (79)\end{array}$

The last two terms of (79) become polynomial in s, d and ϵ, when exponentiated hence they are incorporated into p_n(s, d, ϵ) in (14), which means

$\begin{array}{l}{p}_n(s,d,ϵ)=p_n(s,d)\cdot \exp [\log \left[{(1-ϵ)}^{-3/2}{d}^{-3/2}\right]+\hfill \\ \log s^{{\log }_2(1-ϵ)-3/2}]\hfill & (80)\end{array}$

$\begin{array}{ll}=2^{-3}{s}^{9/2}{e}^{1/4}·{(1-ϵ)}^{-3/2}{d}^{-3/2}{s}^{{log}_2(1-ϵ)-3/2}& (81)\end{array}$

$\begin{array}{ll}=\frac{\sqrt[4]{e}·s^{{log}_2(1-ϵ)+3}}{\sqrt{2^6{(1-ϵ)}^3{d}^3}},& (82)\end{array}$

which is (14). The first two terms of (79) will grow faster than a polynomial in s, d and ϵ when exponentiated, hence they replace in (75), the factor multiplying $\frac{1}{n}$. Therefore, (79) is modified as thus

$\begin{array}{l}-\frac{\eta \left(\beta -1\right)}{\beta }{\displaystyle \sum _{i\in \Omega }}\left(\frac{s}{2i}·\frac{a_i}{n}\right)+\frac{1}{n}\left[\frac{5log2}{2}{log}_2^{2}s+logd{log}_{2}s\right]\\ =:{\Psi }_n(s,d,ϵ).& (83)\end{array}$

The factor $\sum _{i\in \Omega }\left(\frac{s}{2i}·\frac{a_i}{n}\right)$ in (83) is lower bounded as follows, see proof in section 6.8.

$\begin{array}{ll}{\displaystyle \sum _{i\in \Omega }}\left(\frac{s}{2i}·\frac{a_i}{n}\right)\ge \frac{{log}_2(s/2)}{2n}(1-ϵ)ds.& (84)\end{array}$

Using this bound in (83) gives (15), thus concluding the proof.□

6.6. Lemma 6.1

Recall that we have a formula for the expected values of the a_i as

$\begin{array}{ll}{\widehat{a}}_{2i}={\widehat{a}}_i\left(2-\frac{{\widehat{a}}_i}{n}\right)\text{ for }i\in {\left\{2^j\right\}}_{j=0}^{{log}_2(s)-1},& (85)\end{array}$

which follow a relatively simple formulas, and then the coupled system of cubics as

$\begin{array}{ll}{a}_{2i}^3-2a_i{a}_{2i}^2+2a_i^2{a}_{2i}-a_i^2{a}_{4i}=0\text{ for }i\in {\left\{2^j\right\}}_{j=0}^{{log}_2(s)-2},& (86)\end{array}$

for when the final a_s is constrained to be less than â_s. To simplify the notation of the indexing in (86), observe that if i = 2^j for a fixed j, then 2i = 2^j+1 and 4i = 2^j+2. Therefore, it suffice to use the index a_j, a_j+1, and a_j+2 rather than a_i, a_2i, and a_4i. Moving the second two terms in (86) to the right and dividing the quadratic multiples we get the relation

$\begin{array}{ll}\frac{a_{j+2}-2a_{j+1}}{a_{j+1}^2}=\frac{a_{j+1}-2a_j}{a_j^2},& (87)\end{array}$

which is the same expression on the right and left, but with j increased by one on the left. This implies that the fraction is independent of j, so

$\begin{array}{ll}\frac{a_{j+1}-2a_j}{a_j^2}=c,\Rightarrow a_{j+1}=a_j(2+c{a}_j),& (88)\end{array}$

for some constant c independent of j (though not necessarily of n). This is in fact the relation (85), if we set c to be equal to −1/n. One can then wonder what is the behavior of c if we fix the final a_s. Moreover, (88) is equivalent to

$\begin{array}{ll}c{a}_{j+1}+1={(c{a}_j+1)}^2,& (89)\end{array}$

which inductively leads to

$\begin{array}{ll}c{a}_l+1={(c{a}_0+1)}^{2^l},l>0,& (90)\end{array}$

so that one has a relation of the l^th stage in terms of the first stage. Note this does not require the a_s to be fixed, (90) is how one simply computes all a_l for l > 0 once one has a₀ and c. The point is that c to match the a_s one has to select c appropriately. So the way we calculate c is by knowing a₀ and a_s, then solving (90) for l = s. Unfortunately there is not an easy way to solve for c in (90) so we need to do some asymptotic approximation. Let's assume that a_l is close to â_l. So we do an asymptotic expansion in terms of the difference from â_l.

To simplify things a bit lets insert a₀ = d (since a₀ is a₁ in our standard notation) and then we insert what we know for â_l. For â_l we have c = −n⁻¹, see (85). We then have from (90) that

$\begin{array}{ll}{a}_l=c^{-1}{(cd+1)}^{2^l}-c^{-1}\text{ and }{â}_l=-n{(-d/n+1)}^{2^l}+n.& (91)\end{array}$

So if we write a_l = (1 − ε_n)â_l and consider the case of ε_n → 0 as n → 0. The point of this is that instead of working with â_l we can now work in terms of ε_n. Setting a_l = (1 − ε_n)â_l gives

$\begin{array}{ll}{c}^{-1}{(cd+1)}^{2^l}-c^{-1}=-n(1-{\epsilon }_n)\left[{(-d/n+1)}^{2^l}-1\right].& (92)\end{array}$

We now solve for c as a function of ε_n and d. As ε_n goes to zero we should have c converging to −n⁻¹.

Let αn = 2^l, for 0 < α ≤ 1 and c = −β(ε_n, d)/n, then, dropping the argument of β(·, ·), (92) becomes

$\begin{array}{ll}\frac{n}{\beta }{\left(1-\frac{\beta d}{n}\right)}^{\alpha n}-\frac{n}{\beta }=n\left(1-{\epsilon }_n\right){\left(1-\frac{d}{n}\right)}^{\alpha n}-n\left(1-ϵ\right).& (93)\end{array}$

Multiplying through by β/n and performing a change of variables of k = αn, (93) becomes

$\begin{array}{ll}{\left(1-\frac{\alpha \beta d}{k}\right)}^k-1=\beta \left(1-{\epsilon }_n\right){\left(1-\frac{\alpha d}{k}\right)}^k-\beta \left(1-{\epsilon }_n\right).& (94)\end{array}$

The left hand side of (94) simplifies to

$\begin{array}{ll}{e}^{-\alpha \beta d}-1-\frac{e^{-\alpha \beta d}{\alpha }^2{\beta }^2{d}^2}{2k}+{O}\left(k^{-2}\right).& (95)\end{array}$

The right hand side of (94) simplifies to

$\begin{array}{ll}\beta (1-{\epsilon }_n)e^{-\alpha d}-\beta (1-{\epsilon }_n)-\frac{\beta (1-{\epsilon }_n)e^{-\alpha d}{\alpha }^2{d}^2}{2k}+{O}(k^{-2}).& (96)\end{array}$

Matching powers of k in (95) and (96) for k⁰ and k⁻¹ yields the following.

$\begin{array}{ll}{e}^{-\alpha \beta d}-1=\beta (1-{\epsilon }_n)e^{-\alpha d}-\beta (1-{\epsilon }_n),\text{ and}& (97)\end{array}$

$\begin{array}{ll}{\alpha }^2{\beta }^2{d}^2{e}^{-\alpha \beta d}=\beta (1-{\epsilon }_n){\alpha }^2{d}^2{e}^{-\alpha d}.& (98)\end{array}$

Both of which respectively simplify to the following.

$\begin{array}{ll}{e}^{-\alpha \beta d}-1=\beta (1-{\epsilon }_n)\left(e^{-\alpha d}-1\right),\text{ and}& (99)\end{array}$

$\begin{array}{ll}\beta e^{-\alpha \beta d}=(1-{\epsilon }_n)e^{-\alpha d}.& (100)\end{array}$

Multiply (99) by β and subtract the two equations, (99) and (100), to get

$\begin{array}{ll}(1-{\epsilon }_n)\left(1-e^{-\alpha d}\right){\beta }^2-\beta +(1-{\epsilon }_n)e^{-\alpha d}=0.& (101)\end{array}$

This yields

$\begin{array}{ll}\beta =\frac{1\pm \sqrt{1-4{(1-{\epsilon }_n)}^2(1-e^{-\alpha d})e^{-\alpha d}}}{2(1-{\epsilon }_n)(1-e^{-\alpha d})}.& (102)\end{array}$

To be consistent with what c ought to be as ε_n → 0, we choose

$\begin{array}{ll}\beta ({\epsilon }_n,d)=\frac{1+\sqrt{1-4{(1-{\epsilon }_n)}^2(1-e^{-\alpha d})e^{-\alpha d}}}{2(1-{\epsilon }_n)(1-e^{-\alpha d})},& (103)\end{array}$

as required – concluding the proof.□

6.7. Proposition 6.2

We use Lemma 6.1 to express ψ_i in (72) as follows

$\begin{array}{l}{\psi }_i=-n\left[{H}\left(\frac{a_i}{n}\right)-{H}\left(\frac{a_i+c{a}_i^2}{n-a_i}\right)\right]\\ +a_i\left[{H}\left(\frac{a_i+c{a}_i^2}{a_i}\right)-{H}\left(\frac{a_i+c{a}_i^2}{n-a_i}\right)\right]& (104)\end{array}$

$\begin{array}{l}=-n\left[{H}\left(\frac{a_i}{n}\right)-{H}\left(\frac{a_i}{n}·\frac{1+c{a}_i}{1-\frac{a_i}{n}}\right)\right]\\ +a_i\left[{H}\left(1+c{a}_i\right)-{H}\left(\frac{a_i}{n}·\frac{1+c{a}_i}{1-\frac{a_i}{n}}\right)\right].& (105)\end{array}$

Note that for regimes of small s/n considered

$\begin{array}{ll}-\frac{\beta }{n}=c\le -\frac{1}{n},\Rightarrow c{a}_i\le -\frac{a_i}{n},\text{ and }1+c{a}_i\le 1-\frac{a_i}{n}.& (106)\end{array}$

We need the following expressions for the Shannon entropy and it's first and second derivatives

$\begin{array}{ll}{H}(z)=-zlogz-(1-z)log(1-z),& (107)\end{array}$

$\begin{array}{ll}{{H}}^{\prime }\left(z\right)=log\left(\frac{1-z}{z}\right),\text{ and}& (108)\end{array}$

$\begin{array}{ll}{{H}}^{″}(z)=-\frac{1}{z(1-z)}.& (109)\end{array}$

But also ${H}(z)={H}(1-z)$ due to the symmetry about z = 1/2. Similarly, ${{H}}^{″}(z)$ is symmetric about z = 1/2; while ${{H}}^{\prime }(z)$ is anti-symmetric, i.e., ${{H}}^{\prime }(z)=-{{H}}^{\prime }(1-z)$. Using the symmetry of ${H}(z)$ we rewrite ψ_i in (105) as follows.

$\begin{array}{l}{\psi }_i=-n\left[{H}\left(\frac{a_i}{n}\right)-{H}\left(\frac{a_i}{n}·\frac{1+c{a}_i}{1-\frac{a_i}{n}}\right)\right]\\ +a_i\left[{H}\left(-c{a}_i\right)-{H}\left(\frac{a_i}{n}·\frac{1+c{a}_i}{1-\frac{a_i}{n}}\right)\right].& (110)\end{array}$

From (106), we deduce the following ordering

$\begin{array}{ll}\frac{a_i}{n}·\frac{1+c{a}_i}{1-\frac{a_i}{n}}\le \frac{a_i}{n}\le -c{a}_i\le 1/2.& (111)\end{array}$

The last equality in (111) follows from that we assume 2βa_i ≤ n, where 1 < β < 2 but very close to 1. This is a valid constraint on the cardinality of the supports a_i. To simplify notation, let $x_1=\frac{a_i}{n}·\frac{1+c{a}_i}{1-\frac{a_i}{n}}$, $x_2=\frac{a_i}{n}$, and x₃ = −ca_i, which implies that x₁ ≤ x₂ ≤ x₃ ≤ 1/2. Therefore, from (110), we have

$\begin{array}{ll}{\psi }_i=-n\left[{H}(x_2)-{H}(x_1)\right]+a_i\left[{H}(x_3)-{H}(x_1)\right]& (112)\end{array}$

$\begin{array}{l}=-n\left[\mathscr{H}\left(x_2\right)-\mathscr{H}\left(x_1\right)\right]+a_i[\mathscr{H}\left(x_3\right)-\mathscr{H}\left(x_2\right)\\ +\mathscr{H}\left(x_2\right)-\mathscr{H}\left(x_1\right)]& (113)\end{array}$

$\begin{array}{ll}=-(n-a_i)\left[{H}(x_2)-{H}(x_1)\right]+a_i\left[{H}(x_3)-{H}(x_2)\right]& (114)\end{array}$

$\begin{array}{ll}\le -(n-a_i)(x_2-x_1){{H}}^{\prime }(x_2)+a_i(x_3-x_2){{H}}^{\prime }(x_2)& (115)\end{array}$

$\begin{array}{ll}=\left[a_i(x_3-x_2)-(n-a_i)(x_2-x_1)\right]{{H}}^{\prime }(x_2).& (116)\end{array}$

Observe that the expression in the square brackets on the right hand side of (116) is zero, which implies that

$\begin{array}{ll}{a}_i(x_3-x_2)=(n-a_i)(x_2-x_1).& (117)\end{array}$

This is very easy to check by substituting the values of x₁, x₂, and x₃. So instead of bound (115), we alternatively upper bound (114) as follows

$\begin{array}{ll}{\psi }_i\le -(n-a_i)(x_2-x_1){{H}}^{\prime }({\xi }_{21})+a_i(x_3-x_2){{H}}^{\prime }({\xi }_{32}),& (118)\end{array}$

where ξ₂₁ ∈ (x₁, x₂), and ξ₃₂ ∈ (x₂, x₃), which implies

$\begin{array}{ll}{\xi }_{21}<{\xi }_{32},\text{ and }{{H}}^{\prime }({\xi }_{21})>{{H}}^{\prime }({\xi }_{32}).& (119)\end{array}$

Using relation (117), bound (118) simplifies to the following.

$\begin{array}{lll}{\psi }_i& \le -a_i(x_3-x_2){{H}}^{\prime }({\xi }_{21})+a_i(x_3-x_2){{H}}^{\prime }({\xi }_{32})& (120)\end{array}$

$\begin{array}{ll}=-a_i(x_3-x_2)\left[{{H}}^{\prime }({\xi }_{21})-{{H}}^{\prime }({\xi }_{32})\right]& (121)\end{array}$

$\begin{array}{ll}\le -a_i(x_3-x_2)({\xi }_{21}-{\xi }_{32}){{H}}^{″}({\xi }_{31}),& (122)\end{array}$

for ξ₃₁ ∈ (x₁, x₃). Since ξ₂₁ < ξ₃₂, we rewrite bound (122) as follows.

$\begin{array}{lll}{\psi }_i& \le a_i(x_3-x_2)({\xi }_{32}-{\xi }_{21}){{H}}^{″}({\xi }_{31})& (123)\end{array}$

$\begin{array}{ll}\le a_i\eta (x_3-x_2){{H}}^{″}(x_3),& (124)\end{array}$

where η = ξ₃₂ − ξ₂₁ > 0, and the last bound is due to the fact that x₃ > ξ₃₁.

Going back to our normal notation, we rewrite bound (124) as follows.

$\begin{array}{lll}{\psi }_i& \le a_i\eta \left(-c{a}_i-\frac{a_i}{n}\right){{H}}^{″}\left(-c{a}_i\right)& (125)\end{array}$

$\begin{array}{ll}=a_i\eta \frac{a_i}{n}\left(\beta -1\right)\frac{-1}{\beta \frac{a_i}{n}\left(1-\beta \frac{a_i}{n}\right)}& (126)\end{array}$

$\begin{array}{ll}=-\frac{a_i\eta \left(\beta -1\right)}{\beta \left(1-\beta \frac{a_i}{n}\right)},& (127)\end{array}$

This concludes the proof.□

6.8. Inequality 84

The series bound (84) is derived as follows.

$\begin{array}{ll}{\displaystyle \sum _{i\in \Omega }}\left(\frac{s}{2i}·\frac{a_i}{n}\right)=\left(\frac{s}{2}·\frac{a_1}{n}\right)+\left(\frac{s}{4}·\frac{a_2}{n}\right)+\cdots +\left(\frac{s}{s}·\frac{a_{s/2}}{n}\right)& (128)\end{array}$

$\begin{array}{l}\ge \left(\frac{s}{2n}·\frac{\left(1-ϵ\right)ds}{2^{{log}_{2}s}}\right)+\left(\frac{s}{4n}·\frac{\left(1-ϵ\right)ds}{2^{{log}_2\left(s/2\right)}}\right)+\cdots \\ +\left(\frac{s}{sn}·\frac{\left(1-ϵ\right)ds}{2}\right)& (129)\end{array}$

$\begin{array}{l}=[\left(\frac{s}{2n}·\frac{1}{s}\right)+\left(\frac{s}{4n}·\frac{2}{s}\right)+\cdots \\ +\left(\frac{s}{sn}·\frac{1}{2}\right)](1-ϵ)ds& (130)\end{array}$

$\begin{array}{l}=\left(\frac{1}{2n}+\frac{1}{2n}+\cdots +\frac{1}{2n}\right)\left(1-ϵ\right)ds\\ =\frac{{log}_2\left(s/2\right)}{2n}\left(1-ϵ\right)ds.& (131)\end{array}$

That is the required bounds, hence concluding the proof.□

7. Conclusion

We considered the construction of sparse matrices that are invaluable for dimensionality reduction with application in diverse fields. These sparse matrices are more efficient computationally compared to their dense counterparts also used for the purpose of dimensionality reduction. Our construction is probabilistic based on the dyadic splitting method we introduced in Bah and Tanner [1]. By better approximation of the bounds we achieve a novel result, which is a reduced complexity of the sparsity per column of these matrices. Precisely, a complexity that is a state-of-the-art divided by logs, where s is the intrinsic dimension of the problem.

Our approach is one of a few that gives quantitative sampling theorems for existence of such sparse matrices. Moreover, using the phase transition framework comparison, our construction is better than existing probabilistic constructions. We are also able to compare performance of combinatorial compressed sensing algorithms by comparing their phase transition curves. This is one perspective in algorithm comparison amongst a couple of others like runtime and iteration complexities.

Evidently, our results holds true for the construction of expander graphs, which is a graph theory problem and is of interest to communities in theoretical computer science and pure mathematics.

Author Contributions

JT contributed to conception of the idea. Both JT and BB worked on the derivation of the initial results, which was substantially improved by BB. BB wrote the first draft of the manuscript. JT did some revisions and gave feedback.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

BB acknowledges the support from the funding by the German Federal Ministry of Education and Research (BMBF) for the German Research Chair at AIMS South Africa, funding for which is administered by Alexander von Humboldt Foundation (AvH). JT acknowledges support from The Alan Turing Institute under the EPSRC grant EP/N510129/1.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fams.2018.00039/full#supplementary-material

References

1. Bah B, Tanner J. Vanishingly sparse matrices and expander graphs, with application to compressed sensing. IEEE Trans Inform Theor. (2013) 59:7491–508. doi: 10.1109/TIT.2013.2274267

CrossRef Full Text | Google Scholar

2. Ahn KJ, Guha S, McGregor A. Graph sketches: sparsification, spanners, and subgraphs. In: Proceedings of the 31st Symposium on Principles of Database Systems. Scottsdale, AZ: ACM (2012). p. 5–14.

Google Scholar

3. Gilbert AC, Levchenko K. Compressing network graphs. In: Proceedings of the LinkKDD Workshop at the 10th ACM Conference on KDD. Seatle, WA (2004).

Google Scholar

4. Vardi Y. Network tomography: estimating source-destination traffic intensities from link data. J Am Stat Assoc. (1996) 91:365–77.

Google Scholar

5. Castro R, Coates M, Liang G, Nowak R, Yu B. Network tomography: recent developments. Stat Sci. (2004) 19:499–517. doi: 10.1214/088342304000000422

CrossRef Full Text | Google Scholar

6. Muthukrishnan S. Data Streams: Algorithms and Applications. Now Publishers Inc (2005).

Google Scholar

7. Indyk P. Sketching, Streaming and Sublinear-Space Algorithms. Graduate course notest (2007).

Google Scholar

8. Dwork C, McSherry F, Talwar K. The price of privacy and the limits of LP decoding. In: Proceedings of the Thirty-ninth Annual ACM Symposium on Theory of Computing. San Diego, CA: ACM (2007). p. 85–94.

Google Scholar

9. Donoho DL, Johnstone IM, Hoch JC, Stern AS. Maximum entropy and the nearly black object. J R Stat Soc Ser B (1992) 54:41–81.

Google Scholar

10. Donoho DL. Compressed sensing. IEEE Trans Inform Theor. (2006) 52:1289–306. doi: 10.1109/TIT.2006.871582

CrossRef Full Text | Google Scholar

11. Xu W, Hassibi B. Efficient compressive sensing with deterministic guarantees using expander graphs. In: Information Theory Workshop, 2007. ITW'07. Tahoe City, CA: IEEE (2007). p. 414–9.

Google Scholar

12. Jafarpour S, Xu W, Hassibi B, Calderbank R. Efficient and robust compressed sensing using optimized expander graphs. IEEE Trans Inform Theor. (2009) 55:4299–308. doi: 10.1109/TIT.2009.2025528

CrossRef Full Text | Google Scholar

13. Berinde R, Indyk P. Sequential sparse matching pursuit. In: 47th Annual Allerton Conference on Communication, Control, and Computing. IEEE (2009). p. 36–43.

Google Scholar

14. Mendoza-Smith R, Tanner J. Expander ℓ₀-decoding. Appl. Comput. Harmon. Anal. (2017) 45:642–67. doi: 10.1016/j.acha.2017.03.001

CrossRef Full Text | Google Scholar

15. Mendoza-Smith R, Tanner J, Wechsung F. A robust parallel algorithm for combinatorial compressed sensing. arXiv [preprint] arXiv:170409012 (2017).

Google Scholar

16. Du DZ, Hwang FK. Combinatorial Group Testing and Its Applications. World Scientific (2000).

17. Gilbert AC, Iwen MA, Strauss MJ. Group testing and sparse signal recovery. In: Signals, Systems and Computers, 2008 42nd Asilomar Conference on. Pacific Grove, CA: IEEE (2008). p. 1059–63.

Google Scholar

18. Berinde R, Gilbert AC, Indyk P, Karloff H, Strauss MJ. Combining geometry and combinatorics: A unified approach to sparse signal recovery. In: Communication, Control, and Computing, 2008 46th Annual Allerton Conference on. Urbana-Champaign, IL: IEEE (2008). p. 798–805.

Google Scholar

19. Foucart S, Rauhut H. A Mathematical Introduction to Compressive Sensing. New York, NY: Springer (2013).

Google Scholar

20. Hoory S, Linial N, Wigderson A. Expander graphs and their applications. Bull Am Math Soc. (2006) 43:439–562. doi: 10.1090/S0273-0979-06-01126-8

CrossRef Full Text | Google Scholar

21. Guruswami V, Umans C, Vadhan S. Unbalanced expanders and randomness extractors from Parvaresh–Vardy codes. J ACM (2009) 56:20. doi: 10.1145/1538902.1538904

CrossRef Full Text | Google Scholar

22. Bassalygo LA, Pinsker MS. Complexity of an optimum nonblocking switching network without reconnections. Problemy Peredachi Informatsii (1973) 9:84–87.

Google Scholar

23. Indyk P, Razenshteyn I. On model-based RIP-1 matrices. In: International Colloquium on Automata, Languages, and Programming. Berlin; Heidelberg: Springer (2013). p. 564–75.

Google Scholar

24. Bah B, Baldassarre L, Cevher V. Model-based sketching and recovery with expanders. In: SODA. SIAM (2014). p. 1529–43.

Google Scholar

25. Donoho DL, Tanner J. Thresholds for the recovery of sparse solutions via l₁ minimization. In: Information Sciences and Systems, 2006 40th Annual Conference on. IEEE (2006). p. 202–6.

Google Scholar

26. Buhrman H, Miltersen PB, Radhakrishnan J, Venkatesh S. Are bitvectors optimal? SIAM J Comput. (2002) 31:1723–44. doi: 10.1137/S0097539702405292

CrossRef Full Text | Google Scholar

27. Berinde R. Advances in Sparse Signal Recovery Methods. Massachusetts Institute of Technology (2009).

28. Candès EJ, Romberg J, Tao T. Stable signal recovery from incomplete and inaccurate measurements. Commun Pure Appl Math. (2006) 59:1207–23. doi: 10.1002/cpa.20124

CrossRef Full Text | Google Scholar

29. Capalbo M, Reingold O, Vadhan S, Wigderson A. Randomness conductors and constant-degree lossless expanders. In: Proceedings of the Thirty-fourth Annual ACM Symposium on Theory of Computing. New York, NY: ACM (2002). p. 659–68.

Google Scholar

30. Blanchard JD, Cartis C, Tanner J. Compressed sensing: how sharp is the restricted isometry property? SIAM Rev. (2011) 53:105–25. doi: 10.1137/090748160

CrossRef Full Text | Google Scholar

31. Bah B, Tanner J. Improved bounds on restricted isometry constants for Gaussian matrices. SIAM J Matrix Anal Appl. (2010) 31:2882–98. doi: 10.1137/100788884

CrossRef Full Text | Google Scholar

32. Indyk P, Ruzic M. Near-optimal sparse recovery in the ℓ₁-norm. In: Foundations of Computer Science, 2008. FOCS'08. IEEE 49th Annual IEEE Symposium on. Philadelphia, PA: IEEE (2008). p. 199–207.

Google Scholar

33. Berinde R, Indyk P, Ruzic M. Practical near-optimal sparse recovery in the ℓ₁-norm. In: 46th Annual Allerton Conference on Communication, Control, and Computing. Piscataway, NJ: IEEE (2008). p. 198–205.

Google Scholar