Optimal blending of multiple independent prediction models

Introduction

Participants of the Netflix competition used model blending heavily—refer to, for example Töscher et al. (2009), Amatriain (2013), Xiang and Yang (2009), Coscrato et al. (2020), Koren (2009), Jahrer et al. (2010), and Bothos et al. (2011). Ensemble modeling (blending) was popular not only in Netflix competition but is used also on other machine learning problems such as image processing, for example for CIFAR-10 dataset, refer to Abouelnaga et al. (2016) and Bruno et al. (2022), for the MNIST dataset refer to Ciresan et al. (2011). Ensemble modeling is used also in many other fields, for example, refer to Schuhen et al. (2012) and Ardabili et al. (2020). In this study, we derive blending coefficients based on variances of different models with only the assumption of model independence. While the formula for the final variance of the blended model and its coefficients is already derived in Kay (1993) without a proof (Equations 6.7 and 6.8 in chapter 6.4), we provide proofs both for the formula for blending coefficients and the variance of the combined model as well as the lower and upper bound estimate for the final variance based on the minimal and maximal variance of all the combined models. We also compare these results with machine learning with counts, where only binary information is used from the features to make the decision, in the last section and show very similar conclusions.

Let ŷ_k,j be a prediction of model k ∈ [1, N] for element j ∈ [1, M], where N is the number of different independent models and M is the number of measurements we have:

$\begin{array}{l}{ŷ}_{k,j}=y_j+r_{k,j},\end{array}$

where y_j is an expected prediction and r_k,j is a random variable with normal distribution $R_k~{N}(0,{\sigma }_k^2)$, which has a zero average (the expected value of the variable is 0). In this study, we derive optimal blending coefficients α_k such that the blended prediction ŷ_B is optimal:

$\begin{array}{l}{ŷ}_{B,j}={\displaystyle \sum _{k=1}^N}{\alpha }_k{ŷ}_{k,j}=y_j{\displaystyle \sum _{k=1}^N}{\alpha }_k+{\displaystyle \sum _{k=1}^N}{\alpha }_k{r}_{k,j}=y_j+{\displaystyle \sum _{k=1}^N}{\alpha }_k{r}_{k,j}\end{array}$

with minimum variance ${\sigma }_B^2$, where $\sum _{k=1}^N{\alpha }_k=1$.

Blending two independent models

Here we present two independent models

$\begin{array}{l}{ŷ}_{1,j}=y_j+r_{1,j}\\ {ŷ}_{2,j}=y_j+r_{2,j},\end{array}$

where $R_1~{N}(0,{\sigma }_1^2)$ and $R_2~{N}(0,{\sigma }_2^2)$. We derive $\widehat{\alpha }\in \left[0,1\right]$ for which we get the optimal blending model

$\begin{array}{r}{ŷ}_{B,j}=\alpha (y_j+r_{1,j})+(1-\alpha )(y_j+r_{2,j})=y_j+\alpha r_{1,j}\\ \hfill +(1-\alpha )r_{2,j}.\end{array}$

It is well-known fact that a random variable combining two random variables αR₁ + (1 − α)R₂, where $R_1~{N}(0,{\sigma }_1^2)$, $R_2~{N}(0,{\sigma }_2^2)$ and R₁ and R₂ are independent, has a normal distribution ${N}(0,{\sigma }_B^2)$, where ${\sigma }_B^2={\alpha }^2{\sigma }_1^2+{(1-\alpha )}^2{\sigma }_2^2$. For the mean we get:

$\begin{array}{l}E(Y_B)=\frac{1}{M}{\displaystyle \sum _{j=1}^M}(\alpha r_{1,j}+(1-\alpha )r_{2,j})\\ =\alpha E(R_1)+(1-\alpha )E(R_2)=0\end{array}$

and for the variance we get:

$\begin{array}{l}\hfill E(Y_B^2)=\frac{1}{M}{\displaystyle \sum _{j=1}^M}{(\alpha r_{1,j}+(1-\alpha )r_{2,j})}^2=\\ ={\alpha }^2\frac{1}{M}{\displaystyle \sum _{j=1}^M}{r}_{1,j}^2+2\alpha (1-\alpha )\frac{1}{M}{\displaystyle \sum _{j=1}^M}{r}_{1,j}{r}_{2,j}+{(1-\alpha )}^2\frac{1}{M}{\displaystyle \sum _{j=1}^M}{r}_{2,j}^2.\end{array}$

Finally as R₁ and R₂ are independent (covariance $\frac{1}{M}\sum _{j=1}{r}_{1,j}{r}_{2,j}=0$ is zero), we can write:

$\begin{array}{l}{\sigma }_B^2=E(Y_B^2)={\alpha }^2{\sigma }_1^2+{(1-\alpha )}^2{\sigma }_2^2.\end{array}$

To find the optimal (we are looking for minimal value and function is convex with one minimum as we have only α⁰, α¹, and α² dependencies—quadratic function and ${\sigma }_1^2+{\sigma }_2^2>0$) blending parameter, we compute where a partial derivative of the new variance of the blended model is zero:

$\begin{array}{l}\frac{\partial {\sigma }_B^2}{\partial \alpha }=2\widehat{\alpha }{\sigma }_1^2-2(1-\widehat{\alpha }){\sigma }_2^2=0,\end{array}$

from which

$\begin{array}{ll}\widehat{\alpha }=\frac{{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2}& (1)\end{array}$

and the optimal variance will be:

$\begin{array}{l}{\sigma }_B^2(\widehat{\alpha })=\frac{{\sigma }_1^2{\sigma }_2^4}{{({\sigma }_1^2+{\sigma }_2^2)}^2}+\frac{{\sigma }_2^2{\sigma }_1^4}{{({\sigma }_1^2+{\sigma }_2^2)}^2}=\frac{{\sigma }_1^2{\sigma }_2^2({\sigma }_1^2+{\sigma }_2^2)}{{({\sigma }_1^2+{\sigma }_2^2)}^2}\\ \hfill =\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2}& (2)\end{array}$

In Figure 1, we show how the variance is changing for different blending parameters α. Script is in Appendix 1. Blue dot—optimal value of blending parameter α matches simulation (minimal value for variance).

FIGURE 1

Figure 1. Red line—variance for different α. Green line—optimal variance ${\sigma }_B^2$. Blue dot—optimal α with its value ${\sigma }_B^2(\widehat{\alpha })$. Python script is in Appendix 1.

Blending three independent models

Now, we consider three independent models

$\begin{array}{l}{ŷ}_{1,j}=y_j+r_{1,j}\\ {ŷ}_{2,j}=y_j+r_{2,j},\\ {ŷ}_{3,j}=y_j+r_{3,j},\end{array}$

where $R_1~{N}(0,{\sigma }_1^2)$, $R_2~{N}(0,{\sigma }_2^2)$, and $R_3~{N}(0,{\sigma }_3^2)$.

Here, we blend optimally the first two models from the previous section:

$\begin{array}{l}{ŷ}_{4,j}=y_j+\widehat{\alpha }{r}_{1,j}+(1-\widehat{\alpha })r_{2,j}=y_j+r_{4,j},\end{array}$

where $R_4~{N}(0,\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2})$ and then we find the blending parameter $\widehat{\beta }$ for ŷ_3,j and ŷ_4,j such that

$\begin{array}{ll}{ŷ}_{B,j}=y_j+\widehat{\beta }{r}_{3,j}+(1-\widehat{\beta })r_{4,j}.& (3)\end{array}$

Based on the Equation (1), we get

$\begin{array}{l}\widehat{\beta }=\frac{\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2}}{{\sigma }_3^2+\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2}}=\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}.\end{array}$

Plugging this back into the Equation (3), we get

$\begin{array}{l}{ŷ}_{B,j}=y_j+\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}{r}_{3,j}\\ +(1-\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2})r_{4,j}\end{array}$

$\begin{array}{l}\hfill {ŷ}_{B,j}=y_j+\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}{r}_{3,j}\\ +\frac{({\sigma }_1^2+{\sigma }_2^2){\sigma }_3^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}(\widehat{\alpha }{r}_{1,j}+(1-\widehat{\alpha })r_{2,j})\end{array}$

$\begin{array}{l}\hfill {ŷ}_{B,j}=y_j+\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}{r}_{3,j}\\ +\frac{({\sigma }_1^2+{\sigma }_2^2){\sigma }_3^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}(\frac{{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2}{r}_{1,j}+(1-\frac{{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2})r_{2,j})\end{array}$

$\begin{array}{l}{ŷ}_{B,j}=y_j+\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}{r}_{3,j}\\ +\frac{{\sigma }_2^2{\sigma }_3^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}{r}_{1,j}+\frac{{\sigma }_1^2{\sigma }_3^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}{r}_{2,j},\end{array}$

which is symmetrical, meaning model combination order is irrelevant. Finally for ${\widehat{\alpha }}_1$, ${\widehat{\alpha }}_2$, ${\widehat{\alpha }}_3$, we get:

$\begin{array}{l}{\widehat{\alpha }}_1=\frac{{\sigma }_2^2{\sigma }_3^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}\\ {\widehat{\alpha }}_2=\frac{{\sigma }_1^2{\sigma }_3^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}\\ {\widehat{\alpha }}_3=\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}.\end{array}$

Combining the second and third models first and then combining the result with the first model would lead to the same optimal blending parameters. The order of the combination is inconsequential. Additionally, for the final variance, we get from Equation (2)

$\begin{array}{l}{\sigma }_B^2(\widehat{\alpha })=\frac{{\sigma }_3^2\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2}}{{\sigma }_3^2+\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2}}=\frac{{\sigma }_1^2{\sigma }_2^2{\sigma }_3^2}{{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_3^2+{\sigma }_2^2{\sigma }_3^2}.\end{array}$

In Figure 2, we show how variance is changing for different blending parameters α₁ and α₂ and α₃ = 1 − α₁ − α₂. The script is in Appendix 2. Blue dot—the optimal value of blending parameters (α₁, α₂, 1 − α₁ − α₂) matches the simulation (minimal value for variance).

FIGURE 2

Figure 2. Grid consisting of red dots—variance for different α₁ and α₂. Grid consisting of green dots—optimal variance ${\sigma }_B^2$. Blue dot—optimal α₁ and α₂ and 1 − α₁ − α₂ with its value ${\sigma }_B^2(\widehat{\alpha })$. Python script is in Appendix 2.

Blending N independent models

Now that we have formulas for two and three different models, we prove formulas for N independent models with normal distributions:

$\begin{array}{l}{ŷ}_{k,j}=y_j+r_{k,j},\end{array}$

where $R_k~{N}(0,{\sigma }_k^2)$. We combine these models as follows:

$\begin{array}{l}{ŷ}_{B,j}={\displaystyle \sum _{k=1}^N}{\alpha }_k{ŷ}_{k,j},\end{array}$

where $\sum _{k=1}^N{\alpha }_k=1$ and α_k > 0 for k ∈ [1, N].

First, we show model independence is still needed.

Lemma 1. Having N independent models with normal distributions $R_k~{N}(0,{\sigma }_k^2)$ for k ∈ [1, N], when combined as $r_{B,j}=\sum _{k=1}^N{\alpha }_k{r}_{k,j}$, variance of R_B is ${\sigma }_B^2=\sum _{k=1}^N{\alpha }_k^2{\sigma }_k^2$.

Proof.

$\begin{array}{l}{\sigma }_B^2=E({({\displaystyle \sum _{k=1}^N}{\alpha }_k{r}_{k,j})}^2)\\ =\frac{1}{M}({\displaystyle \sum _{k=1}^N}{\displaystyle \sum _{j=1}^M}{\alpha }_k^2{r}_{k,j}^2+2{\displaystyle \sum _{k=1}^N}{\displaystyle \sum _{\begin{array}{c}l=1\\ l\ne k\end{array}}^N}{\displaystyle \sum _{j=1}^M}{\alpha }_k{\alpha }_l{r}_{k,j}{r}_{l,j})\\ ={\displaystyle \sum _{k=1}^N}{\alpha }_k^2\frac{1}{M}{\displaystyle \sum _{j=1}^M}{r}_{k,j}^2+2{\displaystyle \sum _{k=1}^N}{\displaystyle \sum _{\begin{array}{c}l=1\\ l\ne k\end{array}}^N}{\alpha }_k{\alpha }_l\frac{1}{M}{\displaystyle \sum _{j=1}^M}{r}_{k,j}{r}_{l,j}\end{array}$

As models are independent (covariance $\frac{1}{M}\sum _{j=1}^M{r}_{k,j}{r}_{l,j}=0$ is zero for l ≠ k), we get

$\begin{array}{l}{\sigma }_B^2={\displaystyle \sum _{k=1}^N}{\alpha }_k^2\frac{1}{M}{\displaystyle \sum _{j=1}^M}{r}_{k,j}^2={\displaystyle \sum _{k=1}^N}{\alpha }_k^2{\sigma }_k^2,\end{array}$

which ends the proof.

Theorem 2. Having N independent models with normal distributions $R_k~{N}(0,{\sigma }_k^2)$ for k ∈ [1, N], we get an optimal blend with parameters

$\begin{array}{l}\widehat{{\alpha }_k}=\frac{{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2},\end{array}$

and these independent models form normal distribution ${N}(0,{\sigma }_B^2)$, which has variance

$\begin{array}{l}{\sigma }_B^2=\frac{{\displaystyle {\prod }_{j=1}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}.\end{array}$

Proof. For N = 2, we have shown it in Section 2. Now we use induction, if it is true for N, then it is true also for N + 1.

Remark. We have shown this also for three models in Section 3, but as for induction it is not needed, Section 3 is only a motivational section for how to derive final formulas for N models.

We combine two normal distributions ${N}(0,\frac{{\prod }_{j=1}^N{\sigma }_j^2}{{\sum }_{i=1}^N{\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N{\sigma }_j^2})$ (assuming it is true for N) and ${N}(0,{\sigma }_{N+1}^2)$. From Equation (2), (lemma 1 is incorporated in this equation) we get

$\begin{array}{l}{\sigma }_B^2=\frac{\frac{{\displaystyle {\prod }_{j=1}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}{\sigma }_{N+1}^2}{\frac{{\displaystyle {\prod }_{j=1}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}+{\sigma }_{N+1}^2}\\ =\frac{{\sigma }_{N+1}^2{\displaystyle {\prod }_{j=1}^N}{\sigma }_j^2}{{\displaystyle {\prod }_{j=1}^N}{\sigma }_j^2+{\sigma }_{N+1}^2{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}\\ =\frac{{\displaystyle {\prod }_{j=1}^{N+1}}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^{N+1}}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^{N+1}}{\sigma }_j^2}\end{array}$

and hence, we have shown optimal variance is valid for N + 1. Now we must show the same for the optimal coefficients. From Equation (1), we get

$\begin{array}{l}\widehat{\alpha }=\frac{{\sigma }_{N+1}^2}{\frac{{\displaystyle {\prod }_{j=1}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}+{\sigma }_{N+1}^2}=\frac{{\sigma }_{N+1}^2{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^{N+1}}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^{N+1}}{\sigma }_j^2}\end{array}$

and hence,

$\begin{array}{l}{\widehat{\alpha }}_{N+1}=1-\widehat{\alpha }\\ =1-\frac{{\sigma }_{N+1}^2{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^{N+1}}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^{N+1}}{\sigma }_j^2}\\ =\frac{{\displaystyle {\sum }_{i=1}^{N+1}}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^{N+1}}{\sigma }_j^2-{\sigma }_{N+1}^2{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^{N+1}}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^{N+1}}{\sigma }_j^2}\\ =\frac{{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne N+1\end{array}}^{N+1}}{\sigma }_j^2+{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^{N+1}}{\sigma }_j^2-{\sigma }_{N+1}^2{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^{N+1}}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^{N+1}}{\sigma }_j^2}\\ =\frac{{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne N+1\end{array}}^{N+1}}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^{N+1}}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^{N+1}}{\sigma }_j^2},\end{array}$

which proves ${\widehat{\alpha }}_{N+1}$. Finally to show the same for $\widehat{{\alpha }_k}$ for k ∈ [1, N]:

$\begin{array}{l}\widehat{{\alpha }_k}=\widehat{\alpha }\frac{{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}\\ =\frac{{\sigma }_{N+1}^2{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^{N+1}}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^{N+1}}{\sigma }_j^2}\frac{{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}\\ =\frac{{\sigma }_{N+1}^2}{{\displaystyle {\sum }_{i=1}^{N+1}}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^{N+1}}{\sigma }_j^2}{\displaystyle \prod _{\begin{array}{c}j=1\\ j\ne k\end{array}}^N}{\sigma }_j^2\\ =\frac{{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^{N+1}}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^{N+1}}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^{N+1}}{\sigma }_j^2},\end{array}$

which ends the proof.

Going to infinity

If we can generate infinite independent models with distributions $R_i~{N}(0,{\sigma }^2)$ (same variance), the final variance will be

$\begin{array}{l}{\sigma }_B^2={\displaystyle \underset{N\to +\infty }{lim}}\frac{{\displaystyle {\prod }_{j=1}^N}{\sigma }^2}{{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }^2}={\displaystyle \underset{N\to +\infty }{lim}}\frac{{\sigma }^{2N}}{N{\sigma }^{2(N-1)}}\\ \hfill ={\displaystyle \underset{N\to +\infty }{lim}}\frac{{\sigma }^2}{N}=0,\end{array}$

which means we can combine all these models to get a perfect prediction with no errors. Naturally, creating an infinite amount of independent models (with covariances zero) is a difficult if not impossible task in real applications.

Theorem 3. Having N independent models with normal distributions $R_k~{N}(0,{\sigma }_k^2)$ for k ∈ [1, N] and their variances ${\sigma }_k^2\le {\sigma }_M^2$, where ${\sigma }_M^2$ is their maximum variance, combining them optimally with coefficients from the theorem 2, their combined variance is ${\sigma }_B^2\le \frac{{\sigma }_M^2}{N}$.

Proof. We use induction again. For N = 2, we get

$\begin{array}{l}{\sigma }_B^2=\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2}\le \frac{{\sigma }_M^2}{2}\end{array}$

This is true as

$\begin{array}{l}{\sigma }_1^2{\sigma }_2^2+{\sigma }_1^2{\sigma }_2^2\le {\sigma }_M^2{\sigma }_1^2+{\sigma }_M^2{\sigma }_2^2,\end{array}$

because

$\begin{array}{l}{\sigma }_1^2{\sigma }_2^2\le {\sigma }_M^2{\sigma }_1^2\end{array}$

and

$\begin{array}{l}{\sigma }_1^2{\sigma }_2^2\le {\sigma }_M^2{\sigma }_2^2.\end{array}$

Now if it is true for N, then it is true also for N + 1. If

$\begin{array}{l}{\sigma }_{B,N}^2\le \frac{{\sigma }_M^2}{N},\end{array}$

then

$\begin{array}{l}{\sigma }_{B,N+1}^2\le \frac{{\sigma }_M^2}{N+1}.\end{array}$

That is true as

$\begin{array}{l}{\sigma }_{B,N+1}^2=\frac{{\sigma }_{B,N}^2{\sigma }_{N+1}^2}{{\sigma }_{B,N}^2+{\sigma }_{N+1}^2}\le \frac{{\sigma }_M^2}{N+1},\end{array}$

because

$\begin{array}{l}N{\sigma }_{B,N}^2{\sigma }_{N+1}^2+{\sigma }_{B,N}^2{\sigma }_{N+1}^2\le {\sigma }_M^2({\sigma }_{B,N}^2+{\sigma }_{N+1}^2)\end{array}$

as both - this

$\begin{array}{l}N{\sigma }_{B,N}^2{\sigma }_{N+1}^2\le {\sigma }_M^2{\sigma }_{N+1}^2\end{array}$

and this

$\begin{array}{l}{\sigma }_{B,N}^2{\sigma }_{N+1}^2\le {\sigma }_M^2{\sigma }_{B,N}^2\end{array}$

are true, which ends the proof.

This proof means, that if we combine infinite independent models with distributions $R_i~{N}(0,{\sigma }_i^2)$, where variance ${\sigma }_i^2\le {\sigma }_M^2$, we get variance:

$\begin{array}{l}{\sigma }_B^2={\displaystyle \underset{N\to +\infty }{lim}}\frac{{\displaystyle {\prod }_{j=1}^N}{\sigma }_j^2}{{\displaystyle {\sum }_{i=1}^N}{\displaystyle {\prod }_{\begin{array}{c}j=1\\ j\ne i\end{array}}^N}{\sigma }_j^2}\le {\displaystyle \underset{N\to +\infty }{lim}}\frac{{\sigma }_M^2}{N}=0.\end{array}$

Combining infinite independent models with bounded variances from above leads to perfect prediction with variance zero.

It can be shown the same way as in Theorem 3 that combined variance is bounded also from below (as the proof is almost identical we avoid it here). If all distributions $R_k~{N}(0,{\sigma }_k^2)$ for k ∈ [1, N] have their variance in interval ${\sigma }_k^2\in \left[{\sigma }_{min}^2,{\sigma }_{max}^2\right]$ for k ∈ [1, N], then their combined variance will be in interval ${\sigma }_B^2\in \left[\frac{{\sigma }_{min}^2}{N},\frac{{\sigma }_{max}^2}{N}\right]$.

Similar conclusion with machine learning with counts

When it comes to using only counts (feature says yes or no only) in machine learning for predictions, as it is shown in Taraba (2021) (see section 7) on a nine-features example, we can come to the same conclusion as in the previous chapter that an infinite amount of features can lead to perfect prediction with no error. While the previous approach is statistical, machine learning with counts uses Pascal's triangle and binomial raised to infinity to show this. We use the binomial expansion

$\begin{array}{l}1={(p+(1-p))}^n={\displaystyle \sum _{i=0}^n}\left(\genfrac{}{}{0ex}{}{n}{i}\right)p^{n-i}{(1-p)}^i,\end{array}$

where p is the probability of features to be correct. As we want to have an odd amount of features to be able to make a decision purely on the counts (feature says yes or no), we will replace n with 2k + 1

$\begin{array}{l}1={(p+(1-p))}^{2k+1}={\displaystyle \sum _{i=0}^{2k+1}}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i.\end{array}$

This can be split into two parts, one with probability when the majority of features are correct P_correct and one with probability when the majority of features are incorrect:

$\begin{array}{l}1={(p+(1-p))}^{2k+1}=P_{correct}+P_{incorrect},\end{array}$

where

$\begin{array}{l}{P}_{correct}={\displaystyle \sum _{i=0}^k}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i\end{array}$

and

$\begin{array}{l}{P}_{incorrect}={\displaystyle \sum _{i=k+1}^{2k+1}}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i.\end{array}$

To show that an infinite amount of features can lead to perfect prediction, we have to show that P_correct with the majority of features correct (at least k + 1 of them correct) goes to 1 for all p ∈ (0.5, 1]

$\begin{array}{l}{\displaystyle \underset{k\to \infty }{lim}}{\displaystyle \sum _{i=0}^k}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i=1\end{array}$

We start by showing the simpler case first and that is when p = 0.5 then P_correct and P_incorrect are equal and P_correct = P_incorrect = 0.5. To show this, we can write

$\begin{array}{l}{P}_{correct,p=0.5}={\displaystyle \sum _{i=0}^k}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)0.5^{2k+1}=0.5^{2k+1}{\displaystyle \sum _{i=0}^k}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)\end{array}$

and

$\begin{array}{l}{P}_{incorrect,p=0.5}={\displaystyle \sum _{i=k+1}^{2k+1}}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)0.5^{2k+1}=0.5^{2k+1}{\displaystyle \sum _{i=k+1}^{2k+1}}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)\end{array}$

and those are equal as $\sum _{i=0}^k\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)=\sum _{i=k+1}^{2k+1}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)$, because $\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)=\left(\genfrac{}{}{0ex}{}{2k+1}{2k+1-i}\right)$ for i ∈ {0, 1, …, k}. As P_{correct,p = 0.5} = P_{incorrect,p = 0.5} and their sum is 1 it follows that

$\begin{array}{l}1=P_{correct,p=0.5}+P_{incorrect,p=0.5}=2P_{correct,p=0.5}\end{array}$

and hence,

$\begin{array}{l}{P}_{correct,p=0.5}=P_{incorrect,p=0.5}=0.5.\end{array}$

Now that we have shown what happens when p = 0.5, we show the main limit theorem for p ∈ (0.5, 1].

Theorem 4. $\underset{k\to \infty }{lim}\sum _{i=0}^k\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i=1$ for all p ∈ (0.5, 1].

Proof. To show that $\underset{k\to \infty }{lim}\sum _{i=0}^k\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i=1$ for all p ∈ (0.5, 1], we will show instead that $\underset{k\to \infty }{lim}\sum _{i=k+1}^{2k+1}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i=0$ and as their sum is 1, $\underset{k\to \infty }{lim}\sum _{i=0}^k\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i=1$ will follow.

First, we can rewrite $\sum _{i=k+1}^{2k+1}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i$ as

$\begin{array}{l}\hfill {\displaystyle \sum _{i=k+1}^{2k+1}}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i\\ ={\displaystyle \sum _{i=0}^k}\left(\genfrac{}{}{0ex}{}{2k+1}{k+1+i}\right)p^{2k+1-(k+1+i)}{(1-p)}^{k+1+i}.\end{array}$

It is obvious that the numbers in the pascal triangle are decreasing when starting after the middle:

$\begin{array}{l}\frac{\left(\genfrac{}{}{0ex}{}{2k+1}{k+1+i+1}\right)}{\left(\genfrac{}{}{0ex}{}{2k+1}{k+1+i}\right)}=\frac{\frac{(2k+1)!}{(k+2+i)!(k-i-1)!}}{\frac{(2k+1)!}{(k+1+i)!(k-i)!}}=\frac{k-i}{k+i+2}<1\end{array}$

for i ∈ 0, 1, …, k − 1, and hence, we can write

$\begin{array}{r}{\displaystyle \sum _{i=0}^k}\left(\genfrac{}{}{0ex}{}{2k+1}{k+1+i}\right)p^{2k+1-(k+1+i)}{(1-p)}^{k+1+i}\\ <\left(\genfrac{}{}{0ex}{}{2k+1}{k+1}\right)p^k{(1-p)}^{k+1}{\displaystyle \sum _{i=0}^k}{(\frac{1-p}{p})}^i.\end{array}$

As p is in p ∈ (0.5, 1], then $\frac{1-p}{p}\in [0,1)$, and hence, we can write

$\begin{array}{r}\left(\genfrac{}{}{0ex}{}{2k+1}{k+1}\right)p^k{(1-p)}^{k+1}{\displaystyle \sum _{i=0}^k}{(\frac{1-p}{p})}^i\\ =\left(\genfrac{}{}{0ex}{}{2k+1}{k+1}\right)p^k{(1-p)}^{k+1}\frac{1-{(\frac{1-p}{p})}^{k+1}}{1-(\frac{1-p}{p})}.\end{array}$

With that we can finally look at the original limit and write

$\begin{array}{l}{\displaystyle \underset{k\to \infty }{lim}}{\displaystyle \sum _{i=k+1}^{2k+1}}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i\le {\displaystyle \underset{k\to \infty }{lim}}\\ \hfill \left(\genfrac{}{}{0ex}{}{2k+1}{k+1}\right)p^k{(1-p)}^{k+1}\frac{1-{(\frac{1-p}{p})}^{k+1}}{1-(\frac{1-p}{p})}.\end{array}$

As $\underset{k\to \infty }{lim}{(\frac{1-p}{p})}^{k+1}=0$ for p ∈ (0.5, 1] as $\frac{1-p}{p}\in [0,1)$, we can write

$\begin{array}{l}{\displaystyle \underset{k\to \infty }{lim}}{\displaystyle \sum _{i=k+1}^{2k+1}}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i\\ \le \frac{1}{1-(\frac{1-p}{p})}{\displaystyle \underset{k\to \infty }{lim}}\left(\genfrac{}{}{0ex}{}{2k+1}{k+1}\right)p^k{(1-p)}^{k+1}\\ =\frac{p}{2p-1}{\displaystyle \underset{k\to \infty }{lim}}\left(\genfrac{}{}{0ex}{}{2k+1}{k+1}\right)p^k{(1-p)}^{k+1}.\end{array}$

Now we look at $\underset{k\to \infty }{lim}\left(\genfrac{}{}{0ex}{}{2k+1}{k+1}\right)p^k{(1-p)}^{k+1}$. We can take a member $\left(\genfrac{}{}{0ex}{}{2(k+r)+1}{(k+r)+1}\right)p^{(k+r)}{(1-p)}^{(k+r)+1}$ and compare it with r′ = r + 1 follower $\left(\genfrac{}{}{0ex}{}{2(k+r+1)+1}{(k+r+1)+1}\right)p^{(k+r+1)}{(1-p)}^{(k+r+1)+1}$ by division

$\begin{array}{r}\frac{\left(\begin{array}{c}2(k+r+1)+1\\ (k+r+1)+1\end{array}\right)p^{(k+r+1)}{(1-p)}^{(k+r+1)+1}}{\left(\begin{array}{c}2(k+r)+1\\ (k+r)+1\end{array}\right)p^{(k+r)}{(1-p)}^{(k+r)+1}}=\frac{\frac{(2(k+r)+3)!}{(k+r+2)!(k+r+1)!}}{\frac{(2(k+r)+1)!}{(k+r+1)!(k+r)!}}\\ p(1-p)==\frac{(2(k+r)+3)(2(k+r)+2)}{(k+r+2)(k+r+1)}p(1-p)\\ <4p(1-p)<1,\end{array}$

for all r ≥ 0 and p ∈ (0.5, 1] as p(1 − p) ∈ [0, 0.25). This means we are multiplying a finite number $\left(\genfrac{}{}{0ex}{}{2k+1}{k+1}\right)p^k{(1-p)}^{k+1}$, which is in interval [0, 1], infinitely many times with a number larger or equal than 0 and smaller than 1, hence

$\begin{array}{l}{\displaystyle \underset{k\to \infty }{lim}}\left(\genfrac{}{}{0ex}{}{2k+1}{k+1}\right)p^k{(1-p)}^{k+1}=0\end{array}$

and hence, the original limit

$\begin{array}{l}{\displaystyle \underset{k\to \infty }{lim}}{\displaystyle \sum _{i=k+1}^{2k+1}}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i\le 0\end{array}$

as well. As $\sum _{i=k+1}^{2k+1}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i\ge 0$, it follows that

$\begin{array}{l}{\displaystyle \underset{k\to \infty }{lim}}{\displaystyle \sum _{i=k+1}^{2k+1}}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i=0.\end{array}$

Remark. It is worth mentioning that p is fixed and chosen from the interval (0.5, 1], and we are not looking at the limit of p → 0.5, but the limit of k → ∞. For p = 0.5, we already know that P_{correct,p = 0.5} = P_{incorrect,p = 0.5} = 0.5.

As

$\begin{array}{l}\hfill {\displaystyle \underset{k\to \infty }{lim}}{\displaystyle \sum _{i=0}^k}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i+\\ +{\displaystyle \underset{k\to \infty }{lim}}{\displaystyle \sum _{i=k+1}^{2k+1}}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i=1,\end{array}$

it follows that $\underset{k\to \infty }{lim}\sum _{i=0}^k\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p^{2k+1-i}{(1-p)}^i=1$, which ends the proof.

This could be summarized for all p ∈ [0, 1] in Figure 3 as

$\begin{array}{l}{P}_{correct}=\{\begin{array}{cc}0\hfill & \text{for }p\in [0,0.5)\\ 0.5\hfill & \text{for }p=0.5\\ 1\hfill & \text{for }p\in (0.5,1]\end{array}.\hfill \end{array}$

FIGURE 3

Figure 3. Different outcomes for P_correct for different p intervals when k is going to ∞.

This can be understood intuitively by plotting P_correct with increasing k in Figure 4.

FIGURE 4

Figure 4. Different outcomes for P_correct for different p ∈ [0.0, 1] with increasing k ∈ [0, …, 51]. Python script is in Appendix 3.

It is worth mentioning that with weak features with p = 0.52, and only n = 15001 of them (no need to go to infinity), P_correct is already around 1 (see Figure 5).

FIGURE 5

Figure 5. Different outcomes for P_correct for different p ∈ [0.5, 0.6] with increasing n for n ∈ {5001, 10001, 15001}. Python script is in Appendix 4.

As next, we show that P_correct increases, if any of the probability of the features increases its value $p_i^{\prime }>p_i$. For case k = 0 (n = 1), this is simple, as

$\begin{array}{l}{P}_{correct}(new)=p_1^{\prime }>p_1=P_{correct}(old).\end{array}$

For three features, it is also trivial as

$\begin{array}{l}{P}_{correct}(new)=p_1^{\prime }{p}_2{p}_3+p_1^{\prime }{p}_2(1-p_3)+p_1^{\prime }(1-p_2)p3\\ \hfill +(1-p_1^{\prime })p_2{p}_3\end{array}$

as it can be re-written to $p_1^{\prime }$ independent part and $p_1^{\prime }$ dependent part

$\begin{array}{l}{P}_{correct}(new)=p_2{p}_3+p_1^{\prime }(p_2(1-p_3)+(1-p_2)p_3)\end{array}$

and from that, it immediately follows that P_correct(new) > P_correct(old) as $0\le p_1,p_1^{\prime },p_2,p_3\le 1$ and $p_1^{\prime }>p_1$. To show this for the general case, not only for cases k ∈ {0, 1} (n = 2k + 1 ∈ {1, 3}), we first write the general formula:

$\begin{array}{ll}1={\displaystyle \sum _{i_1=0}^1}{\displaystyle \sum _{i_2=0}^1}\dots {\displaystyle \sum _{i_{2k+1}=0}^1}{\displaystyle \prod _{j=1}^{2k+1}}(p_j{(-1)}^{i_j+1}+(1-i_j)).& (4)\end{array}$

This can be re-written into two parts once again, P_correct, when the majority of features say yes—are correct, and P_incorrect, when the majority of features say no—are incorrect.

$\begin{array}{l}{P}_{correct}(p_1,\dots ,p_{2k+1})={\displaystyle \sum _{i_1=0}^1}\dots {\displaystyle \sum _{i_{2k+1}=0}^1}m(i_1,\dots ,i_{2k+1})\\ \hfill {\displaystyle \prod _{j=1}^{2k+1}}(p_j{(-1)}^{i_j+1}+(1-i_j)),\end{array}$

where

$\begin{array}{l}m(i_1,\dots ,i_{2k+1})=\{\begin{array}{cc}0& \text{for }{\displaystyle {\sum }_{j=1}^{2k+1}}{i}_j\le k\\ 1& \text{for }{\displaystyle {\sum }_{j=1}^{2k+1}}{i}_j>k\end{array}.\end{array}$

Theorem 5. Increasing the probability of one of the features $p_1^{\prime }>p_1$ increases final correct probability of all features, $P_{correct}(p_1^{\prime },p_2,\dots ,p_{2k+1})>P_{correct}(p_1,p_2,\dots ,p_{2k+1})$.

Proof. We show this for increasing the first probability, as probabilities can be re-ordered and their order does not matter for the final correct probability.

Now again, as with example k = 1, n = 3, P_correct can be split into $p_1^{\prime }$ independent and dependent part. The dependent part will only contain $p_1^{\prime }$ and no $(1-p_1^{\prime })$ as for every $(1-p_1^{\prime })$ there is also the exact same case with $p_1^{\prime }$, where even more features are correct and those two can be joined and hence it belongs to the independent part. Hence,

$\begin{array}{l}{P}_{correct}(p_1^{\prime },p_2,\dots ,p_{2k+1})=P_I+p_1^{\prime }{P}_D,\end{array}$

and

$\begin{array}{l}{P}_{correct}(p_1,p_2,\dots ,p_{2k+1})=P_I+p_1{P}_D,\end{array}$

where P_I and P_D are fixed and non-negative (as all p_i ≥ 0 and (1 − p_i) ≥ 0) and dependent on p₂, …, p_2k+1 and hence $P_{correct}(p_1^{\prime },p_2,\dots ,p_{2k+1})=P_I+p_1^{\prime }{P}_D>P_I+p_1{P}_D=P_{correct}(p_1,p_2,\dots ,p_{2k+1})$ as $p_1^{\prime }>p_1$, which ends the proof.

Theorem 6. $P_{correct}(p_1,p_2,\dots ,p_{2k+1})\ge \sum _{i=0}^k\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p_{min}^{2k+1-i}{(1-p_{min})}^i$, where p_min = min{p₁, p₂, …, p_2k+1}.

Proof. This proof directly follows from using the Theorem 5 multiple times by increasing every probability one at a time from the initial value p_min:

$\begin{array}{r}{P}_{correct}(p_1,p_2,\dots ,p_{2k+1})\ge \\ \ge P_{correct}(p_{min},p_2,\dots ,p_{2k+1})\ge \\ \ge P_{correct}(p_{min},p_{min},\dots ,p_{2k+1})\ge \\ ⋮\\ \ge P_{correct}(p_{min},p_{min},\dots ,p_{min})=\\ ={\displaystyle \sum _{i=0}^k}\left(\genfrac{}{}{0ex}{}{2k+1}{i}\right)p_{min}^{2k+1-i}{(1-p_{min})}^i,\end{array}$

which ends the proof.

Theorems 6 and 4 were needed in order to be able to say, that combining an infinite amount of features with their probabilities p_i ∈ (0.5, 1] (min{p₁, …, p_2k+1, …} > 0.5) will lead to perfect prediction with no error for machine learning with counts, as it was in previous Section 5 when looking at the same problem from the statistical point of view (Gaussian distributions). It is important to say once again that these separate probabilities p_i have to be independent, as otherwise Equation (4) would not be valid.

Conclusion and discussion

We have derived blending coefficients for the ensemble of multiple independent prediction models with normal error distribution. This manuscript was mainly inspired by a Netflix competition, in which in the final stages of competition multiple teams joined their efforts to increase the accuracy of their final predictor and blending turned out to be essential to win the competition in a very short time during the final stage. This method was not only used in the Netflix competition but is used for other datasets in machine learning, such as MNIST and CIFAR-10 for image processing. We have also shown that having an infinite amount of independent predictors with their variances bounded from above is sufficient to achieve perfect prediction. While deep learning is very popular these days, one should not forget to include more features (going wider) when in need of improvement in accuracy.

Looking at a similar problem and more specifically machine learning with counts, where we only count how many features are for and against and make a decision based on a voting mechanism and a majority vote winner, we have shown once again, that an infinite amount of independent features will lead to perfect prediction when using only features which have >50% of accuracy.

Naturally, independent features in practice are hard to find and further the study could be made on how to convert dependent (correlated) features into independent in order to achieve as high accuracy as possible, or how to combine these dependent features together and what theoretical accuracies can be achieved. It could be also of interest how to combine features, which are not binary (for and against features), but have more than two possible outcomes.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

The author confirms being the sole contributor of this work and has approved it for publication.

Acknowledgments

We would like to thank the reviewers for their suggestions and comments which led to the improvement of the manuscript.

Conflict of interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frai.2023.1144886/full#supplementary-material

References

Abouelnaga, Y., Ali, O. S., Rady, H., and Moustafa, M. (2016). “Cifar-10: KNN-based ensemble of classifiers,” in 2016 International Conference on Computational Science and Computational Intelligence (CSCI) (Las Vegas, NV), 1192–1195.

Amatriain, X. (2013). “Big & personal: data and models behind netflix recommendations,” in BigMine '13 (Chicago, IL).

Ardabili, S., Mosavi, A., and Várkonyi-Kóczy, A. R. (2020). “Advances in machine learning modeling reviewing hybrid and ensemble methods,” in Engineering for Sustainable Future, ed A. R. Várkonyi-Kóczy (Cham: Springer International Publishing), 215–227.

Google Scholar

Bothos, E., Christidis, K., Apostolou, D., and Mentzas, G. (2011). “Information market based recommender systems fusion,” in Proceedings of the 2nd International Workshop on Information Heterogeneity and Fusion in Recommender Systems, HetRec '11 (New York, NY: Association for Computing Machinery), 1–8.

Google Scholar

Bruno, A., Moroni, D., and Martinelli, M. (2022). Efficient Adaptive Ensembling for Image Classification. Technical report, ISTI Working Paper, 2022. Consiglio Nazionale delle Ricerche.

Google Scholar

Ciresan, D. C., Meier, U., Gambardella, L. M., and Schmidhuber, J. (2011). “Convolutional neural network committees for handwritten character classification,” in 2011 International Conference on Document Analysis and Recognition, 1135–1139.

Google Scholar

Coscrato, V., de Almeida Inácio, M. H., and Izbicki, R. (2020). The NN-stacking: feature weighted linear stacking through neural networks. Neurocomputing 399, 141–152.

Google Scholar

Jahrer, M., Töscher, A., and Legenstein, R. (2010). “Combining predictions for accurate recommender systems,” in Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '10 (New York, NY: Association for Computing Machinery), 693–702.

Kay, S. M. (1993). Fundamentals of Statistical Signal Processing: Estimation Theory. Prentice-Hall, Inc.

Google Scholar

Koren, Y. (2009). The bellkor solution to the netflix grand prize. Netflix Prize Docu. 81, 1–10.

Google Scholar

Schuhen, N., Thorarinsdottir, T. L., and Gneiting, T. (2012). Ensemble model output statistics for wind vectors. Month. Weath. Rev. 140, 3204–3219. doi: 10.1175/MWR-D-12-00028.1

CrossRef Full Text | Google Scholar

Taraba, P. (2021). Linear regression on a set of selected templates from a pool of randomly generated templates. Mach. Learn. Appl. 6:100126. doi: 10.1016/j.mlwa.2021.100126

CrossRef Full Text | Google Scholar

Töscher, A., Jahrer, M., and Bell, R. M. (2009). The bigchaos solution to the netflix grand prize. Netflix Prize Docu. 1–52.

Google Scholar

Xiang, L., and Yang, Q. (2009). “Time-dependent models in collaborative filtering based recommender system,” in 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, Vol. 1 (Milan), 450–457.