Here, we will illustrate the effects of FLE using an asymptotic EXP distribution. The same discussion can be generalized to other distributions (see Supplementary Information section 1).

The EXP distribution is defined as

with β as a sole parameter for x ∈ [x_min, ∞). Maximizing the likelihood function 𝕃=∏i=1NP(X=xi|xmin,β^), we find the estimated parameter

If we use the mean obtained from data 〈x〉_data as 〈x〉 in Equation (8) we will obtain the unadjusted estimator β_unadj. However, due to the FLE, we can only average up till x_max. As such 〈x〉_data will be biased downwards and Equation (8) over-estimates β^.

To adjust for the FLE, we add the truncated part back into 〈x〉_data, to define the adjusted 〈x〉_adj as

Inserting 〈x〉_adj into Equation (8), we obtain a nonlinear equation

that we solve using MATLAB's builtin nonlinear solver function nlinfit() to obtain β^adj.

To test the performance of this adjustment formula, we simulated 1, 000 sets of EXP distributed data for 10-4≤βT≤102, by using the inverse cumulative function for EXP distribution

This transforms U(0, 1) distributed elements {u_i} to EXP distributed elements {x_i}. Using this transformation FEXP-1, 0 and 1 map to x_min and ∞ respectively. It is also useful to note that Equation (11) is the inverse of the CDF of the EXP distribution,

To simulate the effect of a FLE with xmax=FEXP-1(0.9), we sampled 1,000 sets of EXP distributed data using U(0, 0.9) instead of U(0, 1) with x_min = 0. Thereafter, we estimated β^unadj and β^adj using Equations (8) and (10). Figure 2 shows the relative estimation errors

of β^unadj and β^adj with respect to the true beta β_T. As we can see from the Figure 2, Δβ^unadj is about 38% for small samples N ~ 10² and decreases to 34% for large samples N ~ 10⁴. On the other hand, Δβ^adj starts at 20%, but decreases to 2% as the number of data points is increased. Although it can be shown that the bias of β^unadj vanishes with increasing sample sizes [24, 39], we find it converging very slowly with increasing sample size in the unfortunate situation of a small x_max. In contrast, β^adj converges very quickly even for small x_max as we have accounted for the FLE.

In the Supplementary Information section 1, we show details for our derivation of the theoretical estimation

By defining xmax=xmin-βT-1ln(δ), and substitute x_max in Equation (14) with δ, the theoretical relative estimation error is expressed as

Equation (15) shows that the estimation error has no explicit dependence on sample size. This tells us that the β^unadj is always larger than the β_T because of the FLE effect. The convergence rate then depends on how rapidly x_max approaches infinity (δ approaches zero) with increasing sample size.

For a finite sample, F_EXP(x) < 1 for all x < ∞. Mathematically, this means that F_EXP(x) ~ U(0, 1 − δ), where FEXP-1(xmax)=1-δ. This observation is important, because d_KS is obtained by comparing U(s)={ui(s)=FEXP(xi|β^)}i=1N against U(0, 1) (see Equation 4). This tell us that for a fair comparison, we need to rescale all elements in U^(s) by a factor of 1/(1−δ). Figure 3 shows the d_KS measured for the 1000 sets of EXP distributed data with finite largest element xmax=F-1(0.9) for various β_T and sample sizes N. For each sample, we use Equation (10) to estimate the β^adj and transformed this data to U^(s) using Equation (12). After that, we measure d_KS with Equation (4) to obtain unadjusted KS distance, KS_unadj and adjusted KS distance, KS_adj using the non-rescaled and rescaled U^(s), respectively. KS_unadj goes from 0.14 for small samples N ~ 10², to 0.10 for large samples N ~ 10⁵. In contrast, KS_adj decrease from 0.06 for small samples to 0.006 for large samples.

Until now, we have only discussed adjustments to the estimated parameter and the KS distance to eliminate the bias caused by the FLE. Besides the FLE effect, we also need to consider the bias caused by having a finite number of elements in the sample. As we can see from Figure 3, the KS distance decreases as the sample size increases. Therefore, in order to have a fair comparison of the goodness of fit for various sample sizes, we need to determine how d_KS changes as a function of N. To do this, we simulated 10⁶ samples of various sizes N from U(0, 1). For each sample we determined d_KS using Equation (4), so that for each N we end up with 10⁶ KS distances. In Figure 4 we show the KS distances at different deciles, which exhibits the asymptotic behavior

that we settled for, after experimenting with several functional forms (see Supplementary Information section 2). This result agrees with our expectation that d_KS → 0 as N → ∞. It also suggests that if we have two samples with sizes N₁ and N₂ from the same distribution, we should compare N10.492dKS(1) against N20.492dKS(2). Otherwise, if N₂ > N₁ then naturally dKS(2)<dKS(1) and we will be lead to the wrong conclusion that the N₂ sample fits the distribution better.

In this section, we presented explicitly the procedures to obtain the adjusted parameter, as well as the steps to perform significance testing on this estimated parameter. Although we demonstrated this explicitly using the EXP distribution as an example, one should note that this method can also be applied to other distributions. The inclusion of x_max when fitting empirical data have been previously considered by [40–42] for the truncated PL distribution. Like these, the method presented in this paper can be easily extended to fit different distributions, but unlike these, we can easily conduct significance testing across them. This is because by extending x_max to infinity, we can compute the probability integral transform to map arbitrary distributions to the standard uniform distribution, and ensure that during statistical significance testing our goodness-of-fit measure can be distribution independent [see CSN(ii)].

More importantly, fitting data to untruncated distributions defined over [x_min, ∞) is commonly encountered in practice, where no x_max is expected from theoretical considerations, but the largest element in our data is finite. If we fit to the truncated versions of the distributions, we might get better estimates of the distribution parameters, but we will not be able to justify inserting these estimates into the untruncated distributions, in the absence of a limiting procedure involving larger and larger x_max. Moreover, when researchers expect to be dealing with the untruncated distribution, they will not use the truncated distribution for estimation. In contrast, our self-consistent adjustment procedure would be ontologically easier to justify.

As with section 3.3, we simulated 10⁶ samples from U(0, 1) with different N. For each sample, we calculate the distribution noise d_DN using Equation (18) and plot its deciles against N as shown in Figure 6. After experimenting with several functional forms, we write down the relationship between d_DN and N at percentile ℘_DN as

where Φ(x) represents the sign of x, and

is the analytically derived distribution noise, that converges to 1/2 as N → ∞ (refer to Supplementary Information section 2 for more details). This result suggests that if we have two samples with sizes N₁ and N₂ with N₂ > N₁ from the same distribution, we should compare N10.495(dDN(1)-1/2) against N20.495(dDN(2)-1/2). Otherwise, we risk making the wrong conclusion that the N₂ sample fits the distribution better if dDN(1)>dDN(2).

As measures for statistical deviations, d_DN and d_KS are different in that d_DN measures deviation at the probability density level, whereas the d_KS measure it at the cumulative density level. As a result, d_KS assigns more weight to the tail of the distribution, while d_DN is more sensitive to deviations in the body of the distribution. Therefore, if we wish to combine these two measures to estimate the significance level, we need to first investigate the relationship between d_KS and d_DN. We do this by simulating 10⁶ samples from U(0, 1) for various sample sizes, and for each sample, we calculate d_KS and d_DN using Equations (4) and (18) respectively, to obtain 10⁶ pairs of d_KS and d_DN. We then compute the Pearson correlation between d_KS and d_DN and learned that (see Supplementary Information section 2 for the comparison of fits)

As expected, d_KS is positively correlated with d_DN. Since d_KS is a measure at the cumulative level, the random distribution noises cancel each other, thus the correlation between d_KS and d_DN vanishes as N → ∞.

To perform significance testing given d_KS and d_DN, we need the percentile values ℘_KS and ℘_DN. ℘_KS can be obtained by inverting Equation (16), as

Similarly, we invert Equation (19), and solve

to get ℘_DN, where η = d_DN − 〈d_DN〉.

Substituting the empirical KS distance dKS(em) and empirical distribution noise dDN(em) into Equations (22) and (23), we obtain ℘KS(em) and ℘DN(em). This is an alternative way of obtaining the p-value without the need to perform Monte-Carlo (re)sampling again (CSN method), since we have already done so in sections 3 and 4. The percentage of simulated U(0, 1) samples with dKS/DN>dKS/DN(em) is 100−℘KS/DN(em). Since d_KS and d_DN are not independent (Equation 21), we discount the correlation between d_KS and d_DN, and define the significance level (p-value) as

to avoid overestimating the significance level.

We follow the steps outlined in the CSN algorithm (section 2) to fit the empirical data, but with two important modifications: (Ii) the parameters (CSN(ib)) and goodness of fit (CSN(iib)) are adjusted for the finite largest element; and (Iii) the p-value (CSN(ivc)) is adjusted for the finite number of elements effect. Meanwhile, optional modifications are (Oi) to incorporate distribution noise as another dimension for goodness of fit, so that the p-value can be determined via dKS(em), dDN(em), or both; (Oii) instead of using bootstrapping to determine the p-value in the CSN method, which is very slow for large samples, one can use the fast inversion formulae Equations (22), (23), or (24).

Figure 7 shows the fits and p-testing results for Taiwan housing price, Taiwan wealth, and Straits Times Index normalized returns. It is reassuring that after modifications the p-values of all distributions increased. In particular, the two distributions (Figures 7B,C) that did not meet the p > 0.1 criterion (as suggested by Clauset et al. [24]) before modification, now have p > 0.5. This is in agreement with our visual assessment of the three fits. We also understand now that a large δ (small x_max) is the main reason for Taiwan wealth to fail p-testing before adjustment (although the fit is visually good). In general, our correction formulas perform the best when δ is large due to small sample sizes or truncations. Readers can refer to Supplementary Information section 4 for more plots and instances where small δ values affects the significance testing.

There are several limitations one should note while obtaining P_KS/DN using Equations (22) or (23). First, it is only applicable to large samples (see Figures 4, 6). Second, these equations are obtained after experimenting with several functional forms and are only approximate. Lastly, p_KS measured using the CSN method are consistently smaller than that based on Equation (22). This is due to the CSN algorithm having an extra step to select x_min that minimizes d_KS of each simulated sample, and thus the algorithm is stricter than our fast inversion formulae. However, the inversion formulae Equations (22) and (23) are convenient and provide an upper bound for P_KS/DN. We make the codes for the procedures used in parameter estimation and significance testing available at https://github.com/BoonKinTeh/StatisticalSignificanceTesting for both these two methods, but leave it to the reader to decide which method to use.

All in all, when we test for statistical significance, we need to be aware of finite sample effects, namely the finite largest element effect and the finite number of elements effect. Beyond the KS distance measured at the cumulative distribution level, we also introduce an alternative measure of the goodness of fit based on the distribution noise at the probability density level.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.