Skip to main content

BRIEF RESEARCH REPORT article

Front. Public Health, 22 December 2020
Sec. Public Health Policy

Real-Time Estimation of Rt for Supporting Public-Health Policies Against COVID-19

  • 1Centre for Biotechnology and Bioengineering, Universidad de Chile, Santiago, Chile
  • 2Departamento de Ingeniería Química, Biotecnología y Materiales, Facultad de Ciencias Físicas y Matemáticas, Universidad de Chile, Santiago, Chile
  • 3Division of Chemistry and Chemical Engineering, California Institute of Technology, Pasadena, CA, United States
  • 4Laboratorio de Microbiologia Molecular, Departamento de Ciencias de la Vida, Facultad de Ciencias de la Vida, Universidad Andres Bello, Santiago, Chile

In the absence of a consensus protocol to slow down the spread of SARS-CoV-2, policymakers need real-time indicators to support decisions in public health matters. The Effective Reproduction Number (Rt) represents the number of secondary infections generated per each case and can be dramatically modified by applying effective interventions. However, current methodologies to calculate Rt from data remain somewhat cumbersome, thus raising a barrier between its timely calculation and application by policymakers. In this work, we provide a simple mathematical formulation for obtaining the effective reproduction number in real-time using only and directly daily official case reports, obtained by modifying the equations describing the viral spread. We numerically explore the accuracy and limitations of the proposed methodology, which was demonstrated to provide accurate, timely, and intuitive results. We illustrate the use of our methodology to study the evolution of the pandemic in different iconic countries, and to assess the efficacy and promptness of different public health interventions.

Introduction

Several mathematical models to fit public databases on the SARS-CoV-2 outbreak have recently been proposed (14). Despite their particularities, a great part share the same compartmental structure, inspired on the well-known SIR model (5). Besides the interest in modeling the spread of this virus, there is a need for indexes to evaluate the efforts made to prevent new cases, and to assess how likely a particular demographic group is to be infected. One of the parameters used for that means is the Basic Reproduction Number R0, which value represents the average number of persons a single infected individual infects if the population is fully susceptible and unaware of the virus (6). From its definition, R0 ≥ 1 indicates the outbreak has an exponential behavior, while R0 < 1 would account for a disappearing infection. When we study an ongoing outbreak, of which the population is aware and public measures are taking place, the parameter that represents the number of offspring infections produced by a single individual within a generation time [time between consecutive infections; (7)] is the Effective Reproduction Number Rt (8). An intuition on how it works is presented on Figure 1.

FIGURE 1
www.frontiersin.org

Figure 1. Different stages can be identified during the outbreak of an infection, characterized by the number of people that a single infected individual may infect. In the figure, a single infected individual (in red) can spread the virus among different individuals (in black), not reaching part of the population (in gray). Some individuals go into isolation (in green), effectively lowering their contagion chance. At the right-hand side of the plot, Rt represents the number of possible new infections caused by a single patient in each outbreak stage. In the first days of the outbreak, a single individual can infect several people before isolation, but as the amount of cases gets public awareness, health policies restricting movement and self-driven actions may help to control the outbreak, which is effectively captured by a decreasing Rt.

Even though several authors have claimed to have provided guidelines for obtaining both reproduction numbers (6, 911), the truth is that estimation methodologies remain somewhat cumbersome, especially for those readers untrained in mathematical modeling. The above raises a barrier to its applicability, as decision-making actors might consider it not simple enough to evaluate different public health plans. Regarding the mathematical approaches followed to estimate the different reproduction numbers, some rely on the population-level exponential growth rate (12), which, however, might work only in the early stages of the outbreak (13). Similarly, for later stages, compartmental mathematical models have been used to infer the virus's spreading rate and subsequently calculate the reproduction number (1416). Chen et al. (1) uses a time-dependant SIR-model, where the different involved rates are allowed to change over time. After numerically finding the value that minimizes a curve-fitting functional, they evaluate the basic reproduction number considering two types of spreaders. Alternatively to these approaches, Luchini et al. (17) proposes a mathematical model that allows evaluating the convexity/concavity of trends in epidemiological surveillance data, evaluating the progression of the pandemic based on changes between acceleration and deceleration based on epidemiological metrics. The resulting plots of convexity or concavity, which would describe pandemic acceleration or deceleration, can be used as an adjunct criterion to the analysis of surveillance data to assess the pandemic's evolution (18).

In the present work, we propose a useful and simple methodology to calculate Rt directly from available epidemiological data in real time during an outbreak. The key feature of this practical methodology is that no specific knowledge in mathematics or scientific computation is needed to generate estimations of this parameter, thus being particularly handy for its use for day-to-day assessment in public health matters. We provide numerical insights on the accuracy of the proposed methodology, analyzing its stability considering different controlled levels of Gaussian noise. As our methodology does not involve a parameter fitting stage, which would be needed if solving the SIR system numerically to represent continuous trends, we can use it to evaluate the immediate impact of the different actions used to prevent the spread of SARS-CoV-2. In a case study, we assess the effect on Rt of the different ongoing measures taken by the Chilean government, and we compare their result with the current panorama of different iconic countries.

Methods

Assuming the outbreak follows approximately an SIR model (Equations 1–3), we consider three compartments: susceptible S, healthy individuals, susceptible to be infected, infected I, individuals that have already contracted the virus, and removed R, individuals that are no longer infectious nor susceptible. Therefore, the dynamics are represented by:

S=-β(t)SIN,    (1)
I=β(t)SIN-γ(t)I,    (2)
R=γ(t)I.    (3)

In particular, Equations (1) and (2) can be combined applying the chain rule and the derivative of the inverse function theorem, so we can write:

dIdS=β(t)SIN-γ(t)I-β(t)SIN      dIdS=-1+γ(t)β(t)NS.    (4)

In terms of the parameters of the SIR model, we can calculate Rt as the ratio between the time-dependant infection and recovery rates, β(t) and γ(t), respectively (1, 9), multiplied by the probability of finding a susceptible individual (SN):

Rt(t)=β(t)γ(t)SN.    (5)

Note that, from its definition, not only variations in the infection and recovery rates affect the value of the effective reproduction number Rt, but also the depletion of the susceptible pool. Using Equation (5), γ(t)β(t)NS=1Rt(t). Therefore, Equation (4) can be re-written as

dIdS=-1+1Rt(t).    (6)

Equation (6) can be discretized in an interval [ti−1, ti] where we can assume that Rt(t) = Rt(ti) is constant:

Rt(ti)=1ΔiIΔiS+1.    (7)

Extending the classical SIR model to consider also deaths, a population balance dictates the discrete differences to follow ΔiS + ΔiI + ΔiR + ΔiD = 0. Then, Equation (7) takes its final form.

Rt(ti)=11-ΔiIΔiI+ΔiR+ΔiDRt(ti)=ΔiIΔiR+ΔiD+1.    (8)

The last expression can be further simplified noting that ΔiI = (−ΔiS) − (ΔiR + ΔiD), namely, the variation on active cases ΔiI equals new infections (−ΔiS) minus the new -clinical- recoveries plus new deaths (ΔiR + ΔiD). Therefore, Equation (8) can also be expressed as:

Rt(ti)=-ΔiSΔiR+ΔiD,    (9)

or, in words,

Rt(ti)=New InfectionsNew Recoveries + New Deaths    (10)

This last expression is particularly intuitive, it is straightforward to apply to understand the disease progression: when Rt > 1 there are more new infections than recoveries, thus the number of infecting individuals in the population is increasing, while for Rt < 1 the number of infecting individuals must be decreasing for the opposite reason.

Reach, Advantages, and Limitations of the Proposed Methodology

Equation (8) stands in front of other methods because of its simplicity and usability, as there is no need for specific mathematical or scientific computing knowledge for obtaining realistic values of Rt for a given population during an epidemic or pandemic outbreak. However, due to the nature of its dependence on real-time data, uncertainties on the input values would have a significant effect on the outcome. A priori, several practical uncertainty sources could affect the different variables that determine Rt, mainly related to the non-uniformity in the case-reporting protocols, temporal delay on diagnosis, and other report or measurement biases (19). Since most common uncertainties are related to the time frame between contagion, sampling, detection and report (temporal misclassification), recovery criteria (PCR or fix quarantine time after which the patient is considered healed), we suggest to perform a data-pretreatment stage, where delayed curves could be corrected to the day of contagion before applying moving averages to smooth trends and estimate their variability while Rt. The use of an one-week averaging window is suggested for minimizing the impact of the “weekend-effect” (20).

Another limitation of the proposed methodology relates to the low-case limit, where the different quantities involved in Equation (8) are (or are close to) zero. This situation may happen in the event of a sudden outbreak or a second wave in an already declared pandemic, where sensible sanitary measures in order to control the spread should be taken disregarding the value of any model-derived parameter. Moreover, in such cases, where contacts are known, and the number of cases remains low, contact-tracing is useful and should take place disregarding the calculated value of Rt (21). As this parameter relies on the hypotheses of the SIR model, it should be used with care; countries with strong heterogeneity between regions, the existence of different delays both in the infectious timeline and in the reporting procedure might affect its value (14, 19).

Estimations of the effective reproduction number with other methods do not necessarily match in value the ground-truth reproduction number of the disease Rt (22). Yet, they expose the same tipping point when they reach the value 1. This is the typically addressed tipping point which separates exponential growth from exponential extinction at the first stages of an outbreak. Nevertheless, other dynamics than exponential can arise from non-pharmaceutical interventions (NPIs), as the test-trace-and-isolate strategies (23), and in later stages of the outbreak.

Results

On the Accuracy of the Proposed Methodology

To evaluate the accuracy of the proposed methodology, we set up a numerical experiment to infer known values of Rt using a simulated SIR model with different levels of Gaussian noise and several Rt change points as a ground-truth. These change points may account for notable drifts in the population's behavior or for the application of different non-pharmaceutical interventions (NPIs). The standard methodology consisted of the use of 1-week rolling averages for the calculated Rt, and we simulated different levels of noise to be added to the ground-truth ΔI and ΔR values. For a noise level l = {5%, 10%, 25%}, we defined noisy variables (ΔI)noise = (ΔI)model(1 + lN(0, 1)), and we took random samples in a Monte Carlo-like experiment to build up confidence intervals (Figure 2, nsample = 500). We can observe that, as the change points are discontinuous for Rtdata, the relative error is higher in the transitions. However, the method quickly captures the real Rt value for the different levels of noise in a period of less than the averaging-window time. This readily demonstrates the accuracy and robustness of the proposed methodology to capture different values of the time-dependent reproduction number Rt under the assumptions of this work.

FIGURE 2
www.frontiersin.org

Figure 2. Numerical accuracy of the proposed methodology. We try to infer the values of Rt of a noisy time series generated from a SIR model with several change points spaced by 4 weeks (dotted red line–ground truth). (A,D), (B,E), and (C,F) represent the effective reproduction number and the relative error for 5%, 10%, and 25% proportional noise, respectively. The spreading rate β is such that, for a fixed recovery rate γ = 0.1 we obtain the ground-truth Rt = [1.25, 1.15, 1.25, 1.00, 0.90, 0.75] plotted. Initial conditions, N = 50 million people, I0 = 100, S0 = NI0, and R0 = 0 individuals.

Effect of Non-pharmaceutical Interventions (NPIs) on the Spread of COVID-19

Countries that successfully control case numbers should exhibit Rt trends consistently lower than 1. A quick control of the viral spread can be recognized by an earlier decrease in their Rt trends when comparing them from the day of the first reported infection (Figure 3B). The efficacy of Non-Pharmaceutical Interventions can be assessed by the magnitude of the negative slope of the curves. Most countries also show second viral spread waves and even third ones. Noteworthy, those waves seem to be uniformly separated by a period of 2 months, even in cases like Germany, in which low amplitude waves took place with Rt slightly above the control threshold. South Korea seems again to be an exception, with second and third waves separated by approximately 3 months. There is also a notable difference in how second and third waves develop considering western and eastern countries, with the former appearing later than in China and Korea. Regarding peak amplitudes, it is noteworthy that in second and third waves, China, South Korea, and Italy (together with other non-shown European countries) show higher peaks than in other nations. However, this might also be a numerical effect since estimation methodologies for Rt do not agree well in the low case-number scenario. German and Chilean cases, among the countries shown in Figure 3A, seem to have been more successful in controlling the appearance of new spread waves, although using very different NPIs. However, Germany is lately following an ascending trend. In the case of Chile, the Government adopted a strict lockdown approach with border closure made effective 2 weeks after detection of the first case, compulsory quarantine with restriction of travel, movement, and meeting of people inside and between areas with higher case rates, restrictions in the commerce and closure of public and private schools, universities, restaurant, cinemas, shops, and public parks and other heavily transited meeting places, and compulsory use of face masks and hand disinfection measures in public places. Restrictions were applied differently by city, area, and region, depending on the number of cases, and changing over time. In Germany, governmental measures were radically different, and did not include severe lockdown measures or quarantines, relying only on people's compliance, high test rates, and high-surveillance and contact tracing policies.

FIGURE 3
www.frontiersin.org

Figure 3. Comparative analysis of the values of Rt for different countries, using a moving average window of ± 3 days, in both daily reports and the calculated Rt. The control threshold is represented by a horizontal red dashed line (Rt = 1). (A) Daily trends for Rt in different countries, from March 28 to October 13. (B) Rt trends from the day of detection of the first case, and their standard deviation in the moving average time-frame (confidence interval defined as mean plus/minus standard deviation). Note that the plot starts on day 50 after the first case report. Official data from Worldometers.info (October 13, 2020).

Additionally, different sets of governmental measures were applied in Chile during the first months, sometimes with unfortunate public announcements that lead people to misinterpret the evolution of the pandemics and related measures, which are reflected by changes in the Rt number in time. For instance, in March 27, the Chilean Government declared a partial quarantine for districts with high case-density in Santiago (capital city concentrating 42% of the national population) and other cities, which were slowly applied to other districts. From May onward, contagion in Chile shows a steep increase that was maintained for a couple of weeks but then decreased to approach the control threshold, which can be related to new quarantine measures applied afterwards to more districts in Santiago and other cities. We can observe a somewhat irregular oscillatory behavior around Rt = 1, reflecting the efforts of the step by step governmental plan, which systematically release restrictions to boost economical activities, from July 19 on (24).

The USA case deserves special attention. Besides showing a comparatively slower and late decrease in Rt to control the first wave, which took more than three months to be lowered, revealing an inefficient approach to the control threshold, with only short periods around or below 1. Since the country's total population is high, values of Rt above one represent a catastrophic effect in terms of risk to the general population, with 51,564 new cases in October 13th (25).

Discussion

We have developed a fast and simple methodology to estimate the Effective Reproduction Number Rt directly from raw real-time data of an evolving epidemic outbreak. Our results have also shown that this index can be a useful decision parameter to evaluate the impact of Non-Pharmaceutical interventions (NPIs) in controlling COVID-19. We studied the accuracy of the proposed methodology in a numerical experiment, concluding that results remain precise even when including a white noise of 25%. The simplicity of the proposed method to calculate Rt (Equation 8) remarks its applicability, and our analysis of Rt trends in different countries during the current SARS-CoV-2 outbreak highlights how it can be applied to assess both the speed of reaction and the efficacy of public-health measures. This provides decision-makers with a simple and easily calculable tool to timely understand the impact of their policies. As the proposed equation does not need vast volumes of data, it results particularly handy for its use when data resolution is not high enough to fit continuous models, in the analysis of short-time trends, or to compare different regions in the world or even inside a single country with different time density of data. This parameter should be calculated only for populations with shared characteristics, as it relies on the hypotheses of typical compartmental models (14). Countries with high heterogeneity in population density, climate, and behavior of different economic classes should be studied with care. The epidemiologic scenario of those zones concentrating the most significant part of the population would mask the spreading dynamics' local features. This case is especially highlighted in Chile, where the national average is dragged by the city of Santiago, not reflecting the situation of the most extreme regions, where the progress of the infection has been exponential even after remission in the capital, which in turn warns of the need to analyze local data in the same detail as national data. Consistently, as varied as the uses for the proposed methodology are the opportunities to improve it.

We look forward to seeing how this contribution of a real-time estimator of Rt would impact how we analyze the ongoing contingency and how the scientific and decision-making community would adapt it to tailor propagation models and obtain better and timely insights on the application of emergency public-health policies.

Data Availability Statement

Publicly available datasets were analyzed in this study. This data can be found at: https://www.worldometers.info/coronavirus/; https://www.gob.cl/coronavirus/cifrasoficiales/.

Author Contributions

SC, DM-O, and HV: conceptualization, methodology, and investigation. ÁO-N, SC, and CS: validation. SC, DM-O, HV, and ÁO-N: writing, review, and editing. ÁO-N and CS: supervision and project administration. ÁO-N: funding resources. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This manuscript has been released as a pre-print at https://www.medrxiv.org/content/10.1101/2020.04.23.20076984v1 (26). This research has been financed mainly by the Centre for Biotechnology and Bioengineering—CeBiB (PIA project FB0001, Conicyt, Chile). DM-O gratefully acknowledges Conicyt, Chile, for Ph.D. fellowship 21181435. The authors thank Prof. Francisco Melo for the helpful discussions on previous versions of our manuscript.

References

1. Chen Y-C, Lu P-E, Chang C-S, Liu T-H. A time-dependent SIR model for COVID-19 with undetectable infected persons. IEEE Trans Netw Sci Eng. (2020) 1. doi: 10.1109/TNSE.2020.3024723

CrossRef Full Text | Google Scholar

2. Simha A, Prasad RV, Narayana S. A simple stochastic sir model for Covid 19 infection dynamics for Karnataka: learning from Europe. arXiv [Preprint] arXiv:2003.11920 (2020).

Google Scholar

3. Calafiore GC, Novara C, Possieri C. A modified sir model for the covid-19 contagion in Italy. arXiv [Preprint] arXiv:2003.14391 (2020).

Google Scholar

4. Yang Z, Zeng Z, Wang K, Wong SS, Liang W, Zanin M, et al. Modified SEIR and AI prediction of the epidemics trend of covid-19 in China under public health interventions. J Thorac Dis. (2020) 12:165–74. doi: 10.21037/jtd.2020.02.64

CrossRef Full Text | Google Scholar

5. Kermack WO, McKendrick AG. A contribution to the mathematical theory of epidemics. Proc R Soc Lond Ser A. (1927) 115:700–21. doi: 10.1098/rspa.1927.0118

CrossRef Full Text | Google Scholar

6. Perasso A. An introduction to the basic reproduction number in mathematical epidemiology. ESAIM Proc. Surveys (2018) 62:123–38. doi: 10.1051/proc/201862123

CrossRef Full Text | Google Scholar

7. Svensson Å. A note on generation times in epidemic models. Math Biosci. (2007) 208:300–11. doi: 10.1016/j.mbs.2006.10.010

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Medina-Ortiz D, Contreras S, Barrera-Saavedra Y, Cabas-Mora G, Olivera-Nappa Á. Country-wise forecast model for the effective reproduction number RT of coronavirus disease. Front Phys. (2020) 8:304. doi: 10.3389/fphy.2020.00304

CrossRef Full Text | Google Scholar

9. Heesterbeek JAP. A brief history of r 0 and a recipe for its calculation. Acta Biotheor. (2002) 50:189–204. doi: 10.1023/A:1016599411804

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Delamater PL, Street EJ, Leslie TF, Yang YT, Jacobsen KH. Complexity of the basic reproduction number (r0). Emerg Infect Dis. (2019) 25:1. doi: 10.3201/eid2501.171901

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Breban R, Vardavas R, Blower S. Theory versus data: how to calculate r0? PLoS ONE (2007) 2:e282. doi: 10.1371/journal.pone.0000282

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Mills CE, Robins JM, Lipsitch M. Transmissibility of 1918 pandemic influenza. Nature. (2004) 432:904–6. doi: 10.1038/nature03063

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Ma J, Dushoff J, Bolker BM, Earn DJ. Estimating initial epidemic growth rates. Bull Math Biol. (2014) 76:245–60. doi: 10.1007/s11538-013-9918-2

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Contreras S, Villavicencio HA, Medina-Ortiz D, Biron-Lattes JP, Olivera-Nappa A. A multi-group SEIRA model for the spread of COVID-19 among heterogeneous populations. Chaos Solit Fractals (2020) 136:109925. doi: 10.1016/j.chaos.2020.109925

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Asif M, Jan SU, Haider N, Al-Mdallal Q, Abdeljawad T. Numerical modeling of npz and sir models with and without diffusion. Results Phys. (2020) 19:103512. doi: 10.1016/j.rinp.2020.103512

CrossRef Full Text | Google Scholar

16. de Oliveira MC. Estimating the effective reproduction number and variables of disease models for the covid-19 epidemic. arXiv [Preprint] arXiv:2008.08201 (2020).

Google Scholar

17. Luchini S, Teschl M, Pintus P, Baunez C, Degoulet M, Moatti JP. Urgently needed for policy guidance: an operational tool for monitoring the COVID-19 pandemic. SSRN. (2020). doi: 10.2139/ssrn.3563688. [Epub ahead of print].

CrossRef Full Text | Google Scholar

18. Edjoc R, Atchessi N, Lien A, Smith BA, Gabrani-Juma I, Abalos C, et al. Assessing the progression of the covid-19 pandemic in canada using testing data and time-dependent reproduction numbers. Can J Publ Health. (2020) 111:926–938. doi: 10.17269/s41997-020-00428-w

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Contreras S, Biron-Lattes JP, Villavicencio HA, Medina-Ortiz D, Llanovarced-Kawles N, Olivera-Nappa A. Statistically-based methodology for revealing real contagion trends and correcting delay-induced errors in the assessment of COVID-19 pandemic. Chaos Solit Fractals (2020) 139:110087. doi: 10.1016/j.chaos.2020.110087

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Contantinesco A, Israel-Jost V, Choquet P. Spectral analysis of the daily evolution of deaths due to covid-19 in france and in the world shows a weekend effect: myth or reality? medRxiv (2020). doi: 10.1101/2020.06.23.20135442

CrossRef Full Text | Google Scholar

21. Lunz D, Batt G, Ruess J. To isolate, or not to isolate: a theoretical framework for disease control via contact tracing. medRxiv. (2020). doi: 10.1101/2020.05.26.20113340

CrossRef Full Text | Google Scholar

22. Dehning J, Spitzner FP, Linden MC, Mohr SB, Neto JP, Zierenberg J, et al. Model-based and model-free characterization of epidemic outbreaks. medRxiv [Preprint]. (2020). doi: 10.1101/2020.09.16.20187484

CrossRef Full Text | Google Scholar

23. Contreras S, Dehning J, Loidolt M, Spitzner FP, Urrea-Quintero JH, Mohr SB, et al. The challenges of containing sars-cov-2 via test-trace-and-isolate. arXiv [Preprint] arXiv:2009.05732 (2020).

Google Scholar

24. MINSAL (2020). Reporte Coronavirus 13/10/2020. Tech. rep., Ministerio de Salud, Chile.

25. [Dataset] Worldometers.info (2020). Official Numbers for the Coronavirus Outbreak. Available online at: https://www.worldometers.info/coronavirus/ (accessed October 13, 2020).

26. Contreras S, Villavicencio HA, Medina-Ortiz D, Saavedra CP, Olivera-Nappa A. Real-time estimation of r0 for supporting public-health policies against covid-19. medRxiv. (2020). doi: 10.1101/2020.04.23.20076984

CrossRef Full Text | Google Scholar

Keywords: COVID-19, public-health policies, epidemiologic modeling, SARS-CoV-2, effective reproduction number Rt

Citation: Contreras S, Villavicencio HA, Medina-Ortiz D, Saavedra CP and Olivera-Nappa Á (2020) Real-Time Estimation of Rt for Supporting Public-Health Policies Against COVID-19. Front. Public Health 8:556689. doi: 10.3389/fpubh.2020.556689

Received: 13 May 2020; Accepted: 30 November 2020;
Published: 22 December 2020.

Edited by:

Joao Soares Martins, National University of East Timor, Timor-Leste

Reviewed by:

Gregory Dore, University of New South Wales, Australia
Jakob Ruess, Institut National de Recherche en Informatique et en Automatique (INRIA), France

Copyright © 2020 Contreras, Villavicencio, Medina-Ortiz, Saavedra and Olivera-Nappa. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Álvaro Olivera-Nappa, aolivera@ing.uchile.cl

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.