David M. Vickers
John Hardie2
Claudia Chaufan
Steven Pelech- 1Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- 2Retired, Ottawa, ON, Canada
- 3Private Practitioner, Ottawa, ON, Canada
- 4Health Policy and Global Health, Faculty of Health, York University, Toronto, ON, Canada
- 5Department of Medicine, University of British Columbia, Vancouver, BC, Canada
- 6Kinexus Bioinformatics Corporation, Vancouver, BC, Canada
Introduction
Canada's early response to COVID-19 primarily employed non-pharmaceutical interventions (NPIs)—school and business closures, physical distancing measures, stay at home orders, and mandatory public masking—despite little evidence supporting them (1, 2). These responses were not part of existing pandemic plans (3). Indeed, they ignored the basic principles of pandemic planning, which is to minimise serious illness and deaths, as well as limit societal disruptions (4). Canada and the world need detailed analyses of the effectiveness and the costs of the NPIs used to try to control COVID-19. Such analyses must be separated from politics and evidentially based on comprehensive data sets.
Unfortunately, a recent article entitled “Counterfactuals of effects of vaccination and public health measures on COVID-19 cases in Canada: What could have happened?” published in the Canadian Communicable Disease Report by Ogden et al. (5) contains questionable results that sully the evaluation of these important issues. Instead of performing a much-needed analysis of real-world data, Ogden et al. (5) focus on simulation to postulate how the COVID-19 pandemic would have affected Canada had certain public health measures not been implemented. Phrases like “This study illustrates what may have happened…”, and “Canada could have experienced…” are repeatedly expressed by the authors.
While dynamical models can be useful policy simulators, the insights derived from them can all-too-often become the codified opinions of the modeller (6). In this case, the paper by Ogden et al. (5) is less likely an objective lens into the past than a loss of critical distance from an overly complicated model. In this commentary, we discuss selected authors' assumptions and conclude that they are problematic at-best. We first identify empirically antiquated and conceptually ambiguous justifications supporting the article; second, we critique the model used to substantiate the article's conclusions; and finally, we elaborate on the implications of the authors' position for effective public health policy and human wellbeing in times of crisis.
The justifications
In the “Introduction”, the authors assert that at the start of the pandemic, humans had no known immunity to SARS-CoV-2 (5). However, if humans were truly immunologically naïve to SARS-CoV-2, then it becomes difficult to explain-away the existence of cross-reactive neutralising IgG antibody (7) or memory T-cells (8, 9) to conserved epitopes between endemic coronavirus and SARS-CoV-2 proteins. These latter T cells will activate, expand, and produce IL-2 and IFN-γ when stimulated (in vitro) with cognate antigen (8), which very possibly represents a larger central memory response capable of impacting COVID-19 severity (10). Thus, at the very least, exposure to other human coronaviruses, prior to March 2020, appears able to confer measurable levels of immunity that was not considered—even in sensitivity analyses—by Ogden et al. (5) Similar immune responses to the 1918 strain of influenza existed 90 years later (11), and SARS-CoV-2 is unlikely to be the exception.
The authors also state that, unless the public health measures had been adopted, the consequences for Canadians and for Canada's public health system would have been “dire” (5). This claim, which is predicated on other modelling output (12), assumed that because of no effective therapies, Canada experienced an infection fatality rate (IFR) “approaching” 1%—contrasting it with a much lower IFR of 0.4% for seasonal influenza. While the supporting references used by Ogden et al. (5) estimate the case fatality rate (see their main text references 10, 11, 35), other empirical evidence indicates that the IFR for COVID-19 could be much lower than 1%. For instance, Ioannidis (13) determined that the IFR was 0.15–0.20% globally, and 0.03–0.04% for those under 70 years of age. Canadian-specific estimates reported by Ioannidis were 0.59% (overall) and 0.08% (for people aged < 70 years). People aged >70 years only account for 12.9% of the population in Canada (14). Similarly, results from the COVID-19 Forecasting Team (15) were compatible with unadjusted global IFRs between 0.0054% and 0.43% for people aged < 50 years, while Canadian-specific, age-adjusted cumulative IFRs decreased from 0.54% (in April 2020) and 0.35% (in January 2021). As for the alleged lack of effective therapies, early in 2020 the international literature indicated that early outpatient treatment with multidrug therapy successfully reduced hospitalisation and death, even among populations considered at high risk because of occupation, underlying health conditions, or age (16–20).
The article's “Chronology of the Epidemic and Public Health Measures” justifies the adoption of public health measures based on the “unrestrained SARS-CoV-2 transmission in Italy.” This implies the rampant spread of the infection throughout the entire population. However, Professor Walter Ricciardi, Italy's scientific advisor, corrected this assumption when he noted that “The median age of people infected with SARS-CoV-2 who [were] dying in Italy has been 80 years, and the average age of patients requiring critical care support [was] 67 years” (21). Ricciardi's point on risk-stratification provides an entirely different perspective on the epidemic's characteristics, and if true, nullifies a key justification for the author's model—a virus that if “unrestrained” would be an equal opportunity killer. As well, as shown in their Table 2, Ogden et al. (5) assume that the listed countries have agreed on what constitutes a COVID-19 death. However, they overlook that until recently, a COVID-19 death was anyone dying with, not necessarily from COVID-19 (this was corrected in Ontario March 11, 2022) (22). They also overlook the substantial differences, worldwide, in definitions of COVID-19 deaths (e.g., China likely counts them very differently than Canada).
Model structure
Ogden et al. (5) constructed an agent-based model (ABM) for understanding any benefits of Canada's pandemic response. ABMs are computational models for simulating the actions and interactions of autonomous agents (such as people) to understand the behaviour of a system and the processes that govern any outcomes of interest (23). Because ABMs maintain distinct information on every individual in a simulated population, their finer-grained nature allows them to represent certain types of activities, relationships, and interventions with greater precision and flexibility than with more traditional differential equation models used to study pathogen transmission (23, 24). However, the granularity of ABMs carries with it an obligation to understand the minutiae that links a model's structure with its behaviour (24). Ogden et al. (5) have circumvented this necessity by filling-in details of people's movements that are exogenous to the structure of the model, rather than being driven by factors or feedbacks that are internal to its current state. This expanded effort in deepening their model has inhibited a critical broadening of the model boundary that will surely limit its validity (6, 24).
For example, the behaviour of the model's agents has an all-or-none relationship with public health measures: when in-place, people changed their behaviour proportionately to the value Oxford Stringency Index; when not in-place, agents maintain their status quo. This forced dependence on government orders is contrary to other research demonstrating that people's inherent fear of infection, alone, will cause them to alter their behaviour rapidly (25), and often before strict rules are in-place (26).
One consequence of the exogeneous drivers of people's behaviour is that two model-derived metrics have diverged substantially from publicly available data. Cumulative case and death rates reported by April 2022 were approximately 9,000 cases and 98 deaths per 100,000 people, respectively (27). When compared to the estimated values in the “observed baseline” scenario in their Table 3, both these empirical values lie outside the 95th percentiles of their modelled counterparts. This directional bias in model output is indicative of structural errors (28) that could distort comparisons between the calibrated (baseline) model and its counterfactual scenarios.
Over-reliance on counterfactual scenarios
Like all simulation modelling, ABM approaches require that we depart from real-world environments (e.g., workplaces and schools) and create idealised agents and environmental conditions (29, 30). When Ogden et al. discuss Canada's experience with COVID-19 without restrictive measures or vaccination, they failed to note at least two problems. First, “up to 800,000” COVID-19 deaths are greater than any other historic event in Canada over the last 108 years, corresponding to higher per capita death rates than for all Canadian lives lost in World War I, the 1918 influenza pandemic, and World War II (see Table 1) (31–33). Second, the addition of 800,000 deaths would have large consequences for all-cause mortality over the pandemic period, more than doubling it from approximately 640,000 cumulative deaths (34) to 1.4 million. To us, it is an amazing coincidence that all the provincial pandemic responses, which were applied at different times in different sequences (35), could have produced such a large effect that all-cause mortality was reduced to nearly what can be predicted historically—before March 2020—when there was no pandemic and no public health measures, whatsoever [see Figure 5 in Rancourt et al. (36)].
Table 1. Per capita Canadian deaths (per 100,000 people) attributed to World War I, II, and the 1918 influenza pandemic compared to the counterfactual modelling scenario “no public health measures or vaccination” of Ogden et al. (5).
Final comments
It is important that we try to assess the effectiveness and the costs of pandemic interventions dispassionately. The justifications and conclusions in the article by Ogden et al. (5) which are derived from a single model, with no sensitivity analyses around key assumptions, leads us to wonder: what was its point? Was it to showcase an evidence-based analysis of actual policies and their alternatives? Or was it an attempt to justify government action, despite other evidence of their limited benefit (37)?
As academic exercises, counterfactual analyses and what could have happened can possess epistemic benefits. These modes of thought and speech have been the subject of study in philosophy, politics, and history (38), as well as in medical decision making (39). In the analysis by Ogden et al. (5) counterfactuals represent an attractive means for “re-running” history. However, they represent a great disservice to public health in the way they have been used by Canadian public health officials. Just as counterfactual scenarios were used to justify “doing something” during the pandemic (e.g., (40)), the historical revisions of Ogden et al. (5) are used to vindicate that having “done something” was the right course of action. However, unlike modelling the future—which is testable—there is no way to demonstrate whether counterfactual scenarios are right or wrong about “what could have happened.” The historical path, the one involving no interventions, was foreclosed the moment the pandemic responses began (29). Neither Ogden et al. (5) nor anyone else, can ever observe the simultaneous response and non-response of Canada's experience with SARS-CoV-2. Instead, all that remains are “what-if?” statements and modelling output akin to the simulated worlds found in online games; they might appear convincing, but their foundations are imaginary, and their walls are pixel thin (41).
That said, the article by Ogden et al. (5) has two redeeming features. The authors admit that: one, Canada's response to the pandemic was not perfect; and two, that the unintended consequences of the public health measures need to be investigated. It will be a measure of the honesty, courage, and integrity of the Public Health Agency of Canada, and their Provincial partners, if the latter is ever realised.
Author contributions
DV, SE, JH, CC, and SP contributed to the conception, writing, and editing of this opinion article. All authors contributed to the article and approved the submitted version.
Conflict of interest
SP is a Founder and Chief Scientific Officer at Kinexus Bioinformatics Corporation.
The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1. Smith SM, Sonego S, Wallen GR, Waterer G, Cheng AC, Thompson P, et al. Use of non-pharmaceutical interventions to reduce the transmission of influenza in adults: a systematic review. Respirology. (2015) 20:896–903. doi: 10.1111/resp.12541
2. Jefferson T, Dooley L, Ferroni E, Al-Ansary LA, van Driel ML, Bawazeer GA, et al. Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database Syst Rev. (2020) 11:CD006207. doi: 10.1002/14651858.CD006207.pub5
3. Grant JM, Fulford M, Schabas R. Circular logic and flawed modelling compromises non- pharmaceutical intervention article's conclusions. Can Commun Dis Rep. (2022) 48:492–5. Available online at: https://www.canada.ca/en/public-health/services/reports-publications/canada-communicable-disease-report-ccdr/monthly-issue/2022-48/issue-10-october-2022/letter-to-editor-response.html (accessed April 5, 2023).
4. World Health Organization. Non-Pharmaceutical Public Health Measures for Mitigating the Risk and Impact of Epidemic and Pandemic Influenza. Geneva: WHO (2019).
5. Ogden NH, Turgeon P, Fazil A, Clark J, Gabriele-Rivet V, Tam T, et al. Counterfactuals of effects of vaccination and public health measures on COVID-19 cases in Canada: what could have happened? Can Commun Dis Rep. (2022) 48:292–302. doi: 10.14745/ccdr.v48i78a01
6. Koivisto M. Pitfalls in Modelling and simulation. Procedia Comp Sci. (2017) 119:8–15. doi: 10.1016/j.procs.2017.11.154
7. Ng KW, Faulkner N, Cornish GH, Rosa A, Harvey R, Hussain S, et al. Preexisting and de novo humoral immunity to SARS-CoV-2 in humans. Science. (2020) 370:1339–43. doi: 10.1126/science.abe1107
8. Kundu R, Narean JS, Wang L, Fenn J, Pillay T, Fernandez ND, et al. Cross-reactive memory T cells associate with protection against SARS-CoV-2 infection in COVID-19 contacts. Nat Commun. (2022) 13:80. doi: 10.1038/s41467-021-27674-x
9. Mateus J, Grifoni A, Tarke A, Sidney J, Ramirez SI, Dan JM, et al. Selective and cross-reactive SARS-CoV-2 T cell epitopes in unexposed humans. Science. (2020) 370:89–94. doi: 10.1126/science.abd3871
10. Guthmiller JJ, Wilson PC. Remembering seasonal coronaviruses. Science. (2020) 370:1272–3. doi: 10.1126/science.abf4860
11. Yu X, Tsibane T, McGraw PA, House FS, Keefer CJ, Hicar MD, et al. Neutralizing antibodies derived from the B cells of 1918 influenza pandemic survivors. Nature. (2008) 455:532–6. doi: 10.1038/nature07231
12. Ogden NH, Fazil A, Arino J, Berthiaume P, Fisman DN, Greer AL, et al. Modelling scenarios of the epidemic of COVID-19 in Canada. Can Commun Dis Rep. (2020) 46:198–204. doi: 10.14745/ccdr.v46i06a08
13. Ioannidis JPA. Infection fatality rate of COVID-19 inferred from seroprevalence data. Bull World Health Organ. (2021) 99:19–33F. doi: 10.2471/BLT.20.265892
14. Statistics Canada. Population Estimates on July 1 By Age Sex. (2023). Available online at: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1710000501 (accessed February 24, 2023).
15. COVID-19 Forecasting Team. Variation in the COVID-19 infection–fatality ratio by age, time, and geography during the pre-vaccine era: a systematic analysis. Lancet. (2022) 399:1469–88. doi: 10.1016/S0140-6736(21)02867-1
16. Kory P, Meduri GU, Varon J, Iglesias J, Marik PE. Review of the emerging evidence demonstrating the efficacy of Ivermectin in the prophylaxis and treatment of COVID-19. Am J Therapeutics. (2021) 28:e299–318. doi: 10.1097/MJT.0000000000001377
17. Alexander PE, Armstrong R, Fareed G, Lotus J, Oskoui R, Prodromos C, et al. Early multidrug treatment of SARS-CoV-2 infection (COVID-19) and reduced mortality among nursing home (or outpatient/ambulatory) residents. Med Hypoth. (2021) 153:110622. doi: 10.1016/j.mehy.2021.110622
18. Risch HA. Early outpatient treatment of symptomatic, high-risk COVID-19 patients that should be ramped up immediately as key to the pandemic crisis. Am J Epidemiol. (2020) 189:1218–26. doi: 10.1093/aje/kwaa093
19. McCullough PA, Kelly RJ, Ruocco G, McCullough PA, Kelly RJ, Ruocco G, Lerma E, Tumlin J, Wheelan KR, et al. Pathophysiological basis and rationale for early outpatient treatment of SARS-CoV-2 (COVID-19) infection. Am J Med. (2021) 134:16–22. doi: 10.1016/j.amjmed.2020.07.003
20. Carvallo H, Hirsch R, Alkis P, Contreras V. Study of the efficacy and safety of topical Ivermectin + Iota-Carrageenan in the prophylaxis against COVID-19 in health personnel. J Biomed Res Clin Investig. (2020) 2:1007. doi: 10.31546/2633-8653.1007
21. Newey S. Why Have So Many Coronavirus Patients Died in Italy? The Telegraph. (2020) Available online at: https://www.telegraph.co.uk/global-health/science-and-disease/have-many-coronavirus-patients-died-italy/ (accessed March 24, 2023).
22. CBC News. Ontario Removes More Than 400 Deaths From Official COVID-19 Count. (2023). Available online at: https://www.cbc.ca/news/canada/toronto/covid19-ontario-march-11-1.6381330 (accessed March 25, 2023).
23. Columbia University Public Health. Agent-Based Modeling. Website: Population Health Methods. (2020) Available online at: https://www.publichealth.columbia.edu/research/population-health-methods/agent-based-modeling (accessed March 25, 2023).
24. Rahmandad H, Sterman J. Heterogeneity and network structure in the dynamics of diffusion: comparing agent-based and differential equation models. Manag Sci. (2008) 54:998–1014. doi: 10.1287/mnsc.1070.0787
25. Goolsbee A, Syverson C. Fear, lockdown, and diversion: comparing drivers of pandemic economic decline. J Public Econ. (2021) 193:104311. doi: 10.1016/j.jpubeco.2020.104311
26. Berry CR, Fowler A, Glazer T, MacMillen A. Evaluating the effects of shelter-in-place policies during the COVID-19 pandemic. PNAS. (2021) 118:e2019706118. doi: 10.1073/pnas.2019706118
27. Public Health Agency of Canada. COVID-19 Epidemiology Update: Current Situation. (2023). Available online at: https://health-infobase.canada.ca/covid-19/current-situation.html (accessed April 5, 2023).
28. Sterman JD. Appropriate summary statistics for evaluating the historical fit of system dynamics models. Dynamica. (1984) 10:51–66.
29. DeMartino GF. The specter of irreparable ignorance: counterfactuals and causality in economics. Rev Evol Polit Econ. (2021) 2:253–76. doi: 10.1007/s43253-020-00029-w
30. King G, Zeng L. The dangers of extreme counterfactuals. Polit Anal. (2006) 14:131–59. doi: 10.1093/pan/mpj004
31. Statistics Canada. Estimated Population of Canada, 1605 to 2007. (2023). Available online at: https://www150.statcan.gc.ca/n1/pub/98-187-x/4151287-eng.htm (accessed March 25, 2023).
32. Parks Canada. The Spanish Flu in Canada (1918-1920) National Historic Event. (2023). Available online at: https://parks.canada.ca/culture/designation/evenement-event/grippe-espagnole-spanish-flu (accessed March 25, 2023).
33. Library Archives Canada. Service files of the Second World War - War dead, 1939-1947. (2023). Available online at: https://www.bac-lac.gc.ca/eng/discover/military-heritage/second-world-war/second-world-war-dead-1939-1947/Pages/files-second-war-dead.aspx (accessed March 25, 2023).
34. Statistics Canada. Provisional Weekly Estimates of the Number of Deaths, Expected Number of Deaths and Excess Mortality. Ottowa, ON: Statistics Canada (2023).
35. Cameron-Blake E, Breton, C, Sim, P, Tatlow, H, Hale, T, Wood, A,. Variation in the Canadian Provincial Territorial Responses to COVID- 19. Blavatnik School of Government Working Paper 2021. (2023). Available online at: https://www.bsg.ox.ac.uk/research/publications/variation-canadian-provincial-and-territorial-responses-covid-19 (accessed April 5, 2023).
36. Rancourt D, Baudin, M, Mercier, J. Report: Proof that Canada's COVID-19 Mortality Statistics are Incorrect. (2023). Available online at: https://correlation-canada.org/report-proof-that-canadas-covid-19-mortality-statistics-are-incorrect/ (accessed March 25, 2023).
37. Vickers DM, Baral S, Mishra S, Kwong JC, Sundaram M, Katz A, et al. Stringency of containment and closures on the growth of SARS-CoV-2 in Canada prior to accelerated vaccine roll-out. Int J Infect Dis. (2022) 118:73–82. doi: 10.1016/j.ijid.2022.02.030
38. Tetlock PE, Belkin A. Counterfactual Thought Experiments in World Politics: Logical, Methodological, and Psychological Perspectives. Princeton: Princeton University Press. (1996).
39. Petrocelli JV. Pitfalls of counterfactual thinking in medical practice: preventing errors by using more functional reference points. J Public Health Res. (2013) 2:e24. doi: 10.4081/jphr.2013.e24
40. Ferguson NM, Laydon D, Nedjati-Gilani G, Imai N, Ainslie K, Baguelin M. Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand. Imp Res COVID- Team. (2020) 20:77482. doi: 10.25561/77482
41. The Daily Sceptic. If You Think Modelling the Future is Dangerous, Try Modelling the Past. (2023). Available online at: https://dailysceptic.org/2022/12/23/if-you-think-modelling-the-future-is-dangerous-try-modelling-the-past/ (accessed February 22, 2023).
Keywords: COVID−19, non-pharmaceutical interventions, vaccination, evaluation, simulation modelling
Citation: Vickers DM, Hardie J, Eberspaecher S, Chaufan C and Pelech S (2023) Counterfactuals of effects of vaccination and public health measures on COVID-19 cases in Canada: what could have happened? Front. Public Health 11:1173673. doi: 10.3389/fpubh.2023.1173673
Received: 24 February 2023; Accepted: 30 March 2023;
Published: 09 May 2023.
Edited by:
Creuza Rachel Vicente, Federal University of Espirito Santo, BrazilReviewed by:
Awad A. Shehata, University of Sadat City, EgyptCopyright © 2023 Vickers, Hardie, Eberspaecher, Chaufan and Pelech. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: David M. Vickers, david.vickers2@ucalgary.ca