Skip to main content

OPINION article

Front. Public Health, 09 May 2023
Sec. Infectious Diseases: Epidemiology and Prevention
This article is part of the Research Topic Impact of public health and social measures for COVID-19 control on infectious disease epidemiology View all 13 articles

Counterfactuals of effects of vaccination and public health measures on COVID-19 cases in Canada: what could have happened?

\r\nDavid M. Vickers
David M. Vickers1*John HardieJohn Hardie2Stefan EberspaecherStefan Eberspaecher3Claudia ChaufanClaudia Chaufan4Steven Pelech,Steven Pelech5,6
  • 1Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
  • 2Retired, Ottawa, ON, Canada
  • 3Private Practitioner, Ottawa, ON, Canada
  • 4Health Policy and Global Health, Faculty of Health, York University, Toronto, ON, Canada
  • 5Department of Medicine, University of British Columbia, Vancouver, BC, Canada
  • 6Kinexus Bioinformatics Corporation, Vancouver, BC, Canada


Canada's early response to COVID-19 primarily employed non-pharmaceutical interventions (NPIs)—school and business closures, physical distancing measures, stay at home orders, and mandatory public masking—despite little evidence supporting them (1, 2). These responses were not part of existing pandemic plans (3). Indeed, they ignored the basic principles of pandemic planning, which is to minimise serious illness and deaths, as well as limit societal disruptions (4). Canada and the world need detailed analyses of the effectiveness and the costs of the NPIs used to try to control COVID-19. Such analyses must be separated from politics and evidentially based on comprehensive data sets.

Unfortunately, a recent article entitled “Counterfactuals of effects of vaccination and public health measures on COVID-19 cases in Canada: What could have happened?” published in the Canadian Communicable Disease Report by Ogden et al. (5) contains questionable results that sully the evaluation of these important issues. Instead of performing a much-needed analysis of real-world data, Ogden et al. (5) focus on simulation to postulate how the COVID-19 pandemic would have affected Canada had certain public health measures not been implemented. Phrases like “This study illustrates what may have happened…”, and “Canada could have experienced…” are repeatedly expressed by the authors.

While dynamical models can be useful policy simulators, the insights derived from them can all-too-often become the codified opinions of the modeller (6). In this case, the paper by Ogden et al. (5) is less likely an objective lens into the past than a loss of critical distance from an overly complicated model. In this commentary, we discuss selected authors' assumptions and conclude that they are problematic at-best. We first identify empirically antiquated and conceptually ambiguous justifications supporting the article; second, we critique the model used to substantiate the article's conclusions; and finally, we elaborate on the implications of the authors' position for effective public health policy and human wellbeing in times of crisis.

The justifications

In the “Introduction”, the authors assert that at the start of the pandemic, humans had no known immunity to SARS-CoV-2 (5). However, if humans were truly immunologically naïve to SARS-CoV-2, then it becomes difficult to explain-away the existence of cross-reactive neutralising IgG antibody (7) or memory T-cells (8, 9) to conserved epitopes between endemic coronavirus and SARS-CoV-2 proteins. These latter T cells will activate, expand, and produce IL-2 and IFN-γ when stimulated (in vitro) with cognate antigen (8), which very possibly represents a larger central memory response capable of impacting COVID-19 severity (10). Thus, at the very least, exposure to other human coronaviruses, prior to March 2020, appears able to confer measurable levels of immunity that was not considered—even in sensitivity analyses—by Ogden et al. (5) Similar immune responses to the 1918 strain of influenza existed 90 years later (11), and SARS-CoV-2 is unlikely to be the exception.

The authors also state that, unless the public health measures had been adopted, the consequences for Canadians and for Canada's public health system would have been “dire” (5). This claim, which is predicated on other modelling output (12), assumed that because of no effective therapies, Canada experienced an infection fatality rate (IFR) “approaching” 1%—contrasting it with a much lower IFR of 0.4% for seasonal influenza. While the supporting references used by Ogden et al. (5) estimate the case fatality rate (see their main text references 10, 11, 35), other empirical evidence indicates that the IFR for COVID-19 could be much lower than 1%. For instance, Ioannidis (13) determined that the IFR was 0.15–0.20% globally, and 0.03–0.04% for those under 70 years of age. Canadian-specific estimates reported by Ioannidis were 0.59% (overall) and 0.08% (for people aged < 70 years). People aged >70 years only account for 12.9% of the population in Canada (14). Similarly, results from the COVID-19 Forecasting Team (15) were compatible with unadjusted global IFRs between 0.0054% and 0.43% for people aged < 50 years, while Canadian-specific, age-adjusted cumulative IFRs decreased from 0.54% (in April 2020) and 0.35% (in January 2021). As for the alleged lack of effective therapies, early in 2020 the international literature indicated that early outpatient treatment with multidrug therapy successfully reduced hospitalisation and death, even among populations considered at high risk because of occupation, underlying health conditions, or age (1620).

The article's “Chronology of the Epidemic and Public Health Measures” justifies the adoption of public health measures based on the “unrestrained SARS-CoV-2 transmission in Italy.” This implies the rampant spread of the infection throughout the entire population. However, Professor Walter Ricciardi, Italy's scientific advisor, corrected this assumption when he noted that “The median age of people infected with SARS-CoV-2 who [were] dying in Italy has been 80 years, and the average age of patients requiring critical care support [was] 67 years” (21). Ricciardi's point on risk-stratification provides an entirely different perspective on the epidemic's characteristics, and if true, nullifies a key justification for the author's model—a virus that if “unrestrained” would be an equal opportunity killer. As well, as shown in their Table 2, Ogden et al. (5) assume that the listed countries have agreed on what constitutes a COVID-19 death. However, they overlook that until recently, a COVID-19 death was anyone dying with, not necessarily from COVID-19 (this was corrected in Ontario March 11, 2022) (22). They also overlook the substantial differences, worldwide, in definitions of COVID-19 deaths (e.g., China likely counts them very differently than Canada).

Model structure

Ogden et al. (5) constructed an agent-based model (ABM) for understanding any benefits of Canada's pandemic response. ABMs are computational models for simulating the actions and interactions of autonomous agents (such as people) to understand the behaviour of a system and the processes that govern any outcomes of interest (23). Because ABMs maintain distinct information on every individual in a simulated population, their finer-grained nature allows them to represent certain types of activities, relationships, and interventions with greater precision and flexibility than with more traditional differential equation models used to study pathogen transmission (23, 24). However, the granularity of ABMs carries with it an obligation to understand the minutiae that links a model's structure with its behaviour (24). Ogden et al. (5) have circumvented this necessity by filling-in details of people's movements that are exogenous to the structure of the model, rather than being driven by factors or feedbacks that are internal to its current state. This expanded effort in deepening their model has inhibited a critical broadening of the model boundary that will surely limit its validity (6, 24).

For example, the behaviour of the model's agents has an all-or-none relationship with public health measures: when in-place, people changed their behaviour proportionately to the value Oxford Stringency Index; when not in-place, agents maintain their status quo. This forced dependence on government orders is contrary to other research demonstrating that people's inherent fear of infection, alone, will cause them to alter their behaviour rapidly (25), and often before strict rules are in-place (26).

One consequence of the exogeneous drivers of people's behaviour is that two model-derived metrics have diverged substantially from publicly available data. Cumulative case and death rates reported by April 2022 were approximately 9,000 cases and 98 deaths per 100,000 people, respectively (27). When compared to the estimated values in the “observed baseline” scenario in their Table 3, both these empirical values lie outside the 95th percentiles of their modelled counterparts. This directional bias in model output is indicative of structural errors (28) that could distort comparisons between the calibrated (baseline) model and its counterfactual scenarios.

Over-reliance on counterfactual scenarios

Like all simulation modelling, ABM approaches require that we depart from real-world environments (e.g., workplaces and schools) and create idealised agents and environmental conditions (29, 30). When Ogden et al. discuss Canada's experience with COVID-19 without restrictive measures or vaccination, they failed to note at least two problems. First, “up to 800,000” COVID-19 deaths are greater than any other historic event in Canada over the last 108 years, corresponding to higher per capita death rates than for all Canadian lives lost in World War I, the 1918 influenza pandemic, and World War II (see Table 1) (3133). Second, the addition of 800,000 deaths would have large consequences for all-cause mortality over the pandemic period, more than doubling it from approximately 640,000 cumulative deaths (34) to 1.4 million. To us, it is an amazing coincidence that all the provincial pandemic responses, which were applied at different times in different sequences (35), could have produced such a large effect that all-cause mortality was reduced to nearly what can be predicted historically—before March 2020—when there was no pandemic and no public health measures, whatsoever [see Figure 5 in Rancourt et al. (36)].


Table 1. Per capita Canadian deaths (per 100,000 people) attributed to World War I, II, and the 1918 influenza pandemic compared to the counterfactual modelling scenario “no public health measures or vaccination” of Ogden et al. (5).

Final comments

It is important that we try to assess the effectiveness and the costs of pandemic interventions dispassionately. The justifications and conclusions in the article by Ogden et al. (5) which are derived from a single model, with no sensitivity analyses around key assumptions, leads us to wonder: what was its point? Was it to showcase an evidence-based analysis of actual policies and their alternatives? Or was it an attempt to justify government action, despite other evidence of their limited benefit (37)?

As academic exercises, counterfactual analyses and what could have happened can possess epistemic benefits. These modes of thought and speech have been the subject of study in philosophy, politics, and history (38), as well as in medical decision making (39). In the analysis by Ogden et al. (5) counterfactuals represent an attractive means for “re-running” history. However, they represent a great disservice to public health in the way they have been used by Canadian public health officials. Just as counterfactual scenarios were used to justify “doing something” during the pandemic (e.g., (40)), the historical revisions of Ogden et al. (5) are used to vindicate that having “done something” was the right course of action. However, unlike modelling the future—which is testable—there is no way to demonstrate whether counterfactual scenarios are right or wrong about “what could have happened.” The historical path, the one involving no interventions, was foreclosed the moment the pandemic responses began (29). Neither Ogden et al. (5) nor anyone else, can ever observe the simultaneous response and non-response of Canada's experience with SARS-CoV-2. Instead, all that remains are “what-if?” statements and modelling output akin to the simulated worlds found in online games; they might appear convincing, but their foundations are imaginary, and their walls are pixel thin (41).

That said, the article by Ogden et al. (5) has two redeeming features. The authors admit that: one, Canada's response to the pandemic was not perfect; and two, that the unintended consequences of the public health measures need to be investigated. It will be a measure of the honesty, courage, and integrity of the Public Health Agency of Canada, and their Provincial partners, if the latter is ever realised.

Author contributions

DV, SE, JH, CC, and SP contributed to the conception, writing, and editing of this opinion article. All authors contributed to the article and approved the submitted version.

Conflict of interest

SP is a Founder and Chief Scientific Officer at Kinexus Bioinformatics Corporation.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


1. Smith SM, Sonego S, Wallen GR, Waterer G, Cheng AC, Thompson P, et al. Use of non-pharmaceutical interventions to reduce the transmission of influenza in adults: a systematic review. Respirology. (2015) 20:896–903. doi: 10.1111/resp.12541

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Jefferson T, Dooley L, Ferroni E, Al-Ansary LA, van Driel ML, Bawazeer GA, et al. Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database Syst Rev. (2020) 11:CD006207. doi: 10.1002/14651858.CD006207.pub5

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Grant JM, Fulford M, Schabas R. Circular logic and flawed modelling compromises non- pharmaceutical intervention article's conclusions. Can Commun Dis Rep. (2022) 48:492–5. Available online at: (accessed April 5, 2023).

Google Scholar

4. World Health Organization. Non-Pharmaceutical Public Health Measures for Mitigating the Risk and Impact of Epidemic and Pandemic Influenza. Geneva: WHO (2019).

Google Scholar

5. Ogden NH, Turgeon P, Fazil A, Clark J, Gabriele-Rivet V, Tam T, et al. Counterfactuals of effects of vaccination and public health measures on COVID-19 cases in Canada: what could have happened? Can Commun Dis Rep. (2022) 48:292–302. doi: 10.14745/ccdr.v48i78a01

CrossRef Full Text | Google Scholar

6. Koivisto M. Pitfalls in Modelling and simulation. Procedia Comp Sci. (2017) 119:8–15. doi: 10.1016/j.procs.2017.11.154

CrossRef Full Text | Google Scholar

7. Ng KW, Faulkner N, Cornish GH, Rosa A, Harvey R, Hussain S, et al. Preexisting and de novo humoral immunity to SARS-CoV-2 in humans. Science. (2020) 370:1339–43. doi: 10.1126/science.abe1107

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Kundu R, Narean JS, Wang L, Fenn J, Pillay T, Fernandez ND, et al. Cross-reactive memory T cells associate with protection against SARS-CoV-2 infection in COVID-19 contacts. Nat Commun. (2022) 13:80. doi: 10.1038/s41467-021-27674-x

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Mateus J, Grifoni A, Tarke A, Sidney J, Ramirez SI, Dan JM, et al. Selective and cross-reactive SARS-CoV-2 T cell epitopes in unexposed humans. Science. (2020) 370:89–94. doi: 10.1126/science.abd3871

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Guthmiller JJ, Wilson PC. Remembering seasonal coronaviruses. Science. (2020) 370:1272–3. doi: 10.1126/science.abf4860

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Yu X, Tsibane T, McGraw PA, House FS, Keefer CJ, Hicar MD, et al. Neutralizing antibodies derived from the B cells of 1918 influenza pandemic survivors. Nature. (2008) 455:532–6. doi: 10.1038/nature07231

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Ogden NH, Fazil A, Arino J, Berthiaume P, Fisman DN, Greer AL, et al. Modelling scenarios of the epidemic of COVID-19 in Canada. Can Commun Dis Rep. (2020) 46:198–204. doi: 10.14745/ccdr.v46i06a08

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Ioannidis JPA. Infection fatality rate of COVID-19 inferred from seroprevalence data. Bull World Health Organ. (2021) 99:19–33F. doi: 10.2471/BLT.20.265892

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Statistics Canada. Population Estimates on July 1 By Age Sex. (2023). Available online at: (accessed February 24, 2023).

15. COVID-19 Forecasting Team. Variation in the COVID-19 infection–fatality ratio by age, time, and geography during the pre-vaccine era: a systematic analysis. Lancet. (2022) 399:1469–88. doi: 10.1016/S0140-6736(21)02867-1

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Kory P, Meduri GU, Varon J, Iglesias J, Marik PE. Review of the emerging evidence demonstrating the efficacy of Ivermectin in the prophylaxis and treatment of COVID-19. Am J Therapeutics. (2021) 28:e299–318. doi: 10.1097/MJT.0000000000001377

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Alexander PE, Armstrong R, Fareed G, Lotus J, Oskoui R, Prodromos C, et al. Early multidrug treatment of SARS-CoV-2 infection (COVID-19) and reduced mortality among nursing home (or outpatient/ambulatory) residents. Med Hypoth. (2021) 153:110622. doi: 10.1016/j.mehy.2021.110622

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Risch HA. Early outpatient treatment of symptomatic, high-risk COVID-19 patients that should be ramped up immediately as key to the pandemic crisis. Am J Epidemiol. (2020) 189:1218–26. doi: 10.1093/aje/kwaa093

PubMed Abstract | CrossRef Full Text | Google Scholar

19. McCullough PA, Kelly RJ, Ruocco G, McCullough PA, Kelly RJ, Ruocco G, Lerma E, Tumlin J, Wheelan KR, et al. Pathophysiological basis and rationale for early outpatient treatment of SARS-CoV-2 (COVID-19) infection. Am J Med. (2021) 134:16–22. doi: 10.1016/j.amjmed.2020.07.003

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Carvallo H, Hirsch R, Alkis P, Contreras V. Study of the efficacy and safety of topical Ivermectin + Iota-Carrageenan in the prophylaxis against COVID-19 in health personnel. J Biomed Res Clin Investig. (2020) 2:1007. doi: 10.31546/2633-8653.1007

CrossRef Full Text | Google Scholar

21. Newey S. Why Have So Many Coronavirus Patients Died in Italy? The Telegraph. (2020) Available online at: (accessed March 24, 2023).

Google Scholar

22. CBC News. Ontario Removes More Than 400 Deaths From Official COVID-19 Count. (2023). Available online at: (accessed March 25, 2023).

23. Columbia University Public Health. Agent-Based Modeling. Website: Population Health Methods. (2020) Available online at: (accessed March 25, 2023).

24. Rahmandad H, Sterman J. Heterogeneity and network structure in the dynamics of diffusion: comparing agent-based and differential equation models. Manag Sci. (2008) 54:998–1014. doi: 10.1287/mnsc.1070.0787

CrossRef Full Text | Google Scholar

25. Goolsbee A, Syverson C. Fear, lockdown, and diversion: comparing drivers of pandemic economic decline. J Public Econ. (2021) 193:104311. doi: 10.1016/j.jpubeco.2020.104311

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Berry CR, Fowler A, Glazer T, MacMillen A. Evaluating the effects of shelter-in-place policies during the COVID-19 pandemic. PNAS. (2021) 118:e2019706118. doi: 10.1073/pnas.2019706118

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Public Health Agency of Canada. COVID-19 Epidemiology Update: Current Situation. (2023). Available online at: (accessed April 5, 2023).

28. Sterman JD. Appropriate summary statistics for evaluating the historical fit of system dynamics models. Dynamica. (1984) 10:51–66.

Google Scholar

29. DeMartino GF. The specter of irreparable ignorance: counterfactuals and causality in economics. Rev Evol Polit Econ. (2021) 2:253–76. doi: 10.1007/s43253-020-00029-w

CrossRef Full Text | Google Scholar

30. King G, Zeng L. The dangers of extreme counterfactuals. Polit Anal. (2006) 14:131–59. doi: 10.1093/pan/mpj004

CrossRef Full Text | Google Scholar

31. Statistics Canada. Estimated Population of Canada, 1605 to 2007. (2023). Available online at: (accessed March 25, 2023).

32. Parks Canada. The Spanish Flu in Canada (1918-1920) National Historic Event. (2023). Available online at: (accessed March 25, 2023).

Google Scholar

33. Library Archives Canada. Service files of the Second World War - War dead, 1939-1947. (2023). Available online at: (accessed March 25, 2023).

34. Statistics Canada. Provisional Weekly Estimates of the Number of Deaths, Expected Number of Deaths and Excess Mortality. Ottowa, ON: Statistics Canada (2023).

PubMed Abstract | Google Scholar

35. Cameron-Blake E, Breton, C, Sim, P, Tatlow, H, Hale, T, Wood, A,. Variation in the Canadian Provincial Territorial Responses to COVID- 19. Blavatnik School of Government Working Paper 2021. (2023). Available online at: (accessed April 5, 2023).

Google Scholar

36. Rancourt D, Baudin, M, Mercier, J. Report: Proof that Canada's COVID-19 Mortality Statistics are Incorrect. (2023). Available online at: (accessed March 25, 2023).

37. Vickers DM, Baral S, Mishra S, Kwong JC, Sundaram M, Katz A, et al. Stringency of containment and closures on the growth of SARS-CoV-2 in Canada prior to accelerated vaccine roll-out. Int J Infect Dis. (2022) 118:73–82. doi: 10.1016/j.ijid.2022.02.030

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Tetlock PE, Belkin A. Counterfactual Thought Experiments in World Politics: Logical, Methodological, and Psychological Perspectives. Princeton: Princeton University Press. (1996).

Google Scholar

39. Petrocelli JV. Pitfalls of counterfactual thinking in medical practice: preventing errors by using more functional reference points. J Public Health Res. (2013) 2:e24. doi: 10.4081/jphr.2013.e24

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Ferguson NM, Laydon D, Nedjati-Gilani G, Imai N, Ainslie K, Baguelin M. Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand. Imp Res COVID- Team. (2020) 20:77482. doi: 10.25561/77482

PubMed Abstract | CrossRef Full Text | Google Scholar

41. The Daily Sceptic. If You Think Modelling the Future is Dangerous, Try Modelling the Past. (2023). Available online at: (accessed February 22, 2023).

Keywords: COVID−19, non-pharmaceutical interventions, vaccination, evaluation, simulation modelling

Citation: Vickers DM, Hardie J, Eberspaecher S, Chaufan C and Pelech S (2023) Counterfactuals of effects of vaccination and public health measures on COVID-19 cases in Canada: what could have happened? Front. Public Health 11:1173673. doi: 10.3389/fpubh.2023.1173673

Received: 24 February 2023; Accepted: 30 March 2023;
Published: 09 May 2023.

Edited by:

Creuza Rachel Vicente, Federal University of Espirito Santo, Brazil

Reviewed by:

Awad A. Shehata, University of Sadat City, Egypt

Copyright © 2023 Vickers, Hardie, Eberspaecher, Chaufan and Pelech. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: David M. Vickers,

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.