Starting from the first reports coming from China and from countries in Europe and Asia, about the infection produced by SARS-CoV-2, its high contagion, and lethality of up to 14% in older adults, and the subsequent declaration of a COVID-19 pandemic, and together with the measures established by the healthcare authorities to manage the disease, efforts began to develop effective and safe vaccines that would contribute to speeding up the control of this health condition, through the reduction of infections, complications, and deaths associated with this disease (1).

For this reason, pandemic control efforts have focused on developing vaccines against SARS-CoV-2 that are capable of acting against infection, disease, or transmission, and thus contribute to disease control (2). In this context, different research groups have developed vaccines using different platforms, including mRNA, viral vectors, and inactivated viruses (3).

Unlike most drugs, whose benefits are limited to the individual taking them, vaccines have the potential to produce far-reaching effects on general public health and well-being, cognitive development, and, ultimately, economic productivity (4). However, the global advances in vaccination coverage achieved during the first years of the 21st century have been threatened by the emergence of anti-vaccination groups that have questioned vaccine efficacy to create public distrust of vaccines and immunization programs. This requires an adequate and conscious evaluation of both the efficacy/effectiveness and the different aspects that can affect the safety of the people who receive them (5).

In general, vaccines that have gained approval for human use have been effective in preventing COVID-19, particularly in preventing severe disease and death. However, reports on their implementation are mainly based on follow-up studies of the adult population (6). Additionally, if the vaccination prevents symptoms from developing and asymptomatic infections are less likely to be discovered than symptomatic ones, it is feasible that the effectiveness against any infection has been overstated. A competitive tendency toward underestimate arises when estimates are based on tests with inadequate specificity, particularly when testing are conducted more frequently than has been estimated for various COVID-19 vaccinations (7).

All vaccines seem to be safe and efficacious against all variations of interest in preventing hospitalization, death, and severe COVID-19; however, the quality of the data differs significantly between the vaccines under consideration (8).

Different methodological designs have been used to evaluate the effectiveness and safety of vaccines for COVID-19. Most clinical trials were carried out before the appearance of variants of concern, and the duration, subgroups evaluated, and analysis methods were not homogeneous between vaccines, creating uncertainty about some effects and comparisons (9).

CoronaVac is an inactivated whole-virus vaccine against COVID-19 adjuvanted with aluminum hydroxide created from African green monkey kidney cells (Vero cells) inoculated with SARS-CoV-2 (strain CN02). The Chinese company Sinovac Biotech developed the vaccine, and on June 1, 2021, the World Health Organization (WHO) approved the vaccine for emergency use (10). Using two 3 μg doses of CoronaVac, the overall efficacy for avoiding symptomatic COVID-19 (before the emergence of concerning variations) has been assessed at 67.7% (95%CI: 35.9 to 83.7%) (10). Compared to COVID-19 prevention, its impact in preventing hospital stays, ICU admissions, and fatalities has been much stronger. Three-dose regimens have also been shown to raise seroconversion levels of neutralizing antibodies, even against variants like Omicron. Few serious vaccine-related adverse reactions have been reported (10).

However, given the differences that may exist in the methods used to assess the efficacy, effectiveness, and safety of vaccines against COVID-19, our objective was to explore the published research on COVID-19 vaccines, focusing on CoronaVac, in order to answer the question: How has the efficacy/effectiveness and safety of CoronaVac been assessed in different designs and study phases of the vaccines used to control COVID-19?

A scoping review was carried out under a protocol registered in the Open Science Framework (OSF; osf.io/aeut4), and following the methodology proposed by the Joanna Briggs Institute (11), which included studies carried out in humans as of 2020, corresponding to systematic reviews, clinical trials, and analytical or descriptive observational studies in which the effectiveness and/or safety of vaccines for COVID19 were evaluated or described. There were no age restrictions on the study participants.

Abstracts from congresses were not evaluated because they had not been subjected to systematic peer evaluation at the time, nor were studies published in languages other than English or Spanish.

The COVID-19 pandemic has affected the world’s population with a high morbidity and mortality rate. Recent reports have described persistent symptoms that extend beyond the initial period of the disease. It has been observed that adverse consequences, in addition to respiratory effects, are produced at different levels: cardiovascular, neurological, or immunological; cutaneous, gastrointestinal, or kidney manifestations, as well as in mental health, both as a result of acute infection and by the so-called post-COVID-19 syndrome (29). In this context, developing effective and safe vaccines was the determining control measure for pandemic management since, in addition to reducing the transmission of infections and allowing the control of the disease, vaccines had a determining role in reducing severe and fatal complications associated with infection (30). In addition to the above, the time in which the vaccine candidates were available, where it took less than a year for developers to complete the design, manufacturing, efficacy and safety testing and evaluation and approval for use, is an immeasurable scientific and public health learning, as well as an example of cooperation between healthcare authorities, the scientific community and private sector (31).

This review presents an analysis of the methods, populations, and scope of the studies that have evaluated the efficacy/effectiveness and safety of the vaccines available for COVID-19, emphasizing CoronaVac. Differences were found in terms of the proportion of populations evaluated, follow-up times, and times of the studies regarding the appearance of variants of concern.

Although some clinical trials with random assignment have been carried out to assess efficiency and safety outcomes with CoronaVac, these have limitations in terms of feasibility, follow-up times, and with this, the possibility of evaluating safety outcomes that occur with low frequencies (32). In this sense, it is important to carry out observational data analysis. However, not all studies have used homogeneous methods of analysis. Both the prevention of infection, and the prevention of outcomes such as hospitalization or death, have been valued through similar outcomes, but some through multivariate analysis of dependencies, and others through analysis that try to infer causally through different control methods of confounding. Studies have compared the evaluation of the same outcome through different methods, including multivariable logistic regression, propensity matching, propensity adjustment, and propensity-base weighting. However, researchers described that the estimates are very sensitive to the explicit or implicit weighting system in an adjustment technique, so it must be clear for which population a global treatment estimate is most appropriate (33).

It is important to recognize that there are common challenges in the collection, notification, and use of epidemiological data, such as the exhaustiveness and representativeness of the results and their comparability in time, among others. Therefore, it is necessary to identify the strongest analytical designs (among them the interrupted temporal series and comparative longitudinal studies), accompanied by sensitivity analysis of the results and being explicit, starting from the design, in the type of biases and problems that can be found in the data analysis that is available (34).

Concerning the evaluation of the immune response to the different types of vaccines, it has been oriented both to the antibody-mediated response and that mediated by cellular immunity. Among the antibody-mediated response, the reference standard has been established with the specific neutralizing antibody response against spike proteins of the virus, and a proxy to this response assessing neutralizing capacity has been measured in other studies by immunoglobulin G (IgG) antibody levels against the SARS-CoV-2 receptor binding domain (RBD) (35).

In the different studies, the decrease in the response levels to specific neutralizing antibodies was assumed to indicate the vaccine protection level when the levels of specific neutralizing antibodies fell between 4 and 6 months. The statistical methods used for their measurement are not homogeneous among all studies which has been used to recommend the application of boosters with vaccines produced in homologous or heterologous platforms of those received in established vaccination schemes (36, 37).

To assess the duration of vaccine protection in the real world, it is also important to consider the difficulties in assessing the cellular memory immune response. The measurement of the CD4+ and CD8+ T lymphocytes response expressed in the production of different activation markers is heterogeneous, depending on antigenic stimuli such as peptides from circulating virus variants, cells from infected individuals, or peptides from different vaccines, in addition to diversity in the host response, which does not allow to have precise indicators to define optimal vaccination schedules (38, 39).

In this context, inactivated whole virus vaccines, such as CoronaVac, by preserving epitopes of the virus, could respond in a broader spectrum to the different variants of circulating viruses or to new mutations, which could lead to the optimization of global vaccination schedules (10).

The main strength of our study lies in its systematic development, which reduces the possibility of biases in study selection. The use of different databases, including Latin American ones, allows for a broader search, although it is acknowledged that due to the magnitude of research on this topic, there may still be unreported or unfound studies, behaving as gray literature. The review results enabled us to achieve our objective, which was to describe how the efficacy/effectiveness of COVID-19 vaccines has been evaluated, with emphasis on CoronaVac. This allowed for the identification of some differences in these methods and some persisting gaps in defining more homogeneous methods for evaluation, regardless of whether these studies had high or low certainty in their evidence, which should be revisited if the objective is to evaluate the effectiveness and/or safety in the population of these interventions. However, the findings presented could be assessed and discussed with broader groups of experts in the field, which would help generate more accurate recommendations regarding their significance and potential implications.

In addition to the mentioned limitations, it is important to acknowledge that this type of review, having less precise question definitions compared to systematic reviews of effectiveness and safety (with their PICO structure), may result in some gaps in the application of search terms that could affect the results. Additionally, the vast amount of information, as was the case in our review, can create difficulties in synthesis and analysis, so it is crucial, as mentioned, to continue the discussion in groups with increasingly greater expertise in the subject (40). Lastly, while it is tempting to provide quantitative results regarding the synthesis conducted, the most important aspect is to address the original question regarding the gaps in the evaluation of these vaccines.

Published information on the evaluation of the efficacy/effectiveness and safety of the different vaccines against COVID-19 is abundant. However, there are differences in terms of vaccine application schedules, population definition, outcomes evaluated, follow-up times, and safety assessment, as well as non-standardization in the reporting of results, which may hinder the generalizability of the findings. It is important to define the relevance of the analysis methods in advance, considering these differences and the heterogeneity that can be produced in the analysis and meta-analysis of this information. It is important to generate meetings and consensus strategies for the methods and reporting of this type of studies, which will allow to reduce the heterogeneity in their presentation and a better understanding of the effect of these vaccines.

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

JA-Á: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Investigation. PA-V: Writing – original draft, Writing – review & editing, Investigation. DA-L: Writing – original draft, Writing – review & editing, Investigation. AC-E: Writing – original draft, Writing – review & editing, Investigation. AG-H: Writing – original draft, Writing – review & editing, Investigation. LL-C: Writing – original draft, Writing – review & editing, Investigation. IM: Writing – original draft, Writing – review & editing, Investigation. DO-L: Writing – original draft, Writing – review & editing, Investigation. GQ: Writing – original draft, Writing – review & editing, Investigation. SR-B: Writing – original draft, Writing – review & editing, Investigation. CR-B: Funding acquisition, Writing – original draft, Writing – review & editing, Investigation. PR: Writing – original draft, Writing – review & editing, Investigation. MS-O: Writing – original draft, Writing – review & editing, Investigation. CT-A: Writing – original draft, Writing – review & editing, Investigation. MA-M: Funding acquisition, Project administration, Writing – original draft, Writing – review & editing, Investigation, Project Administration.