A large-scale cross-country trial of WHO’s Immune Patrol: toward a game-based digital curriculum for vaccine readiness and global health education

Vaccine-preventable diseases remain a global threat to public health, and despite the proven effectiveness of vaccines in protecting against these diseases, some people still avoid vaccinations for themselves or their children. Since barriers to protective health behaviors are diverse, overcoming them, and thereby reducing immunization gaps, necessitates diverse strategies. One such approach is to leverage health education through digital formats like online platforms and game-based learning, which can be implemented across geographical borders. We report a large-scale test of the World Health Organization’s Immune Patrol, a game-based middle school curriculum on vaccines and immunization. Using a matched quasi-experimental trial, classes in Armenia and the Republic of Moldova were exposed to local language versions of either five fully game-based lessons on topics including immunization, preventable diseases and digital source criticism, or non-game control materials on the same topics matching the digital and physical activities, without overt game elements. Significant learning outcomes were found across both countries, yet with differing gains among the classes within each country. Students assigned to game-based learning were slightly more motivated to learn about vaccine-preventable diseases, and to actively protect people around them through vaccination following the Immune Patrol experience. Qualitative observations highlight how practical and cultural differences can challenge homogenous implementation of learning packages like Immune Patrol across borders, but also how technology-enhanced collaborative classwork is welcomed in different ways across disparate classrooms. Results support the feasibility of global deployment of digital health education, but underscore the need for local support, and sensitivity to variations in culture, resources, teaching approaches and national education systems, when digital education initiatives like WHO’s Immune Patrol are set to transcend global borders.

1 Introduction

The significant association between health literacy and health-seeking behavior in the form of vaccination uptake observed during the recent COVID-19 pandemic, have emphasized the need for comprehensive health literacy across all ages (Fenta et al., 2023; Van den Broucke, 2014). In efforts to achieve and sustain high vaccination coverage to help protect public health and well-being, many strategies focus on educating parents and caregivers about vaccines through various means (Olson et al., 2020; Kaufman et al., 2017), however, the complexity of vaccine-related decisions suggests the necessity for wider-ranging interventions, including incorporating health education into school curricula, which requires innovations to support adoption of credible learning materials (European Centre for Disease Prevention and Control, 2021). As population-wide health literacy is a collective responsibility for society as a whole, innovation and collaboration within the education sector can hopefully promulgate knowledge and strengthen the potential for positive health behaviors from a young age.

Breaking down complex scientific topics like cellular functions and herd immunity, however, is a nontrivial challenge, which requires engagement with several layers of interlocking information, ranging from societal to cellular mechanisms. This requires both socially and educationally supportive environments, as well as credible and easily accessible materials, for learning and behavior change to occur. In this context, schools emerge as key sites for advancing health literacy. School intervention can even have a positive spillover effect, as studies have shown that school-based health education can significantly impact caregivers’ attitudes and behaviors (Abderbwih et al., 2022; Proios et al., 2022; Gulesci et al., 2020; Berniell et al., 2013; Sabater et al., 2020). Therefore, incorporating health topics into school curricula represents an effective and equitable method to enhance children’s health knowledge, and foster health-conscious behaviors.

Motivated by these considerations, the WHO Regional Office for Europe developed Immune Patrol, a hybrid digital and physical (“print-and-play”) game-based educational package designed to acquaint children aged 10–12 with the intricate workings of the immune system, the transmission of diseases, and the pivotal role that vaccines play in combating potentially life-threatening infectious diseases at an individual and societal level. Beyond imparting domain knowledge, Immune Patrol is also intended to nurture essential skills in critical thinking and source criticism, thereby involving children in a discourse concerning misinformation and unsubstantiated media content.

Immune Patrol is designed as an easy and scalable tool to help teachers worldwide incorporate essential information about vaccination and infectious diseases, though its reach may be influenced by resource availability and teacher training. For this purpose, attractiveness, ease of use, and scaffolding throughout activities on subjects where teachers may not be experts, is essential. Immune Patrol thus combines traditional classroom activities, such as discussions and writing tasks, with advancement through a digital online platform, borrowing the esthetics and progress experience of gaming, including digital exercises in the form of mini-games and simulations.

The present study evaluates the implementation of Immune Patrol in the Armenia and the Republic of Moldova through a quasi-experimental mixed methods design. This large-scale trial was carried out to identify barriers to implementation, assess learning outcomes, and test the motivation of students. Random assignment into game and non-game (control) classes was further employed to identify the selective effects of game-based elements. Testing was carried out in two different countries in order to map processes, and potential challenges, to rolling out a digital educational offering like Immune Patrol on a large scale in different contexts. Prior research highlights the influence of sociodemographic factors, teacher intentions, and resources on the effectiveness of educational technology (Arnold, 2011; Hanghøj et al., 2018; Lieberoth et al., 2018). Documenting effects and understanding the challenges pertaining to country- and classroom implementation is therefore critical.

This study serves to test the feasibility of deploying WHO’s Immune Patrol globally in the near future, but also to remedy certain shortcomings in studies of game-based learning, by using active controls to isolate the effect of game components, and by both qualitatively and statistically accounting for variations arising from implementations by very different teachers, in different classrooms, and in our case, in the disparate educational systems of two separate country settings.

2 Literature review

Vaccinations, a mainstay in public health for over two centuries (Mäkelä, 2000), are pivotal in preventing myriad life-threatening diseases and have been instrumental in saving millions of lives each year (World Health Organization, 2019; Kaur et al., 2023; Hakim et al., 2024). However, despite their proven effectiveness, some people still postpone or do not get vaccinated (Carrieri et al., 2023; Leonardelli et al., 2023). Barriers to vaccination are multifactorially influenced by issues like accessibility, safety concerns, trust toward governments and healthcare systems, online/media/local misinformation, and lack of awareness regarding the threat posed by infectious diseases relatively to the benefits of vaccination (Brown et al., 2010; Kaufman et al., 2021; Nuwarda et al., 2022). Overcoming vaccination barriers necessitates diverse strategies to address immunization gaps. One of these approaches leverages our current understanding of health education.

The 2020 Schools for Health in Europe (SHE) monitoring report found that the primary health promotion topics addressed in schools across 24 of its member countries were physical activity, healthy eating, smoking, alcohol, drugs and substance use, hygiene, COVID-19, sexual health, violence, and mental health (Schools for Health in Europe (SHE), 2020). These topics collectively reflect the core areas of health promotion and education initiatives within the school system across the European Region, the topic on vaccines and immunization are not structurally embedded in school curricula.

School-based interventions have shown positive effects on caregivers and communities, including improved knowledge and behavior. For instance, interventions have led to increased awareness of nutrition (Abderbwih et al., 2022), enhanced physical activity among fathers (Berniell et al., 2013), reduced domestic violence attitudes in mothers (Gulesci et al., 2020), and better understanding of heart disease symptoms in grandparents (Proios et al., 2022). Furthermore, school-based interventions have positively impacted parent–child communication, encouraging parental participation in healthy lifestyle programs (Sabater et al., 2020). Research from low- and middle-income countries indicates that children’s compulsory learning enhances parents’ health literacy and promotes healthier behaviors (Sabater et al., 2020; Smith-Greenaway et al., 2018).

Some publications suggest that the age group 10–12 is apt for education on preventable disease, as children at this stage demonstrate the cognitive maturation necessary for abstract thinking (Piaget and Inhelder, 2000; Nielsen and Tanggaard, 2019; Miller et al., 2008). The complexities of the immune system and the significance of immunization require some abstract thinking and a certain level of cognitive development, which children typically attain around the age of 10 (Piaget and Inhelder, 2000; Nielsen and Tanggaard, 2019; Miller et al., 2008). A meta-analysis of 33 studies concluded that digital game-based education is most beneficial and effective for primary school children’s learning, as pupils might be attracted to the novelty of the game and might have a longer time to figure the game’s rules out (Wang et al., 2022). There is growing empirical support for gamification’s positive effects on motivation, engagement, and learning outcomes in educational contexts (Chan and Lo, 2024). Game elements have been found to enhance learning by increasing student engagement, and structuring learning trajectories through a series of challenges supported by feedback and a sense of progress (Dahalan et al., 2023). This ability to establish interest and maintain student attention has been found across educational levels and areas (Noroozi et al., 2020). A systematic review focusing on digital game-based learning in elementary school children identified a notably positive impact in this group, particularly in enhancing knowledge acquisition and understanding of content (Hussein et al., 2019). Incorporation of games in educational contexts can also foster collaborative and peer-supported learning (Martindale et al., 2024).

Interventions, such as ImmunizziAMO, Vaxcards, FightHPV, Morbiquiz, and others (La Torre et al., 2020; Epstein et al., 2021; Fadda et al., 2018; Ruiz-López et al., 2019; Montagni et al., 2020), have integrated game-based learning with vaccination and immunization education, yielding diverse outcomes. A systematic review by Ohannessian et al. (2016) and Gris and Bengtson (2021) underscores the importance of formal evaluation and publication of results for interventions using game-based learning, especially in the context of vaccination education. Such rigorous assessment is often lacking, but crucial for demonstrating efficacy, utility, and transparency (Montagni et al., 2020; Ohannessian et al., 2016) e.g. before rolling out solutions across country borders, as is the intention for WHO’s Immune Patrol. Online platforms mimicking game affordances offer an access point for teachers and their students, and thus hold significant promise for enhancing vaccination knowledge and potentially increasing vaccination coverage, as interventions in school settings have been found to create spillovers to caregivers and communities (Abderbwih et al., 2022; Gulesci et al., 2020; Berniell et al., 2013).

Since global disease prevention spans levels like biology, population statistics, information literacy, and personal attitudes, choosing an approach that can both encompass complexity, engage problem solving, and support motivation is paramount. There is convincing evidence that games can combine precisely these features when employed in the classroom (Perrotta et al., 2013). This prompted WHO Europe to select a game based approach for their new education package which, in turn, needs rigorous testing.

While game-based learning has shown promise in many areas, however, there exists considerable variation in the quality of research and outcomes (Hamari et al., 2014). Furthermore, research for global approaches must be sensitive to variations in cultures and classroom resources. While digital game-based materials are known to inspire motivation and sustain engagement (Ryan et al., 2006), few studies employ closely matched active controls to assess the specific contributions of game elements, relative to the specific learning activities and instructional design choices they scaffold or supplement (Lieberoth et al., 2018; Wang et al., 2022; Nadi-Ravandi and Batooli, 2022). This must be considered, when evaluating a digital curriculum intended for distribution across many different countries and teaching traditions.

3 Methods

The study employed a quasi-experimental design to compare two groups (game and control) on content test performance (dependent variable 1) and motivation to engage with issues of preventable disease (dependent variable 2), following six two-lesson “treatments” simulating the real-world use of Immune Patrol in classrooms. To reduce potential bias and increase internal validity, randomization was conducted at the class level within participating schools. Recruitment followed a predefined sampling frame designed to ensure balance across rural and urban settings, as well as socioeconomic representation. While full randomization across all country settings was not feasible due to institutional and logistical constraints, the chosen design allowed for a controlled comparison that reflects the likely implementation context of a global digital learning tool.

To support insights on the implementation process at schools, the study adopted a concurrent explanatory mixed methods framework, with field data collected during and after the intervention (Greene, 2007). Specifically, qualitative classroom observations and interviews with students and teachers were collected to identify barriers to implementation in different country contexts, and to provide a backdrop for interpreting the statistical outcomes (Schoonenboom and Johnson, 2017; Creswell, 2007).

3.1 Intervention materials: WHO’s Immune Patrol

The Immune Patrol is a digitally distributed game-based educational package intended for learning in groups of three to six pupils. It comprises six 45–90-min modules, sequentially structured to cover diseases and transmission, the immune system, vaccination, herd immunity, vaccine development, and source criticism. Before beginning the first module, students create a group avatar, which accompanies them through a framing narrative, in which the animated instructor Juliette takes pupils through basic training, with each lesson representing a separate room in the Immune Patrol academy building. Each module starts with an explainer video on the topic, followed by a collaborative classroom task, an interactive online mission, and a concluding debriefing by the teacher, during which pupils are encouraged to reflect upon their acquired knowledge. This hybrid approach marries traditional classroom activities and discussions with digital interactivity and a comprehensive information bank, and supplies a supporting structure for delivering each lesson similarly across classrooms and countries.

In order to run Immune Patrol, the teacher accesses an online control panel to project videos, initiates activities and tracks student progress. The platform is supplemented with a teacher information package, consisting of a video overview of Immune Patrol, descriptions of the materials needed for classroom exercises, such as craft tools or printouts from the platform, and pedagogical recommendations. The student version is essentially a playable version, which includes videos, access to a mini-encyclopedia, as well as instructions and input-interfaces for exercises like quizzes, arcade games, simulations, and writing and art tasks. Teacher and student versions operate concurrently on separate devices or browsers, with the teacher initiating and ending each step from their control panel.

To distinguish the learning and motivation effects of game components, this study compares the outcomes of a fully gamified approach (‘game’ group) to control classes experiencing the Immune Patrol platform and exercises, but doing closely matched classroom activities in place of game elements. For instance, the fifth Immune Patrol module includes a mini-game that challenges pupils to spot warning signs and incongruities across a series of simulated online pages made to look like more or less like official websites, popular video apps, etc. while getting continuous feedback on their choices. In the control condition, students used worksheets representing the same challenge, with the same graphics, images and symbols used online, but absent the “game feel” of digital interactions. In another challenge, control students had to guess at appropriate population thresholds for herd immunity under various conditions, and check flash cards in place of running game-like graphical simulations online.

3.2 Recruitment and setting

Before the roll-out of Immune Patrol, the WHO Country Offices in Armenia and the Republic of Moldova collaborating with their respective Ministries of Health and Education, were responsible for recruiting a sufficient number of eligible schools and classes. Schools and teachers had the opportunity to review all Immune Patrol-related materials to ensure they were prepared to implement the program in their classes. As outlined in Table 1, the sampling frame and recruitment protocol utilized within-school randomization to minimize self-selection bias risks during the fall–winter 2022 and spring 2023 periods.

Table 1

Table 1. School inclusion and exclusion criteria.

Following the initial recruitment phase, selected schools identified eligible classes and teachers for participation. Informed verbal and written consent were obtained from all teachers, parents, and students. Classes were then randomly assigned to either the game or control group. Correspondingly, teachers received a Teacher’s Manual via email (see Supplementary Material), tailored to their assigned experimental group(s), along with written instructions for accessing the online Immune Patrol platform and survey site for the pre and posttest, using participant codes, and guidelines for pre- and post-testing. Additionally, teachers were offered a training session, lasting 8 h and conducted by WHO personnel, to familiarize them with the materials.

Ten schools in the Republic of Moldova and six in Armenia were recruited. Each school contributed with at least two volunteer classes per grade level, and was asked to randomly and evenly assign these to the two conditions. In Armenia the level of participation varied, with some schools offering up to six classes, and some seemingly favoring ‘game’ assignment when there were an uneven number of classes in 1 year (see section 3.6). In Republic of Moldova exactly two classes from the same grade in each school participated and the selection of classes simultaneously randomized between the two conditions. The study was conducted in December and January 2022–23 in Armenia and the weeks leading up to the summer break of 2023 in Republic of Moldova.

Participating schools and classes were encouraged to incorporate Immune Patrol into their curriculum for a period of up to 6 weeks, covering at least one module per week. However, schools and teachers had the flexibility to adapt the program to their specific needs and schedules. As a result, some classes completed the entire package, including the pre- and post-surveys, within 2 weeks.

The Immune Patrol rollout and study in both countries followed a unified protocol, but each received additional support during implementation. In Republic of Moldova, trained medical students assisted teachers with technical aspects in each lesson. In Armenia, specialists in Infectious Diseases and Immunology, along with a WHO National Professional Officer, frequently attended educational sessions in schools, conducting briefings and responding scientifically to queries from students and teachers. This support exemplifies the enhanced effectiveness teachers can achieve with experience and confidence in using Immune Patrol. However, it may also have mitigated some initial barriers that would typically arise in the early stages of implementation.

3.3 Participants

In the Republic of Moldova, our sample initially comprised 1,360 responses (750 pre-survey and 610 post-survey). After data cleaning, which involved the exclusion of incomplete and duplicate entries, the sample consisted of 1,085 responses (560 pre-survey and 525 post-survey), from school classes in the 11 to 14 age range (reported age M = 12.96, SD = 0.628, Median = 13, range = 8–14 years), including 52.8% female, 45.6% male, and 1.58% preferring not to specify. Pre- and post-responses from 363 students were successfully paired (195 from game- and 168 from control classes) using IDs, and included in the analysis.

In Armenia, a total of 918 responses were collected, from school classes in the 11 to 14 age range (M = 12.88, SD = 0.916, range = 8–14 years), with 56.65% female, 42.10% male, and 1.25% preferring not to specify. It was not possible to pair responses adequately from pre-survey to post-survey due to incongruous use of school, class, and participant IDs at the two timepoints. After data cleaning, and checking responses against test days the participating schools, all complete responses (639; 319 pre- and 320 post-survey, 427 game- and 212 control class) were included in the analysis based on assurances from the WHO Country Office that the pre- and post-survey responses were valid, despite the inability to reliably match individual students. This preserved the data for between-countries comparison, but unfortunately precluded repeated measures and nested data analyses.

In order to be able to use all data collected and given that pairing of the pre- and post-responses was only possible in Republic of Moldova, this was not a criteria for inclusion into the aggregated data set, which included all valid responses from both countries.

3.4 Qualitative documentation

Three sources of qualitative materials were collected concurrently with the quantitative data, in order to facilitate a convergent mixed-methods interpretation of the findings: classroom observations, group interviews with students and teachers, and individual interviews with teachers. These were generated as informal in-country case studies, in subsets of participating schools selected based on accessibility and national representativeness. Local WHO staff or professionals with backgrounds in education and fieldwork conducted the observations and interviews. Information was collected across a range of urban–rural and wealth-poverty contexts, as well as among various age groups to ensure relative representativeness. This diversity accounted for potential variations in game deployment due to differences in factors such as student composition, teacher resources, and technological access. The recruitment of teachers, students, and classes for interviews and observations was guided by considerations outlined in Table 2.

Table 2

Table 2. Recruitment considerations.

Student interviews were conducted in groups of three to five students. The interviews began with a restatement of the purpose and verbal consent. Students were then asked to engage in an “I used to think…” reflection, where they considered at least one way in which playing Immune Patrol had changed their perceptions and then shared it with the group. A total of three student group interviews were conducted in Armenia each lasting between 30 and 50 min. Group interviews were not conducted in Republic of Moldova. Teacher interviews were conducted in a more informal manner than the student interviews. Teachers were regarded as experts in their field, offering insights into how Immune Patrol integrates into their school and country. They were also considered users of the new technology, capable of sharing their motivations, challenges, and technical issues encountered. Interviews were conducted in person at the school or via phone after the completion of Immune Patrol. A semi-structured interview guide was used, and interviewers were encouraged to incorporate their observation notes if they had previously been in the teacher’s classroom. In Armenia, six individual teacher interviews were conducted, each lasting 15–20 min. In addition, a focus group discussion with 11 other teachers was conducted to gather insights from teachers that implemented Immune Patrol across different settings. In Republic of Moldova, an in-person semi-structured group interview was carried out with 10 teachers as a closing event for the Immune Patrol trial. During this session, teachers shared both their own experiences with and their students’ perspectives on Immune Patrol, thereby compensating for the absence of direct student interviews.

3.5 Quantitative measures

The quantitative component aimed to assess the effects of Immune Patrol on curriculum test performance and motivations, and test whether the game-based elements contributed additionally to these outcomes.

We tested changes to relevant curriculum knowledge, and changes to identified health regulation (see below), from before to after the Immune Patrol experience across both countries. We further tested the hypotheses that game classes would benefit from increased motivation during the activity, relative to controls, and that this would lead to better learning outcomes. Finally we assessed differences between countries using difference-in-difference analysis.

Survey measures were deployed during the first lesson, and on the day of the final lesson in both the game and control classes. The surveys were translated into the local language in collaboration with local WHO Country Offices, who ensured that the terminology was appropriate to the cultural context, and distributed by teachers. The pre-survey and post-survey consisted of the same questions with a 9-question version of The Children’s Locus of Causality scale (c-PLOC) (Pannekoek et al., 2014) added to the post-survey (see Table 3).

Table 3

Table 3. Themes of questions in the pre- and post-survey.

In addition to the established c-PLOC measures of intrinsic motivation (α = 0.80, interest and enjoyment derived from a school activity), extrinsic regulation (α = 0.85, participation due to institutional demands) and identified regulation (α = 0.83, wanting to learn and do well), a health-oriented identified health regulation version of the latter was adapted to serve as a proxy measure of motivations to protect yourself and others though learning about preventable disease, which showed excellent reliability (α = 0.85, 1.” I want to learn things that could be important for my health, 2. “it is important to me that people around me are healthy, and avoid getting sick,” 3. “I want to know more about health and diseases”).

The quiz test comprised 5 × 5 items corresponding to each of the five main lessons, excluding an introductory “module 0.” Students were required to assess each statement as “true” or “false” on a symmetrical scale with the options; “true, very certain,” “true, certain,” “true, uncertain,” “false, uncertain,” “false, certain,” “false, very certain.” The use of certainties invites students to make educated guesses, and allows researchers to investigate patterns of certainty regarding erroneous beliefs, which is not part of the present analysis. The pool of quiz items was collaboratively developed by the WHO Regional Office for Europe, Aarhus University, and the Danish game developers. A larger pool of initial true/false items was generated, and item face validity determined through an early pilot in Denmark, combined with WHO expert judgment of thematic centrality, and then reduced to five items per lesson. The selection favored statements that were comprehensible at the post-survey stage, unlikely to unintentionally seed false beliefs, and displayed sufficient item facility, while striving for a balance between true and false correct answers.

3.6 Data analysis and assumption checks

Assumption tests indicated heteroscedasticity for quiz test scores from most lessons, leading to the use of nonparametric tests when available. Main effects were found for age and gender, with females scoring slightly higher on both the curriculum test, Welch’s t (1,495.94) = 2.72, p = 0.018, and the identified health regulation scale, Welch’s t (1,517.82) = 3.00, p = 0.008., and higher student age correlated with slightly better test scores (r = 0.10, p < 0.001).

Simple comparisons revealed significant differences between Armenia and the Republic of Moldova at the time of both pre- and/or post-test for all but one lesson (4: vaccines), leading to the inclusion of country variance as a second step in the difference-in-difference analyses used to test effects of game vs. control assignment. The distribution of condition assignments with (67% of observations in Armenia coming from the game condition versus 52% in Republic of Moldova, was also found to be statistically uneven χ²(1) = 34.5; p > 0.001, V = 0.14), which must be taken into account when interpreting results.

In the following analyses we treated quiz test scores as the primary dependent variable, and identified health regulation as the secondary dependent variable. In all analyses we performed a first step combining all outcomes, and a second step accounting for between-countries variance.

4 Results

All measures before and after Immune Patrol are summarized in Table 4.

Table 4

Table 4. Changes in participants’ subject knowledge (% correct) combined and by country.

Test scores indicate that pupils’ overall knowledge increased significantly from before to after Immune Patrol across countries, Welch’s t (1,694) = 16.53, p > 0.001, d = 0.80. Significant improvements were seen for all lessons, except the initial introduction to the immune system. Effect sizes were largest for the lessons on herd immunity and vaccine development (see Table 4 and Figure 1). The large overall effect size suggests accelerated learning compared to typical instruction (Hattie, 2009).

Figure 1

Figure 1. (A) Total % correct on test scores before and after Immune Patrol. (B) Self-reported identified health regulation before and after Immune Patrol. (A,B) Violin plot of test scores and identified health regulation pre- and post-Immune Patrol (scale 0–100). Dot density and “violin girth” illustrate distributions of responses, while boxplots show median and interquartile range. While %correct suggest Gaussian distributions skewed toward above-chance performance pre and post, but with a more uneven distribution pre, the “splattered top” of identified health regulation suggests a ceiling-e!ect, which is additionally pronounced after Immune Patrol.

Pupils’ recognition of the personal need to understand and act on health issues as measured by the identified health regulation scale similarly increased from before to after, Welch’s t (1,721) = 4.01, p > 0.001, d = 0.19. The most prominent improvement (d = 0.32) was seen in Armenia.

Next, we analyzed the relative effects of game versus non-game condition assignment, and country implementation, on learning outcomes.

A difference-in-difference analysis (Jiménez and Perdiguero, 2019) revealed significant effects of condition (game vs. control) F (1) = 5.04, p = 0.025, η²p = 0.008, and of time (pretest vs. posttest), F (1) = 225.92, p < 0.001, η²p = 0.098 on test scores, but no significant interaction between time and condition F (1) = 2.28, p = 0.131, η²p = 0.002 (Figure 2A). When added in step 2, the effect of country on test outcome was also significant F (1) = 34.449, p < 0.001, η²p = 0.017, as were interactions between country and time F (1) = 18.310, p < 0.001, η²p = 0.011, showing that the larger gain seen in The Republic of Moldova (Table 4) was significant both between countries and over time.

Figure 2

Figure 2. Estimated marginal means of identified health regulation from before to after Immune Patrol in game vs. control conditions, error bars 95% CI. (A) Step 1: Identified health regulation accounting for country variance. (B) Step 2: Identified health regulation accounting for country variance.

There was also a significant interaction between country and condition F (1) = 49.808, p < 0.001, η²p = 0.028. No significant interactions were found between condition and time F (1) = 451.72, p = 0.064, η²p = 0.002, or between condition, time and the country of implementation F (1) = 7.30, p < 0.814, η²p < 0.001 (Figure 2B).

Post-hoc comparisons (Tables 5, 6) reveal that the largest systematic change from before to after the Immune Patrol experience occurred in the game condition. Game-classes scored lower on quizzes than controls before the intervention, but not after.

Table 5

Table 5. Post hoc comparisons of test performance in game vs. control conditions over time.

Table 6

Table 6. Estimated marginal means of test performance.

Similarly, a significant effect of time, F (1) = 12.90, p < 0.001, η²p = 0.01, but not of condition F (1) = 8.73, p = 0.855, η²p = 0.01, was found for identified health regulation. A very small interaction was also detected between time and condition, F (1) = 5.97, p = 0.015, η²p = 0.003, indicating that although values did not differ significantly, identified health regulation changed more in the game condition from pre to posttest, than in control classes (Figure 3A).

Figure 3

Figure 3. Estimated marginal means of identified health regulation from before to after Immune Patrol in game vs. control conditions, error bars 95% CI. (A) Step”1: Identified health regulation accounting for country variance. (B) Step”2: Identified health regulation accounting for country variance.

In the second step, significant effects were found for country, F (1) = 22.15, p < 0.001, η²p = 0.01, and for time F (1) = 12.15, p < 0.001, η²p = 0.01, but not condition, F (1) = 0.17, p = 0.678, η²p < 0.001. Interactions were found between time and condition F (1) = 5.35, p = 0.021, η²p = 0.003 as well as between country, time and condition, F (1) = 4.07, p = 0.030, η²p = 0.003, but interactions between country and time F (1) = 2.87, p = 0.090, η²p = 0.002, and country and condition F (1) = 2.86, p = 0.091, η²p = 0.002 did not reach significance.

Post-hoc analyses (Tables 7, 8) explain this pattern. While no significant change in identified health regulation was seen in the control condition, pupils in the game condition expressed significantly higher identification with the need to learn and act on communal health issues, than they had before. Even though they started slightly below controls, the game group also ended up expressing significantly higher identified health regulation than controls after Immune Patrol. Note, however, that effect sizes fall in the minor range.

Table 7

Table 7. Post hoc comparisons of identified health regulation in game vs. control conditions over time.

Table 8

Table 8. Estimated marginal means of identified health regulation.

Finally, both intrinsic motivation (Table 9), identified regulation were found to correlate with students’ test-scores after Immune Patrol, and (to a larger degree, which is unsurprising given the close relation between items content) with their level of identified health regulation. A correlation matrix with Pearson coefficients (right upper half) and distribution plots showing each slope (left lower half) can be seen in Figure 4. General identified regulation thus had a greater impact on students’ recognition of the need to learn about health and vaccination, in order to make a difference for themselves and others (see Figures 3A,B; Table 6), than on knowledge outcomes (Figures 2A,B; Table 8). A Mann–Whitney U tests revealed significantly higher levels of intrinsic motivation in the game condition, z = 80,256.50, p = 0.009, r = 0.09, but no between-groups difference for identified regulation, z = 83,354.50, p = 0.134, r = 0.06 or extrinsic regulation, z = 82,280.00, p = 0.08, r = 0.07.

Figure 4

Figure 4. Correlations between motivation measures (reported at posttest) and dependent variables.

Table 9

Table 9. Students’ reported motivations during Immune Patrol.

4.1 Qualitative findings on implementation and reception: students

The feedback from students illustrated various sides of learning complex concepts through the platform, exercises, and game elements. Most expressed that Immune Patrol as a whole made topics that they typically perceived as less interesting, more accessible and engaging. Many, however, also noted that the initial learning curve was steep, particularly in adjusting to the group work and hybrid online-offline style, which diverged from their typical classroom experience.

Students frequently mentioned liking the graphics and technology, which aligns with previous findings that game aesthetics or cartoon graphics can initially boost engagement (Wang et al., 2022; Lieberoth, 2015). However, in contrast to the initial novelty effect, concise explanations with videos and interactive uses of the mini encyclopedia were separately remembered for their facility in breaking down complex scientific topics, such as cellular functions and immunity. Students also relayed how the collaborative group-based structure, combined with the online resources, allowed them to develop, discuss and explain challenging concepts to each other across lessons. This highlights how the adaptive, learner-centric online organization of knowledge in Immune Patrol allows students to revisit relatively compounded content at their own pace, as the need arises across lessons. This was seen to enhance understanding, but also foster a sense of autonomy and collaborative learning, confirming the ambition that Immune Patrol should provide a both socially and technologically supportive educational environment for learners irrespective of local school systems. The variety of hands-on activities like drawing and staging plays were also described as fun, different and conducive to learning, compared to typical lessons.

Some students, however, also found it difficult to familiarize themselves with the online interface, and navigate the expectations they were meant to meet in this unfamiliar teaching format. Once started, however, the majority of students reported that transitioning and adapting to Immune Patrol was relatively straightforward. This underscores the importance of initial communication regarding the program’s purpose and structure, and teacher expectations. Conveying this, in turn, requires familiarity and ongoing supportive monitoring on the part of teachers. Furthermore, when several modules were taught in a day, the pacing was perceived as too intense. Stretching Immune Patrol over several weeks, as will typically be the case when a curriculum is used during a school year, would enhance consolidation and reduce cognitive load, making the educational experience more manageable and effective (Cromley and Byrnes, 2012; Carlos et al., 2023).

4.2 Qualitative findings on implementation and reception: teachers

When interviewed, teachers believed that student learning was accelerated, and they reported observing great student enthusiasm. In the Republic of Moldova, it was noted that performance on a subsequent biology test was considerably above normal, particularly in topics related to immunity and vaccines. One suggested that this was because Immune Patrol enriched students’ vocabulary by introducing new scientific terms like herd immunity or dendritic cells, which later were observed to be revisited in both assignments and group discussions.

However, integrating Immune Patrol into the classroom also presented challenges. Teachers expressed that the volume of the mini-encyclopedia could be overwhelming, though the structured content was appreciated. Despite the user-friendly design, teachers also found that their guidance was essential in order for students to gain the full benefits of the program, supporting previous findings stressing the crucial role of teachers for fruitful game-based learning (Hanghøj et al., 2018; Bressler et al., 2019).

Immune Patrol’s blend of online elements with physical tasks like crafts and guessing games, combined with traditional classroom instruction, presented many teachers with a novel challenge. They had to navigate both online and in-classroom components, which was experienced as a particularly demanding task in Armenia where, unlike in Republic of Moldova, teachers did not have access to technical support from study assistants. While Armenian teachers received professional support from WHO specialists, they still faced technical challenges and uncertainty about managing class activities.

The teachers felt that students took an unusual degree of responsibility for their own learning, showed personal engagement in health topics during debates, and transitioned effortlessly between the virtual elements of the game and physical classroom activities, but the interactive nature of Immune Patrol should thus not be taken to suggest that a teacher is not needed. Given Immune Patrol’s high interactivity, adequate teacher preparation, including studying the manual, familiarizing with the platform, and understanding student needs, became crucial for effective teaching and practical implementation. However, teachers in both countries emphasized that Immune Patrol was generally appropriate for students aged 10–14 and found ways to make the experience work for themselves and their students. They concluded that Immune Patrol struck a balance by being neither overly challenging for the younger children nor excessively simplistic for the adolescents.

Infrastructure limitations in schools also complicated the implementation. Wi-Fi network quality affected device connectivity, and limited access to necessary equipment like tablets, computers and projectors in some classrooms required special arrangements. Students also complained that the game elements “lagged” and teachers encountered frustrating waiting periods, both resulting from limited internet speed, Wi-Fi connectivity, and slow computers, in addition to human mistakes. Even though the Immune Patrol platform is not very heavy in terms of data transfer or CPU usage and could run in all locations selected in a serviceable fashion, local IT infrastructure thus still determines the opportunities to use global offerings in digital health education. Technology will affect user experience when not functioning perfectly.

The qualitative data thus calls attention to technical and logistical challenges that the educators needed to navigate, as well as the importance of preparation and appropriate spacing of activities across school weeks.

5 Discussion

The purpose of this international trial was to evaluate the learning impact and feasibility of implementing a WHO digital health education package across countries and classrooms. The large-scale experiment conducted in 2022–2023 sought to study the practical dimensions of implementing an international curriculum platform like Immune Patrol, to measure learning outcomes, and finally to understand the selective impact of game elements on student engagement and learning.

By employing the same protocol in two countries, we aimed to account for the fact that implementations and outcomes are likely to differ in non-identical educational systems, by incorporating between-countries differences into the analysis.

Firstly, marked learning gains were seen in both experimental conditions, and in both countries, with slightly larger changes in the Republic of Moldova, and in classes using the full game version. This supports the efficacy of both the hybrid online-offline model in general, and the combination of game-based learning with collaborative classroom tasks in particular, for teaching the fairly complex themes of health and preventable disease in middle school (Tamim et al., 2011; Yu et al., 2022). The greater improvement measured in Republic of Moldova also suggests that added preparation and institutional support have tangible benefits for learning—at least when a global digital offering like Immune Patrol is first adopted.

Secondly, while the integration of game elements into the lessons markedly improved intrinsic motivation during tasks, identified regulation, i.e., “seeing the point” of participating and getting acquainted with the subject, was high in both groups. These positive measures of motivation were much more related to final test scores than regulation by extrinsic demands like teacher authority (or, indeed, the upcoming test in Republic of Moldova). While the game element appeared to add extra fun and curiosity as expected given the voluminous existing literature on game based learning, helping students to develop a personal identification with the need to learn was a stronger driver for learning with Immune Patrol (Xu et al., 2023; Wouters et al., 2013). This outcome is consistent with other studies in showing a positive influence on learners’ attitudes and emotional engagement (Chan and Lo, 2024).

Lastly, the process of implementing Immune Patrol in Armenia and Republic of Moldova emphasized the need for teacher preparation (Molin, 2017; Ifenthaler and Schweinbenz, 2013), and in some cases technical support for successful integration of an international digital curriculum like Immune Patrol.

Indeed, although gains were seen in both Armenia and the Republic of Moldova, interactions between time, group and country also suggest that differences in the local context affected the impact of lessons, perhaps reflecting established teaching practices, the existing curriculum, and students’ prior experience. This is consistent with other studies that have shown the influence of context, culture, and learner diversity on the effectiveness of gamification (Martindale et al., 2024). The added level of close support during implementation in Republic of Moldova, particularly the involvement of trained medical students that provided technical as well as informational support during the lessons, as well as permanent methodological supervision and support, may also have created a more conducive and secure environment for teachers and students alike, hopefully mirroring what Immune Patrol would look like once teachers become experienced problem-solvers in their own right. In keeping with the existing literature (Molin, 2017; Jääskä and Aaltonen, 2022), the qualitative explanations underscore how local teachers can face challenges and an extensive cognitive load, when implementing international offerings like Immune Patrol. Global deployment of online tools demands a certain level of technical endowment, as well as proficiency and experience among teachers in using computers in educational settings in addition to the nontrivial task of understanding the entirety of the six-lesson process, and aligning expectations with students, who may not be familiar with the format either.

Together, the analyses show consistent improvements in knowledge and identification with the need to know about health and preventable disease. The nature and effect sizes of the results support the hypothesis that game-based learning added somewhat to outcomes, but not that Immune Patrol’s impact should be ascribed to the game elements alone. Instead, the platform, ongoing access to content, and student-centered group work appears to have been effective, in and of themselves.

Convergences between qualitative data from students and teachers call attention to the benefits of a coherent structure and information ecology maintained between Immune Patrol lessons. The rhythm of online “seeding” of both motivation and knowledge, followed by collaborative classroom work with continuing access to online resources, allows groups to revisit relatively compounded subjects like population-level immunity and online source criticism in the health domain, as the need arose across sessions. Immune Patrol thus seems to bring a socially and academically supportive learning environment into being, when teachers and students successfully align around the hybrid online-offline premise, that may differ somewhat from everyday offerings.

6 Limitations

Reliability concerns were evident in especially the Armenian data due to recurring or missing student IDs, and mismatched school codes. The disparity in group sizes in Armenia, hinting at either non-adherence to the protocol or a disproportionate use of game materials, also introduces potential biases. This irregularity could influence the findings and their generalizability for other cross-country comparisons. The exclusion of incomplete or mismatched responses would, however, waste costly international data, and could non-randomly favor more conscientious pupils or classrooms with stronger support. By including all data from Armenia, we accepted unknown noise from non-repeated responses, and these results may not be entirely comparable to Moldovan data. The motivation for doing so rests on ethical considerations respecting the effort and contribution made by schools and local offices in both countries, and the fact that the data in toto would still yield a valid representation of pre-post and between-groups differences. Consequently, the inability to conduct paired analyses and control for classroom variations as a hierarchical moderator limits the study’s comprehensiveness and applicability, but did not preclude meaningful analysis.

Limitations in the quantitative data are somewhat mitigated by our mixed-methods approach, where the qualitative layer provided an added, more nuanced, understanding of the intervention’s effects. The information gleaned from talking to students and teachers was instrumental in gleaning not just the multifaceted impact of an educational intervention that deviated markedly from what both students and teacher were used to, but it also offered a more complete picture of the context of the quantitative findings, including classes having filled in the relevant tests.

The detailed teacher training on how to facilitate both the case and control versions of the Immune Patrol was provided by the same WHO officers in both countries. However, the support may have inadvertently shielded the true extent of difficulties faced by teachers in implementing Immune Patrol. Consequently, our study may not fully reflect the barriers that can emerge in environments with limited or no technical support, potentially leading to an overestimation of the program’s ease of use and effectiveness. At the same time, our study shows what outcomes may look like in the early stages of adoption, while the teachers expressed much greater confidence in their ability to run Immune Patrol after this, their first experience.

Our sample size was determined by logistical necessity, and a priori analyses suggested that it would fall below the ideal size to detect the small effect sizes suggested by prior research (Hattie, 2009; Cohen, 1992). Since we did find quite small effects, this turned out not to be a major concern. While the increased risk of type 1 errors arising from multiple comparisons can be offset with statistical corrections, the effect sizes found also underscore the importance of not overemphasizing p-values, without considering the magnitude of findings. Cohen’s (1992) classification of effect sizes, while statistically conventional, does not directly translate to the effectiveness of educational interventions. In this context, Hattie’s (2009) analysis suggests a benchmark where effect sizes above those corresponding to a d of 0.15 are acceptable in educational settings, and 0.40 indicate substantial contributions beyond typical expectations. While some findings on motivational change and the selective contribution of game elements are on the small side, and Identified Health Regulation scores show patterns indicative of ceiling effects (see violin plots, Figure 4B), the overall learning effect of Immune Patrol is well within Hattie’s zone of accelerated learning effects.

The test we designed to gauge learning outcomes, however, is unlikely to fully capture all aspects of learning, and especially the more complex dynamics that characterize the interplay between biological, behavioral, and societal levels in immunization processes. Thus, test scores serve as a relative indicator rather than a definitive measure of all possible learning outcomes. A Rasch analysis at pretest showed good (Xu et al., 2023) item fit statistics of 1.00, and most items followed the shape of expected score curves, with the full item set visible in the slightly correct-skewed wright map. The most difficult items related to remembering very specific information such as the role of phagocytes, or whether 90% vaccine coverage ensures herd immunity. Inter-item correlations (0.01–0.37, with only two instances of r > 0.3) largely supported the assumption of local independence. Person reliability of ~35% while below the ideal for ability testing (Wright and Linacra, 1994) increased to ~54% after students had gone through Immune Patrol, suggesting a better correspondence with the curriculum than with prior knowledge. This is also a likely result of multidimensionality as subscales intentionally related to each of the five main lessons. More items with greater ability to distinguish high vs. low ability would be needed for a test of individual student ability for each lesson. As a measure of pre-post chance for Immune Patrol only, however, test scores varied enough (pre/post M = 61.96%/71.43%, Mdn = 64%/72%, SD = 11.34/12.41) to allow distinction over time, even though clearer learning results of Immune Patrol would probably be found using an item set with stronger Rasch properties. Identified health regulation was also highly correlated with its “parent” from c-PLOC, but not perfectly, suggesting some level of discriminant validity. As a very short measure, it was never intended to reach perfect psychometric properties, but rather to mirror the short scales designed for young pupils. High initial scores also suggest a ceiling-effect, which can be seen quite clearly in the violin plot (Figure 4B).

Finally, the delayed post-test gauges only the intervention’s immediate impact, which limits our ability to speculate on learning retention and long-term behavioral changes. For this, a follow up or staggered time series design would be needed, which was not feasible given the large number of stakeholders involved. The present absence of follow-up data restricts our understanding of the persistence of knowledge and behavioral changes over time.

7 Practical implications and future directions

By carefully designing matched control activities, which previous research has often lacked (Lieberoth et al., 2018; Nadi-Ravandi and Batooli, 2022; Wang and Lieberoth, 2016), we were able to dissociate the effectiveness of the Immune Patrol game elements as only one component in a broader learning design. Since the interaction between condition and country reached a medium effect size, design effects appear to be somewhat dependent on the contexts they are implemented in. In other words, the hybrid online+offline format, including the offline tasks with no added game motivation, seemed to work well to support active learning, as evidenced by how effect sizes related to the game vs. control conditions fell in the small to minor range. Immune Patrol’s game elements measurably influenced motivation and student outlooks in support of slightly better learning, but game-based learning is never better than the instructional design found beneath the (Creswell, 2007) graphics and game elements. The game elements seem to have especially enhanced the felt immediacy and personal relevance for the middle school students. Both in terms of engagement during lessons, and identification with the need to understand and act on preventable disease more broadly. Practical approaches to incorporating game elements also suggested in other studies include aligning game elements with learner needs, ensuring adequate teacher training, and addressing technological infrastructure (Chan and Lo, 2024; Martindale et al., 2024). As we have recently argued with colleagues (Yu et al., 2022) health knowledge cannot, and will not, make a substantial international impact without both a human impetus and practical avenues for behavior change.

Immune Patrol has since the test been implemented into the ‘Healthy Lifestyle’ course and the 9th-grade biology curriculum in Armenia. This choice supports its application and adaptability and represents an important case study in implementation and policy. Immune Patrol has also been translated into a number of additional languages by local WHO offices, to ensure that the materials are presented in culturally appropriate terms, while preserving the scientific content. Lessons can also be accessed in English, e.g., as part of language education (Brown et al., 2010). This process is typically buttressed by classroom testing and engagement with teachers or government, depending on local norms and educational policy. To this end, Immune Patrol was designed to combine matter that can fit into national subjects like biology, language arts, technological literacy, and civics in different ways, allowing teachers to fit all six lessons, or only a selected portion of them, into their teaching schedules depending on year and lesson plan. The positive response to Immune Patrol in participating countries, as well as the documentation of intended learning outcomes, is encouraging to the prospect of scaling the offering to reach across the WHO European Region and beyond. Future research should delve into its enduring impact on student(s) future behavior and attitudes, particularly toward vaccination, and assess the platform’s efficacy and applicability in diverse learning environments. A key to future research will be exploring implementation barriers across different regions, ranging from technological infrastructure, like hardware availability and Wi-Fi quality, to educational elements such as teacher training, school culture, and curriculum placement strategies.

Observed differences between classrooms, and the nature of observed student inquiries and stumbling blocks, also highlight the need for specialized teacher training, or very user-friendly introductory packages aimed at schools and teachers alike. A guide to address frequent student questions developed in Armenia is a step toward enhancing teacher efficacy in providing scientifically accurate information. Our study may thus serve as an example of how locally tailored teacher support, culturally sensitive adaptation and resource development within countries is likely to help the effectiveness of global education offerings even more broadly. The process also highlights how researchers and officials in large-scale cross-country comparisons should be prepared to adapt to variations in implementation and data collection that will invariably occur depending on local organization and partnerships. A combination of transparency and adaptability are key to combining the scientific and practical aspects of international large-scale health work, respectively.

Methodologically, our findings also highlight the need for properly matched control conditions in studies extolling new learning technologies with high international stakes. Indeed, the impact of game elements might easily be conflated with the other design elements like technology, aesthetics, usability, and the quality of the overall learning design, not to mention novelty effects. Likewise, “learning” when conceived as short-term academic memory is often an inadequate outcome for making an impact in community health. When health education is aimed at reaching beyond school walls, personal motivation is more than paramount: Without identified regulation, knowledge about biology, source criticism and preventable diseases is unlikely to make a difference at all. The practical impact (if any) of the relationship between learning and motivation as seeds for future health behaviors, however, needs to be observed in the long term.

Finally, the present paper is only able to document short-term outcomes. The national adaptations now set in motion, might allow for comparative tracking of, e.g., knowledge, attitudes and even HPV-vaccination acceptance in the age group. Testing Immune Patrol as a soft behavioral intervention in direct sync with national vaccination schedules, involving families more directly over time, or re-using recognizable Immune Patrol imagery in campaigns aimed at cohorts who were taught with the curriculum, would also be interesting to investigate the more direct behavioral potential in public health.

8 Concluding remarks

Our research underscores the transformative potential of global digital game-based initiatives such as Immune Patrol in health education. Digital school interventions offer an effective approach to integrating complex health topics into teaching practices and curricula. While not entirely plug-and-play, the positive response and significant learning outcomes, reinforce the value of scaling up trustworthy educational initiatives by organizations like the World Health Organization, across countries where the technology infrastructure is increasingly in place. This study thus serves as a crucial step toward defining how global health education initiatives can be effectively offered in schools across the world, whenever no curriculum on infectious diseases, immunization, vaccination and source criticism is available.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors without undue reservation.

Ethics statement

The studies involving humans were approved by World Health Organisation (WHO) European Office. The studies were conducted in accordance with the local legislation and institutional requirements. Written informed consent for participation in this study was provided by the participants’ legal guardians/next of kin.

Author contributions

AL: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Resources, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing. SN: Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Writing – original draft, Writing – review & editing. CR-d-K: Conceptualization, Investigation, Methodology, Resources, Supervision, Writing – review & editing, Project administration. CR: Data curation, Formal analysis, Investigation, Methodology, Validation, Writing – original draft, Writing – review & editing. LN: Investigation, Project administration, Writing – original draft, Writing – review & editing, Funding acquisition, Supervision. PP: Data curation, Investigation, Project administration, Writing – original draft, Writing – review & editing, Formal analysis, Validation. BC: Conceptualization, Data curation, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing – original draft, Writing – review & editing. OK: Formal analysis, Project administration, Validation, Visualization, Writing – original draft, Writing – review & editing. DD: Data curation, Investigation, Project administration, Supervision, Writing – review & editing. AP: Project administration, Resources, Supervision, Writing – review & editing. VG: Project administration, Resources, Supervision, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. Development and piloting of Immune Patrol was partially funded by Gavi, the Vaccine Alliance. WHO consultancy contracts were provided to AL and OK, and research time was partially provided by Aarhus University. All research activities were funded by the World Health Organization (WHO).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Author disclaimer

The authors affiliated with the World Health Organization (WHO) are alone responsible for the views expressed in this publication and they do not necessarily represent the decisions or policies of the WHO.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2025.1472018/full#supplementary-material

References

Abderbwih, E., Mahanani, M. R., Deckert, A., Antia, K., Agbaria, N., and Dambach, P. (2022). The impact of school-based nutrition interventions on parents and other family members: a systematic literature review. Nutrients 14:2399. doi: 10.3390/nu14122399

Crossref Full Text | Google Scholar

Arnold, I. (2011). John Hattie: visible learning: a synthesis of over 800 meta-analyses relating to achievement: Routledge, Abingdon, 2008, 392 pp, ISBN 978-0-415-47618-8 (pbk). Int. Rev. Educ. 57, 219–221. doi: 10.1007/s11159-011-9198-8

Crossref Full Text | Google Scholar

Berniell, L., De La Mata, D., and Valdés, N. (2013). Spillovers of health education at school on parents’ physical activity. Health Econ. 22, 1004–1020. doi: 10.1002/hec.2958

PubMed Abstract | Crossref Full Text | Google Scholar

Bressler, D. M., Bodzin, A. M., and Tutwiler, M. S. (2019). Engaging middle school students in scientific practice with a collaborative mobile game. Comput. Assist. Learn. 35, 197–207. doi: 10.1111/jcal.12321

Crossref Full Text | Google Scholar

Brown, K. F., Kroll, J. S., Hudson, M. J., Ramsay, M., Green, J., and Long, S. J. (2010). Factors underlying parental decisions about combination childhood vaccinations including MMR: a systematic review. Vaccine 28, 4235–4248. doi: 10.1016/j.vaccine.2010.04.052

PubMed Abstract | Crossref Full Text | Google Scholar

Carlos, V., Rodrigues, M., Matos, B., Gonçalves, L., Ribeiro, F., and Fardilha, M. (2023). Engaging large classes of higher education students: a combination of spaced learning and team-based learning. Front. Educ. 8:1129763. doi: 10.3389/feduc.2023.1129763

Crossref Full Text | Google Scholar

Carrieri, V., Guthmuller, S., and Wübker, A. (2023). Trust and COVID-19 vaccine hesitancy. Sci. Rep. 13:9245. doi: 10.1038/s41598-023-35974-z

PubMed Abstract | Crossref Full Text | Google Scholar

Chan, S., and Lo, N. (2024). Enhancing efl/esl instruction through gamification: a comprehensive review of empirical evidence. Front. Educ. 9:1395155. doi: 10.3389/feduc.2024.1395155

Crossref Full Text | Google Scholar

Cohen, J. (1992). Statistical power analysis. Curr. Dir. Psychol. Sci. 1, 98–101. doi: 10.1111/1467-8721.ep10768783

Crossref Full Text | Google Scholar

Creswell, J. W. (2007). “Philosophical, paradigm, and interpretive frameworks” in Qualitative inquiry & research design. Choosing among five approaches. 2ned ed (Thousand Oaks, CA: SAGE Publications), 15–34.

Google Scholar

Cromley, J. G., and Byrnes, J. P. (2012). Instruction and cognition. WIRES Cogn. Sci. 3, 545–553. doi: 10.1002/wcs.1192

PubMed Abstract | Crossref Full Text | Google Scholar

Dahalan, F., Alias, N., and Shaharom, M. S. N. (2023). Gamification and game based learning for vocational education and training: a systematic literature review. Educ. Inf. Technol. 29, 1279–1317 doi: 10.1007/s10639-022-11548-w

Crossref Full Text | Google Scholar

Epstein, D. S., Enticott, J. C., Larson, H. J., and Barton, C. (2021). Recruiting for research on sensitive topics in schools: an experience with Vaxcards, a collectable vaccine card game. Trials 22:320. doi: 10.1186/s13063-021-05288-4

PubMed Abstract | Crossref Full Text | Google Scholar

European Centre for Disease Prevention and Control (2021). Countering online vaccine misinformation in the EU/EEA. [internet]. LU: Publications Office.

Google Scholar

Fadda, M., Galimberti, E., Fiordelli, M., and Schulz, P. J. (2018). Evaluation of a mobile phone–based intervention to increase parents’ knowledge about the measles-mumps-rubella vaccination and their psychological empowerment: mixed-method approach. JMIR Mhealth Uhealth 6:e59. doi: 10.2196/mhealth.8263

PubMed Abstract | Crossref Full Text | Google Scholar

Fenta, E. T., Tiruneh, M. G., Delie, A. M., Kidie, A. A., Ayal, B. G., and Limenh, L. W. (2023). Health literacy and COVID-19 vaccine acceptance worldwide: a systematic review. SAGE Open Med. 11:20503121231197869. doi: 10.1177/20503121231197869

PubMed Abstract | Crossref Full Text | Google Scholar

Greene, J. C. (2007). Mixed methods in social inquiry. 1st Edn. San Francisco, CA: Jossey-Bass, 216.

Google Scholar

Gris, G., and Bengtson, C. (2021). Assessment measures in game-based learning research: a systematic review. IJSG 8, 3–26. doi: 10.17083/ijsg.v8i1.383

Crossref Full Text | Google Scholar

Gulesci, S., Meyersson, E., and Trommlerová, S. K. (2020). The effect of compulsory schooling expansion on mothers’ attitudes toward domestic violence in Turkey. World Bank Econ. Rev. 34, 464–484. doi: 10.1093/wber/lhy021

Crossref Full Text | Google Scholar

Hakim, H., Driedger, S. M., Gagnon, D., Chevrier, J., Roch, G., and Dubé, E. (2024). Digital gamification tools to enhance vaccine uptake: scoping review. JMIR Serious Games 12:e47257. doi: 10.2196/47257

PubMed Abstract | Crossref Full Text | Google Scholar

Hamari, J., Koivisto, J., and Sarsa, H. (2014). “Does gamification work? -- a literature review of empirical studies on gamification” in 2014 47th Hawaii international conference on system sciences [internet] (Waikoloa, HI: IEEE), 3025–3034.

Google Scholar

Hanghøj, T., Lieberoth, A., and Misfeldt, M. (2018). Can cooperative video games encourage social and motivational inclusion of at-risk students? Br. J. Educ. Technol. 49, 775–799. doi: 10.1111/bjet.12642

Crossref Full Text | Google Scholar

Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. London; New York: Routledge, 378.

Google Scholar

Hussein, M. H., Ow, S. H., Cheong, L. S., Thong, M. K., and Ale Ebrahim, N. (2019). Effects of digital game-based learning on elementary science learning: a systematic review. IEEE Access 7, 62465–62478. doi: 10.1109/ACCESS.2019.2916324

Crossref Full Text | Google Scholar

Ifenthaler, D., and Schweinbenz, V. (2013). The acceptance of tablet-PCs in classroom instruction: the teachers’ perspectives. Comput. Hum. Behav. 29, 525–534. doi: 10.1016/j.chb.2012.11.004

Crossref Full Text | Google Scholar

Jääskä, E., and Aaltonen, K. (2022). Teachers’ experiences of using game-based learning methods in project management higher education. Project Leadersh. Soc. 3:100041. doi: 10.1016/j.plas.2022.100041

PubMed Abstract | Crossref Full Text | Google Scholar

Jiménez, J. L., and Perdiguero, J. (2019). “Difference-in-difference” in Encyclopedia of law and economics [internet]. eds. A. Marciano and G. B. Ramello (New York, NY: Springer New York), 551–555.

Google Scholar

Kaufman, J., Ames, H., Bosch-Capblanch, X., Cartier, Y., Cliff, J., and Glenton, C. (2017). The comprehensive ‘communicate to vaccinate’ taxonomy of communication interventions for childhood vaccination in routine and campaign contexts. BMC Public Health 17:423. doi: 10.1186/s12889-017-4320-x

PubMed Abstract | Crossref Full Text | Google Scholar

Kaufman, J., Tuckerman, J., Bonner, C., Durrheim, D. N., Costa, D., and Trevena, L. (2021). Parent-level barriers to uptake of childhood vaccination: a global overview of systematic reviews. BMJ Glob. Health 6:e006860. doi: 10.1136/bmjgh-2021-006860

PubMed Abstract | Crossref Full Text | Google Scholar

Kaur, G., Danovaro-Holliday, M. C., Mwinnyaa, G., Gacic-Dobo, M., Francis, L., and Grevendonk, J. (2023). Routine vaccination coverage — worldwide, 2022. MMWR Morb. Mortal Wkly. Rep. 72, 1155–1161. doi: 10.15585/mmwr.mm7243a1

PubMed Abstract | Crossref Full Text | Google Scholar

La Torre, G., D’Egidio, V., Sestili, C., Cocchiara, R. A., Cianfanelli, S., Di Bella, O., et al. (2020). Immunizziamo: a school-based field trial to teach new generations the importance of vaccination through games and to fight vaccine hesitancy in Italy. Vaccine 8:280.

Google Scholar

Lee, S., and Lee, D. K. (2018). What is the proper way to apply the multiple comparison test? Korean J. Anesthesiol. 71, 353–360. doi: 10.4097/kja.d.18.00242

PubMed Abstract | Crossref Full Text | Google Scholar

Leonardelli, M., Mele, F., Marrone, M., Germinario, C. A., Tafuri, S., and Moscara, L. (2023). The effects of the COVID-19 pandemic on vaccination hesitancy: a viewpoint. Vaccine 11:1191. doi: 10.3390/vaccines11071191

Crossref Full Text | Google Scholar

Lieberoth, A. (2015). Shallow gamification: testing psychological effects of framing an activity as a game. Games Cult. 10, 229–248. doi: 10.1177/1555412014559978

Crossref Full Text | Google Scholar

Lieberoth, A., Hanhøj, T., and Misfeldt, M.. (2018). Same game, different impact diagnosing the successes and failures of one game-based intervention across four schools. Available online at: https://www.researchgate.net/publication/339738726_Same_game_different_impact_diagnosing_the_successes_and_failures_of_one_game-based_intervention_across_four_schools

Google Scholar

Mäkelä, P. (2000). Vaccines, coming of age after 200 years. FEMS Microbiol. Rev. 24, 9–20. doi: 10.1016/S0168-6445(99)00025-X

PubMed Abstract | Crossref Full Text | Google Scholar

Martindale, T. B., Amankwatia, T., Cifuentes, L. A., and Piña, A. (2024). Handbook of research in online learning: insights and advances. Leiden, NL: Brill.

Google Scholar

Miller, D. N., Gilman, R., and Martens, M. P. (2008). Wellness promotion in the schools: enhancing students’ mental and physical health. Psychol. Schs. 45, 5–15. doi: 10.1002/pits.20274

Crossref Full Text | Google Scholar

Molin, G. (2017). “The role of the teacher in game-based learning: a review and outlook” in Serious games and edutainment applications [internet]. eds. M. Ma and A. Oikonomou (Cham: Springer International Publishing), 649–674.

Google Scholar

Montagni, I., Mabchour, I., and Tzourio, C. (2020). Digital gamification to enhance vaccine knowledge and uptake: scoping review. JMIR Serious Games 8:e16983. doi: 10.2196/16983

PubMed Abstract | Crossref Full Text | Google Scholar

Nadi-Ravandi, S., and Batooli, Z. (2022). Gamification in education: a scientometric, content and co-occurrence analysis of systematic review and meta-analysis articles. Educ. Inf. Technol. 27, 10207–10238. doi: 10.1007/s10639-022-11048-x

PubMed Abstract | Crossref Full Text | Google Scholar

Nielsen, K., and Tanggaard, L. (2019). Pædagogisk psykologi: en grundbog. 2nd Edn. Frederiksberg: Samfundslitteratur.

Google Scholar

Noroozi, O., Dehghanzadeh, H., and Talaee, E. (2020). A systematic review on the impacts of game-based learning on argumentation skills. Entertain. Comput. 35:100369.

Google Scholar

Nuwarda, R. F., Ramzan, I., Weekes, L., and Kayser, V. (2022). Vaccine hesitancy: contemporary issues and historical background. Vaccine 10:1595.

Google Scholar

Ohannessian, R., Yaghobian, S., Verger, P., and Vanhems, P. (2016). A systematic review of serious video games used for vaccination. Vaccine 34, 4478–4483. doi: 10.1016/j.vaccine.2016.07.048

PubMed Abstract | Crossref Full Text | Google Scholar

Olson, O., Berry, C., and Kumar, N. (2020). Addressing parental vaccine hesitancy towards childhood vaccines in the United States: a systematic literature review of communication interventions and strategies. Vaccine 8:590. doi: 10.3390/vaccines8040590

PubMed Abstract | Crossref Full Text | Google Scholar

Pannekoek, L., Piek, J. P., and Hagger, M. S. (2014). The children’s perceived locus of causality scale for physical education. J. Teach Phys. Edu. 33, 162–185.

Google Scholar

Perrotta, C., Featherstone, G., Aston, H., and Houghton, E. (2013). Game-based learning: latest evidence and future directions [Internet]. s. 49. Available online at: https://ocw.metu.edu.tr/pluginfile.php/10919/mod_resource/content/1/GAME01.pdf

Google Scholar

Piaget, J., and Inhelder, B. (2000). The psychology of the child. New York: Basic Books.

Google Scholar

Proios, H., Baskini, M., Keramydas, C., Pourliaka, T., and Tsakpounidou, K. (2022). Can we learn from our children about stroke? Effectiveness of a school-based educational programme in Greece. J. Stroke Cerebrovasc. Dis. 31:106544. doi: 10.1016/j.jstrokecerebrovasdis.2022.106544

PubMed Abstract | Crossref Full Text | Google Scholar

Ruiz-López, T., Sen, S., Jakobsen, E., Tropé, A., Castle, P. E., Hansen, B. T., et al. (2019). FightHPV: design and evaluation of a mobile game to raise awareness about human papillomavirus and nudge people to take action against cervical cancer. JMIR Serious Games 7:e8540. doi: 10.2196/games.8540

PubMed Abstract | Crossref Full Text | Google Scholar

Ryan, R. M., Rigby, C. S., and Przybylski, A. (2006). The motivational pull of video games: a self-determination theory approach. Motiv. Emot. 30, 344–360. doi: 10.1007/s11031-006-9051-8

Crossref Full Text | Google Scholar

Sabater, A., Graham, E., and Marshall, A. (2020). Does having highly educated adult children reduce mortality risks for parents with low educational attainment in Europe? Ageing Soc. 40, 2635–2670. doi: 10.1017/S0144686X19000795

Crossref Full Text | Google Scholar

Schools for Health in Europe (SHE). (2020). SHE monitoring report 2020: Overall report of the SHE member countries [internet]. s. 7. Available online at: https://www.schoolsforhealth.org/sites/default/files/editor/mapping/monitor-report-overall-2020.pdf

Google Scholar

Schoonenboom, J., and Johnson, R. B. (2017). How to construct a mixed methods research design. Kolner Z Soz Sozpsychol. 69, 107–131. doi: 10.1007/s11577-017-0454-1

PubMed Abstract | Crossref Full Text | Google Scholar

Smith-Greenaway, E., Brauner-Otto, S., and Axinn, W. (2018). Offspring education and parental mortality: evidence from South Asia. Soc. Sci. Res. 76, 157–168. doi: 10.1016/j.ssresearch.2018.07.001

PubMed Abstract | Crossref Full Text | Google Scholar

Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., and Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: a second-order meta-analysis and validation study. Rev. Educ. Res. 81, 4–28. doi: 10.3102/0034654310393361

Crossref Full Text | Google Scholar

Van den Broucke, S. (2014). Health literacy: a critical concept for public health. Arch. Public Health 72:10. doi: 10.1186/2049-3258-72-10

PubMed Abstract | Crossref Full Text | Google Scholar

Wang, L. H., Chen, B., Hwang, G. J., Guan, J. Q., and Wang, Y. Q. (2022). Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis. IJ STEM Ed. 9:26. doi: 10.1186/s40594-022-00344-0

Crossref Full Text | Google Scholar

Wang, A. I., and Lieberoth, A. (2016). The effect of points and audio on concentration, engagement, enjoyment, learning, motivation, and classroom dynamics using Kahoot! Available online at: https://www.researchgate.net/publication/309292067_The_effect_of_points_and_audio_on_concentration_engagement_enjoyment_learning_motivation_and_classroom_dynamics_using_Kahoot

Google Scholar

World Health Organization. (2019). Thirteenth general programme of work, 2019–2023: promote health, keep the world safe, serve the vulnerable [Internet]. s. 60. Available online at: https://www.who.int/publications/i/item/thirteenth-general-programme-of-work-2019-2023

Google Scholar

Wouters, P., Van Nimwegen, C., Van Oostendorp, H., and Van Der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. J. Educ. Psychol. 105, 249–265. doi: 10.1037/a0031311

Crossref Full Text | Google Scholar

Wright, B. D., and Linacra, J. M. (1994). Reasonable mean-square fit values. Rasch Meas. Trans. 2, 370–371.

Google Scholar

Xu, M., Luo, Y., Zhang, Y., Xia, R., Qian, H., and Zou, X. (2023). Game-based learning in medical education. Front. Public Health 11:1113682. doi: 10.3389/fpubh.2023.1113682

PubMed Abstract | Crossref Full Text | Google Scholar

Yu, Z., Xu, W., and Sukjairungwattana, P. (2022). Meta-analyses of differences in blended and traditional learning outcomes and students’ attitudes. Front. Psychol. 13:926947. doi: 10.3389/fpsyg.2022.926947

PubMed Abstract | Crossref Full Text | Google Scholar