The COVID-19 pandemic has led to school closures around the world. As schools shifted to distance learning in early 2020, teachers were forced to swiftly revise their teaching methods (Hall et al., 2020; Lorente et al., 2020; Mohan et al., 2020). The rapid transition has caused widespread concern about compromised learning and mental health in children (UNICEF, 2020). Indeed, initial reports have suggested learning decrements Engzell et al. (2020), Kuhfeld et al. (2020) and adverse mental health effects (Bignardi et al., 2020; Pearcey et al., 2020; Wright et al., 2020) in primary school students, compared to preceding years. In many cases, the shift to distance learning also reinforced existing inequalities: students from disadvantaged backgrounds were found to suffer greater learning setbacks than their more advantaged peers (Alvi and Gupta, 2020; Sevilla et al., 2020; Dietrich et al., 2020; Doyle, 2020; Engzell et al., 2020), and there were reports of disparities in access to resources and preparedness of instruction materials required for distance learning between educational levels and schools (Bol, 2020; Mohan et al., 2020; van de Werfhorst et al., 2020).

While many studies have reported on pandemic-related learning losses in primary education, relatively little is known about the impact of the pandemic on older students. Surveys of secondary school students have suggested that these students spent significantly less time on school work during lockdown (Anger et al., 2020; Grätz and Lipps, 2021), and had difficulty concentrating at home (de Haas et al., 2020). In higher education, preliminary results point to a similar drop in motivation and effort, but indicate that academic performance was unaffected or even somewhat improved (Gonzalez et al., 2020; Jacques et al., 2020; Meeter et al., 2020). These findings suggest that learning losses may vary with student age—perhaps as a function of students’ developing ability to engage in self-regulated learning (Paris and Newman, 1990).

One factor that likely contributed to variation in the extent to which students were affected by school closures is the availability of digital educational materials and online learning tools (Ferri et al., 2020; Hall et al., 2020; Klapproth et al., 2020). German secondary school students reported spending more time on school work if their school offered more digital learning materials (Anger et al., 2020). A study of French university students showed that, in a course redesigned around appropriate online tools, distance learning achieved similar learning outcomes to a course taught in person (Jacques et al., 2020). Offering online adaptive practice software as part of the curriculum could even benefit students, as a study of Dutch primary schools showed that including such software in mathematics education led to students performing better, not worse, than they would otherwise have (Meeter, 2021). Dutch educators surveyed before and during the lockdown reported seeing the benefits of digital learning tools, remarking on their efficiency and ability to offer a personalised learning experience in particular (van der Spoel et al., 2020). The development of suitable digital materials and learning tools may thus mitigate (some of) the impact of school closures on learning.

To explore this further, we studied the use of online learning tools in secondary education in the Netherlands during the spring 2020 school closures. Educational systems—and the effects of the pandemic on these systems—differ substantially between countries (e.g., Crawford et al., 2020; Loima, 2020; Toquero, 2020; Assunção Flores and Gago, 2020; Wu, 2021; Watermeyer et al., 2021). Robust national-level empirical studies are therefore essential in understanding the impact of COVID-19 in the unique context of each country. The Netherlands was relatively well-prepared for distance learning in terms of technological infrastructure (Engzell et al., 2020); the regular curriculum already included digital components, and, by-and-large, students had access to the necessary tools at home (de Haas et al., 2020; Bol, 2020). Nevertheless, the sudden lockdown meant that schools and teachers had little time to prepare for the switch to fully-online teaching. In many cases, scheduled lesson hours were reduced or shortened, and more emphasis was placed on digital components of the curriculum (Voogel, 2020). Dutch teachers reported having to quickly reevaluate and adapt their teaching methods, which was particularly challenging for those less familiar with educational technology (van der Spoel et al., 2020). Furthermore, a survey of students’ parents indicates that, in secondary education, school involvement differed between educational levels: compared to students in the pre-vocational track (vmbo), students in the higher pre-university track (vwo) were more likely to have digital classes, a structured lesson programme, and schoolwork checks (Bol, 2020).

The online learning tool that we studied enables self-regulated, autonomous rehearsal of foreign-language vocabulary through retrieval practice, a study method requiring learners to actively recall information (Roediger and Butler, 2011). The tool was accessible to a large number of students in different year groups (12–16 years old) and educational tracks (pre-vocational, general secondary, and pre-university), and was already widely used before the lockdown. This enabled us to compare usage and performance during the school closure period to measurements from earlier in the same school year and from the same period in the year before. As such, the data collected from this tool provides valuable insight into the effects of lockdown on Dutch secondary students’ foreign-language learning.

In this paper, we use the collected data to address two research questions:

1. Did usage of the retrieval practice tool change during the lockdown period?

The analyses that we report are limited to the domain of the online learning tool, which means that other components of the curriculum and learning environment are left out of consideration. Nevertheless, these results, when combined with other studies addressing the impact of lockdown on other facets of education, bring us closer to a complete picture of the effects of distance learning on education.

We recorded the online retrieval practice activity of a large sample of secondary education students in the Netherlands during two consecutive school years (18/19 and 19/20; both school years lasting from 1 August to 31 July). The sample includes students from each of the three educational tracks in Dutch secondary education—pre-vocational (vmbo), general secondary (havo) and pre-university (vwo)—and from year groups 1 (age: 12; corresponds to grade 7) through 4 (age: 16; corresponds to grade 10).

Students in the sample used SlimStampen, an online, adaptive retrieval practice tool made available to them through the educational publisher Noordhoff Uitgevers as part of the foreign language learning curriculum for English and French. Retrieval practice is a well-established and effective study method in which learners are prompted to actively recall the learning material (Roediger and Butler, 2011; Rowland, 2014). The workings of the retrieval practice tool used here are described in detail in Sense et al. (2016); van der Velde et al. (2021); van Rijn et al. (2009). The tool enabled students to rehearse the course material through retrieval practice sessions, both upon their teacher’s instruction and of their own volition. Sessions consisted of a sequence of trials in which students rehearsed a set of foreign vocabulary items by answering retrieval prompts in various formats. For example, a student practising French vocabulary might see the Dutch prompt waarom (English: why) and be asked to retrieve and type its French translation pourquoi. Data recorded in each trial included the response accuracy and response time, as well as information about the question format, the studied item, and the corresponding textbook chapter.

Approval to analyse the anonymised activity data was granted by the Ethics Committee Psychology of the University of Groningen (study code: PSY-1920-S-0397).

Over the course of two school years and across both courses, a total of 133,450 students completed 115,232,555 trials (18/19: 52,917,284 trials; 19/20: 62,315,271 trials). Supplementary Table 1 gives a breakdown of the data in the sample by course and school year.

We addressed the two research questions outlined in the Introduction by first looking at usage of the retrieval practice tool over time, and then investigating how study performance changed during lockdown.

All analyses were conducted in R (version 3.6.3; R Core Team, 2020). Regression models were fitted using the lme4 package version 1.1–21; Bates et al. (2015) and the lmerTest package (version 3.1–0; Kuznetsova et al., 2017). The analysis code is available at https://osf.io/t25fe/. Additional figures and tables are available in the supplement to this article. Wherever analyses are split by year group and educational track, we only show the French results in the main article. Results for English-language study are included in the supplement.

Our results show that the shift to distance learning during the COVID-19 lockdown of spring 2020 coincided with an increase in usage of an online retrieval practice tool by students in Dutch secondary education, with activity increasing on weekday mornings in particular. In general, we found little evidence of major study delays as a result of the lockdown; students in the highest educational track were likely to be on or even somewhat ahead of the previous year’s schedule, and students in lower tracks were also roughly on schedule. Students’ trial-to-trial learning performance was, if anything, better during the lockdown period than the year before. In particular, we saw higher accuracy and higher response times on open answer questions, perhaps indicating that students experienced less time pressure or were better able to focus at home.

The online learning data reported here offer a detailed look at day-to-day changes in behaviour of a large and diverse group of students. Of course, observing students’ activity in a single online learning tool provides only a limited view into their learning, and students who do not use the tool (anymore) fly completely under the radar. A more complete analysis of the effects of lockdown should also consider other components of the curriculum and students’ environment. As previous work has shown, there are likely to be differences among school subjects, students, and schools in how much learning fell behind during lockdown (Engzell et al., 2020; Lek et al., 2020; Maldonado and Witte, 2020; Meeter, 2021). Identifying the causes of such differences can be helpful in effectively combating the negative consequences of lockdown. Analyses like the current one contribute to this goal. In addition, it is possible that the initial sudden switch to distance learning caused certain learning behaviours to persist out of inertia. It would therefore be valuable to compare our current findings to data from the second lockdown in the Netherlands, during the winter and spring of the 20/21 school year, at which point schools and students would have had more time to adapt to distance learning.

Because of differences between countries in educational systems, access to education and technology, and responses to the COVID-19 pandemic, it is important to interpret these findings within their national context. Compared to other countries, the Netherlands was relatively well-prepared for distance learning, with good availability of the necessary technological infrastructure for digital education (Engzell et al., 2020; de Haas et al., 2020). Technological hurdles related to the accessibility of digital resources have been much more problematic in developing and newly industrialised countries (Alvi and Gupta, 2020; Owusu-Fordjour et al., 2020; Lorente et al., 2020). Nevertheless, even in the Netherlands access to appropriate digital educational materials and online learning tools was not universal, but was found to vary depending on socio-economic factors (Bol, 2020), teachers’ familiarity and experience with these resources (van der Spoel et al., 2020; Voogel, 2020), and, in secondary education, students’ educational track (Bol, 2020). While our findings are based on a large sample of Dutch secondary school students, they offer only limited insight when it comes to such issues of accessibility. They depend on students actively using a particular online retrieval practice tool, and no further information about the students, teachers, or schools was recorded. We did observe slight differences between educational tracks in pre-lockdown usage of the online retrieval practice tool, but usage increased fairly uniformly during the lockdown. We therefore did not see evidence in our sample of disparities in technological preparedness between educational tracks that Bol (2020) reported.

Where studies have reported decrements in learning during the school closures, these could be caused by both lower quantity of learning, and lower quality. Surveys have already shown that secondary school students, on average, spent much less time on school-related work during the lockdown than they would normally do (Anger et al., 2020; Grätz and Lipps, 2021). In contrast, we found an increase in usage of the retrieval practice tool. However, learning activity increased during what would normally be students’ regular school hours, suggesting that retrieval practice may have replaced other parts of the language-learning curriculum, rather than being added on top of it. The retrieval practice tool we studied addresses the learning of vocabulary, which is only one part of learning a foreign language. It may be that students focused on vocabulary learning, which may be relatively well-suited for distance learning, to the detriment of other aspects of language learning that could not be done as easily from home, in effect leading to a narrowing of the curriculum (Voogel, 2020). Our results do suggest that in the specific case of vocabulary learning, the quality of learning remained constant or improved slightly, as indicated by normal or above-normal progress through the chapters of the textbook and stable or increasing response accuracy.

This work contributes to a growing literature aiming to understand the positive and negative impact of distance learning on students and to identify the factors that may amplify positive effects or mitigate negative effects. The reported findings offer a retrospective view of the impact of lockdown in a specific educational context that, together with similar studies on other aspects of education, can help shape future educational policy (World Bank, 2020). Online learning tools such as the one discussed here have the potential to offer an efficient, personalised learning environment and give immediate and detailed insight into students’ learning. This can be very helpful for teachers and curriculum designers, particularly when regular classroom teaching becomes impossible.

The online retrieval practice tool SlimStampen is licensed to Noordhoff Uitgevers by the University of Groningen.