Long-Term Consequences of Repeated School Closures During the COVID-19 Pandemic for Reading and Mathematics Competencies

Introduction

The worldwide COVID-19 pandemic had a severe impact on formal education. Schools were closed repeatedly to prevent infections and to effectively mitigate the spread of SARS-CoV-2 (Mendez-Brito et al., 2021; see Isphording et al., 2021, for compensatory effects of mandatory virus testing and mask-wearing). School closures disrupted learning fundamentally. The switch to remote teaching and learning was a challenge for teachers, students, and their families. On average, students spent less time learning and reported more distractions (Huber and Helm, 2020). The corresponding learning loss during the first pandemic wave has been summarized by several systematic reviews and meta-analyses (De Witte and Smet, 2021; Donnelly and Patrinos, 2021; Hammerstein et al., 2021; Panagouli et al., 2021; Zierer, 2021). Standardized learning losses were typically smaller in native language domains such as reading than in mathematics (e.g., Zierer, 2021, reported d_2020–pre = –0.10 in reading compared to d_2020–pre = –0.17 in mathematics). Students from disadvantaged backgrounds tended to have the largest learning deficits (Hammerstein et al., 2021). The present study uses data from educational large-scale assessments to extend our understanding of the learning gains and losses associated with school closures during the COVID-19 pandemic.

After the first pandemic wave in spring 2020, most schools resumed in-person learning. However, subsequent pandemic waves followed in most countries (including Germany) in late 2020 and early 2021. Again, school closures were among the instruments to reduce the spread of COVID-19 infections, despite the risk of further learning losses. Remote learning became once more an obstacle for learning that was particularly detrimental for the cognitive and socio-emotional development of students from disadvantaged backgrounds (Werner and Woessmann, 2021). The present study is based on data from Baden-Württemberg where the measures consisted of additional days off (no formal teaching), hybrid learning (e.g., in-person learning during the first half of the week and remote learning during the second half of the week; teaching half of the class in person and half of the class online), and remote learning (e.g., online teaching; self-administered work sheets). There were 5 weeks during which the learning mode depended on the local incidence of infections. This lead to a different number of days with in-person learning between schools depending on their location.

The focus of the present study is on academic achievement in terms of basic reading and mathematical competencies, because these competencies are crucial for learning in other domains and for future educational attainment. Understanding the learning gains and learning losses related to the measures to cope with the pandemic situation in educational institutions can help school management and policy makers weigh the benefits and disadvantages of specific learning modes (e.g., in-person, hybrid, and remote) from pedagogical, economical, and epidemiological perspectives.

The worst-case scenario would be a cumulative learning loss due to repeated school closures. Just as additional years of education tend to improve students’ cognitive abilities (Ritchie and Tucker-Drob, 2018), a prolonged absence of regular schooling can have a negative impact on individuals and society as a whole (Azevedo et al., 2021). More realistically, however, survey studies suggest a qualitative difference between learning during the first pandemic wave in spring 2020 and learning during subsequent waves. In spring 2020, regular in-person learning was disrupted abruptly. Teachers and students were often unprepared to switch to digital devices for communication and learning at that point (Schneider et al., 2021). The ministry of education in Baden-Württemberg distributed the learning management system Moodle to its more than 4,000 schools at short notice in March 2020. Moodle (along with other online learning tools that were introduced eventually) assisted with remote learning but was new to many teachers at first. Thus, it was often challenging for teachers and students to profit from the possibilities of this software before technical issues were solved and in-service training for the software was rolled out. By the time wave two and wave three led to school closures, technical infrastructure and workflows in the (virtual) classroom had improved, fostering remote learning. Teachers in Germany reported that they were better prepared and equipped for remote teaching in late 2020 and early 2021, being able to offer more joint virtual lessons, more assignments, and more feedback (Werner and Woessmann, 2021). When teachers and students were equipped to handle remote and hybrid learning, learning outcomes suffered less from school closures. Extended periods of in-person learning between school lockdowns after the first pandemic wave could have further helped to catch up on missed lessons. German students also reported to have spent more time on average with learning activities during the second school lockdown (4.6 h per workday) than during the first school lockdown (3.7 h per workday). Still, learning time was well below pre-pandemic levels (7.5 h per workday), in particular for low-achieving students (Werner and Woessmann, 2021).

Some studies already addressed the effects of repeated school closures on students’ academic achievement. In a large United States sample, the academic achievement of students in primary and secondary schools declined after each period of school closures (Kuhfeld et al., 2022). For example, fifth-graders had lower test scores in fall 2021 than in fall 2019 (reading d_2021–2019 = –0.15, mathematics d_2021–2019 = –0.25, each year’s n > 450,000; Kuhfeld et al., 2022). Another United States study reports a slight learning rebound in winter 2020 (n > 1.5 million; see Renaissance Learning, 2021). A third study linked reduced pass rates in standardized tests to longer periods of remote learning in some US states than in others with shorter periods of remote learning (Halloran et al., 2021). Educational large-scale assessments from Belgium suggest additional learning losses in reading (about 70% of the initial learning loss in spring 2020), but not in mathematics (each year’s n > 35,000; Gambi and DeWitte, 2021). Dutch data from about a third of the country’s schools suggests that some of the learning losses had been compensated by subsequent in-person classes after two school lockdowns, especially in reading (d_2021–2019 = –0.07, mathematics d_2021–2019 = –0.10; Haelermans et al., 2021; see also Engzell et al., 2021). In Denmark, where primary schools were closed for just 3 weeks during the first wave and for 5 weeks during the second wave, recent test scores in reading and mathematics were actually above pre-pandemic levels (each year’s n > 200,000; Birkelund and Karlson, 2021).

The heterogeneous pattern of initial findings reflects differences regarding the length of school closures and the amount as well as the quality of remote teaching (between regions but also between subjects and grades). Also, each study design of the studies listed above has its strengths and weaknesses. Sample composition and attrition can be a major issue when participation in the tests is voluntary (e.g., Gambi and DeWitte, 2021). In the largest study so far, based on several million United States students, the 2020/21 samples were about 20% smaller than the pre-pandemic 2018/19 samples. In Denmark, whole student population cohorts were tested, but the most recent assessments in 2021 took place about 2 months later than in pre-pandemic school years (Birkelund and Karlson, 2021). This can bias learning loss estimates, because learning gains are not spread equally (i.e., linearly) across the (school) year. Finally, studies using a regression approach with dummy variables that indicate assessments after school closures rely on point estimates of students’ ability (e.g., Birkelund and Karlson, 2021; Gambi and DeWitte, 2021; Haelermans et al., 2021). Depending on the estimation procedure, the variance of point estimates tends to be too large or too small. In order to achieve unbiased variance estimates, plausible values can be used (Wu, 2005).

The present study provides insights from one of the largest educational assessments in Germany that continued basically undisrupted throughout the COVID-19 pandemic. The state-wide tests in Baden-Württemberg were administered every year in September to the population cohort of incoming fifth-graders at the beginning of German secondary school. Because taking part in the test was mandatory for all schools and students, there was no sampling bias and only negligible attrition (e.g., due to illness-related absence of students). There were no differences in the general testing procedure across the years and no other confounders in terms of pandemic restrictions (e.g., no distance learning at the time when the assessment took place). Previous findings from 2020 suggest that students’ competencies were on average lower after the first school lockdown in 2020 compared to the pre-pandemic years 2017–2019 (reading: d_2020–pre = –0.07; mathematical operations: d_2020–pre = –0.09; numbers: d_2020–pre = –0.03; see Schult et al., 2022).

According to the theoretical framework of Huber and Helm (2020), students’ academic achievement is the result of instruction and learning processes, which in turn are influenced by students’ families and peers, their personalities, their teachers and schools as well as system-level settings. Accordingly, we focus on two important context variables at the school level that are known to be related to student achievement, namely prevalence of migration background and socio-cultural school characteristics (e.g., Kempert et al., 2016). At the school-system level, we also investigate the effects of the duration of school closures measured by days without in-person learning (with a similar approach like Halloran et al., 2021). Building on the work of Schult et al. (2022) to further investigate the long-term effects of school closures and their impact on learning outcomes based on standardized assessment data, we aimed to answer the following three research questions. While the first two questions pertain to the level of individual students, the third question pertains to school level data.

RQ1: Do the competencies of incoming fifth-graders in reading and mathematics in September 2021 (i.e., 18 months into the pandemic) show an additive learning loss (after repeated school lockdowns), an attenuated learning loss, or a rebound as compared to September 2020 (i.e., 6 months into the pandemic) and to pre-pandemic levels?

RQ2: Do high-, average-, and low-achieving student groups differ in terms of learning gains and learning losses 18 months into the pandemic?

RQ3: Do student performance levels before and after the school lockdowns relate to school characteristics (i.e., days without in-person learning, prevalence of students with migration background and socio-cultural capital)?

Materials and Methods

Study Design

The present study is based on “Lernstand 5,” a mandatory large-scale assessment that is administered in the German federal state Baden-Württemberg at the beginning of each school year. Local legislation mandates the tests for all incoming fifth-graders in public schools (n > 80,000; see Table 1 for sample details across the years and subject domains). “Lernstand 5” consists of competence tests in German and mathematics that assess competencies that students should have acquired in primary school (i.e., in grades 1–4) and also offers (optional) tailored material for teachers to best support students’ learning (Schult and Wagner, 2021; Schult et al., 2022).

TABLE 1

Table 1. Descriptive statistics for the competence assessment across years.

Teachers administered the standardized paper-and-pencil tests to their classes during the second and third week of the school year in late September. Teachers graded the tests according to standardized instructions and submitted the results to a website that gave them instant feedback regarding the performance of students in their class. Students who were absent on the day of testing (e.g., due to illness) were excluded from the assessment. Participation rates in 2021 (reading: 95.5%, mathematics: 95.4%) were only slightly lower than in pre-pandemic years (>96.0%), but higher than in 2020 (after the first pandemic wave; reading: 94.9%, mathematics: 94.6%; see Table 2).

TABLE 2

Table 2. Psychometric properties of the competence assessments and participation rates.

Third- and fourth-graders in Baden-Württemberg missed a total of 10 weeks of in-person learning due to school closures during the first pandemic wave in spring 2020. During subsequent waves, in-person learning was partially suspended from December 16, 2020 to March 14, 2021 (including five additional days off before Christmas), and again from April 12 until May 21, 2021. In total, there were 7 weeks of remote learning, 3 weeks of hybrid learning, and 5 weeks during which the learning mode was determined by the local infection rates: Classes received hybrid teaching in districts that had moderate infection rates while schools in districts with high infection rates had to switch to 100% remote learning¹. Figure 1 shows the mandated learning mode for each school day of the school year 2020/21 (separately for each of Baden-Württemberg’s 44 districts). On average across all investigated schools, this corresponds to approximately 13 weeks of school absence (i.e., 65 out of 192 school days²).

FIGURE 1

Figure 1. Learning Modes for Each School Day (columns) During the School Year 2020/21 for Grade 4 According to Regulations and Local Incidence Rates in Each of Baden-Württemberg’s 44 Districts (rows).

Instruments and Measures

The “Lernstand 5” Assessment

The reading test consisted of four texts. Students had 50 min to read the texts and answer the items. The items assess reading processes such as retrieving explicitly stated information, making inferences, interpreting ideas, and evaluating content and textual elements (Fischer et al., 2017).

The mathematics test assessed students’ competencies in operations and numbers (each 20 min). The operations domain pertains to applying and combining arithmetic processes (ranging from understanding simple operations in clear situations to multistep operations in problem solving tasks). The numbers domain pertains to competencies that are often formed in grades 1 and 2, already, such as understanding, interpreting, and applying different representations (numeric, verbal, and figural) of natural numbers (up to one million; Schulz et al., 2017; Schult and Lindner, 2019). Psychometric properties of the assessments from 2017 to 2021 such as number of items and reliability estimates are presented in Table 2.

Each test was Rasch scaled (de Ayala, 2009) using item difficulty parameter estimates that were obtained through pilot studies. The pilot studies contained new items for the upcoming tests along with a set of linking items which are kept constant between the years. We normed the first “Lernstand 5” in 2015 to have a mean of 500 and a standard deviation of 100 in each domain. Further details regarding the instruments and their scaling can be found in Schult et al. (2022).

Determination of School Closure

The number of school days without in-person learning was estimated for the schools within each of Baden-Württemberg’s 44 districts (“Landkreise”) based on the federal and state regulations that applied to the district in question (see Figure 1). In periods of hybrid learning, we counted each day as 0.5 days of remote learning to account for the regular alternation between remote learning and learning in school for half of a class, as it was practiced in most of the schools. Additional closures of particular schools depended on instructions from the local health authorities (e.g., in case of local school outbreaks of COVID-19).

Migration Background at the School Level

The definition of migration background in the current study corresponds to having a foreign nationality, having a foreign place of birth, or speaking a different language than German at home. Official school statistics provided the proportion of students with migration background as a school level variable. To obtain an estimate of the migration background prevalence at each school, the proportions of students with migration background for grades 5–7 of the school year 2019/20 were averaged (Cronbach’s alpha = 0.95).

Socio-Cultural Capital at the School Level

The socio-cultural capital at the school level was indicated by the average number of books at students’ homes. This information was given by eighth-graders in March 2019 and March 2020 during the VERA 8 assessment (r_2019,2020 = 0.88; for study details see Schult and Wagner, 2020). Students could choose one of six responses: 1 = “0–10 books,” 2 = “11–25 books,” 3 = “26–100 books,” 4 = “101–200 books,” 5 = “201–500 books,” and 6 = “more than 500 books.”

Data Analysis

First, we estimated each student’s ability. Mean competence scores were then computed for each cohort. To investigate the extent to which students’ academic achievement changed and how it has progressed after 18 months into the pandemic (RQ1), the mean competence scores at the population level (in reading comprehension, operations, and numbers) from 2021 were compared with the pooled score means from 2017–2019 and the score means from 2020. Pooling three recent pre-pandemic years smooths out year-specific fluctuations (due to slight cohort or test differences); this procedure is in line with, for example, Engzell et al. (2021). Score differences were quantified by the effect size Cohen’s d.

Second, we computed the population percentiles at 5, 25, 50, 75, and 95% for each cohort using the individual ability estimates. The percentiles from 2021 were compared with the corresponding percentiles from 2017–2019 and 2020 to evaluate how the high-, average-, and low-achieving student group performance was affected at different times during the pandemic as compared to pre-pandemic levels (RQ2).

Third, the analysis switched to the school level to answer RQ3. For each school, we averaged the students’ ability score estimates for reading comprehension, mathematical operations, and numbers. Schools’ mean ability scores were then correlated with the duration of school closures, the proportion of students with migration background, and students’ average socio-cultural capital. Given the correlations between the school characteristics, we also performed a multivariate analysis: The learning loss at the school level was regressed on the number of days without in-person learning, using migration background, socio-cultural capital, and pre-pandemic achievement as control variables. The dependent variable learning loss was computed by subtracting the schools’ pre-pandemic mean score (2017 through 2019) from the schools’ 2021 mean score. The independent variables have been empirically and theoretically linked to learning slides during the pandemic (e.g., Halloran et al., 2021; Haelermans et al., 2022; Kuhfeld et al., 2022). Pre-pandemic achievement was also used as control variable because initial findings suggested that low-performing student groups were more likely to accumulate learning backlogs during periods of school closure (Engzell et al., 2021). While this model omits other potentially important variables that may have affected students’ learning losses, the analysis provides a glimpse into the interplay of learning loss, school characteristics, and length of school closures.

In order to further investigate the role of school characteristics, schools were split into four equally large groups using the quartiles of migration background and socio-cultural capital, respectively. The average student achievement was calculated within each quartile group. Due to administrative changes and the unclear school track information in the 2017–2019 cohorts k = 174 schools had to be excluded from the school-level analysis, leaving k = 1055 schools for the analyses that were conducted to answer RQ3.

The present analysis uses plausible values (k = 10) as ability estimates in order to obtain unbiased variances (Wu, 2005). Statistical significance testing was not performed, because the data consist of full population cohorts. All analyses were run in R 4.0.4 (R R Core Team, 2021) using the packages eatRep (Weirich et al., 2021), ggplot2 (Wickham, 2016), mitools (Lumley, 2019), plyr (Wickham, 2011), tidyr (Wickham, 2021), and TAM (Robitzsch et al., 2021).

Results

Research Question 1: Mean Cohort Differences (Student-Level Data)

Descriptive statistics are listed in Table 1 for the pre-pandemic comparison group (2017–2019), the 2020 cohort (after the first wave of COVID-19), and the 2021 cohort (after the third wave). The standardized mean differences between the 2021 cohort and the pre-pandemic cohorts were negative for reading (d_2021–pre = –0.02) and for operations (d_2021–pre = –0.13), indicating attenuated achievements 18 months into the pandemic. We found no difference for numbers (d_2021–pre = 0.00). The standardized mean differences between the 2021 cohort and the 2020 cohort were positive for reading (d_2021–2020 = 0.05) and for numbers (d_2021–2020 = 0.03), but negative for operations (d_2021–2020 = –0.03). This corresponds to an average (partial) rebound in reading and numbers as well as to further learning loss in operations across all investigated students (RQ1).

Research Question 2: Differences Depending on Achievement Levels (Student-Level Data)

The 5, 25, 50, 75, and 95% population percentiles for 2017–2019, 2020, and 2021 are shown in Figure 2 (see also Table 3). The median achievements in 2021 and the median pre-pandemic achievements (2017–2019) show a similar pattern like the arithmetic means. For students at the 75% percentiles, we find an indication of a rebound in 2021, whereas students at the 25% percentiles show a slight downward trend. Student at the 95% percentiles (i.e., the points above which the best five percent of students scored) also slightly shifted downward. However, students at the 5% percentiles show a pronounced achievement drop in each of the three domains. This low-achieving group scored between 5 (numbers) and 23 points (operations) less after the school closures during the first three pandemic waves as compared to the low-achieving group in the pre-pandemic cohorts. Low-achieving students’ learning therefore seems to be particularly negatively affected by the pandemic situation and phases of remote learning (RQ2).

FIGURE 2

Figure 2. Five Selected Percentiles for the Cohorts in the Years before the Pandemic (2017–2019), After the First Wave of the Pandemic (2020), and After the Third Wave of the Pandemic (2021).

TABLE 3

Table 3. Selected percentiles for the competence distributions across years in reading and mathematics (see Figure 2).

Research Question 3: Relations Between Learning Loss, Duration of School Closures and School Characteristics (School-Level Data)

On average in Baden-Württemberg, fourth-graders had 65 regular school days without in-person learning in their classes during the school year 2020/21. While some districts had to close schools for up to 71.5 days, other schools were somewhat less affected by pandemic containment measures (with a minimum of 54.5 days without in-person learning; median = 65.5 days). At the school level, the present data show that learning losses after three pandemic waves in 2021 were weakly associated with the number of days without in-person learning (–0.13 ≤ r ≤ –0.07; see Table 4). However, the linear regressions with learning loss across the three tested domains as the dependent variables and days without in-person learning, migration background, socio-cultural capital and pre-pandemic achievement as independent variables show that each day of school closure was associated with an average learning loss of –0.38 and –0.35 unstandardized points in reading and mathematical operations, respectively, and –0.14 unstandardized points in the domain of numbers (RQ3; see Table 5).

TABLE 4

Table 4. Descriptive statistics and correlation matrix for schools’ average achievement scores, achievement change scores δ_{2021−⁣−pre} (2021 compared to the three pre-pandemic years 2017–2019), and school characteristics.

TABLE 5

Table 5. linear regressions of achievement change scores δ_{2021−−pre} (2021 compared to the three pre-pandemic years 2017–2019) on length of school closure, school characteristics, and previous achievement.

Differential learning losses were also weakly associated with the estimated proportion of students with migration background at the school (–0.18 ≤ r ≤ –0.02), and with students’ average socio-cultural capital (0.04 ≤ r ≤ 0.28; see Table 4). Students in schools that are already disadvantaged regarding their social background tended to exhibit larger learning losses in mathematics than their peers in schools with higher average socio-cultural capital (RQ3). The achievement means for each quartile of the background variables are illustrated in Figure 3. The differential learning loss in reading was very small (up to 2 points difference between quartiles) but larger in mathematics (between 6 and 11 points difference between quartiles; see Table 6 for quartile means).

FIGURE 3

Figure 3. Average School Scores for Each Quartile of the Proportion of Students With Migration Background (Left) and for Each Quartile of the Schools’ Average Socio-Cultural Capital (i.e., Number of Books at Home; Right). Means were weighted by school size.

TABLE 6

Table 6. Average school scores for each quartile of the proportion of students with migration background and for each quartile of the schools’ mean socio-cultural capital (see Figure 3).

Discussion

Eighteen months into the COVID-19 pandemic, long-term consequences of repeated school closures for learning start to emerge. The present study analyzed the competencies of population cohorts of incoming fifth-graders in a German federal state. A downward trend appeared after the first pandemic wave and long school closures in summer 2020 (see Schult et al., 2022). One year later, reading competencies had improved slightly but did not reach pre-pandemic levels. Average competencies in the domain of mathematical operations got slightly worse whereas competencies in the domain of numbers returned to pre-pandemic levels (RQ1), despite further periods of remote and hybrid learning. The effect sizes are smaller than the results reported by Kuhfeld et al. (2022) in the respective grades and subjects for very large United States samples.

To put the learning loss into perspective, one needs an estimate of the annual learning gains during a regular school year. While United States data from the last century report annual learning gains of d = 0.40 in reading and d = 0.56 in mathematics (in grade 4; Bloom et al., 2008), more recent German data suggest learning gains of d = 0.61 in reading and d = 0.71 in mathematics (Schult et al., 2022). The extrapolation of learning loss associated with summer break slides and long-term absenteeism provides estimates of how much students would learn if pandemic school closures had similar effects: For grade 4, there would be 37–80% of the learning gains expected for a typical school year in reading and 67–82% in mathematics (see Kuhfeld et al., 2020; Table 5). Combining these values yields expected annual learning losses of –0.08 > d > –0.38 in reading and –0.10 > d > –0.23 in mathematics. The observed learning loss in the present study is much smaller than these pessimistic estimates. Also, the observed learning loss in the second year (2021) was smaller compared to the first year (2020), even reverting the negative trend in the reading and numbers assessment.

The partial rebound of students’ achievement is in line with international findings from spring 2021 (e.g., Gambi and DeWitte, 2021; Haelermans et al., 2021). Both students and teachers had more experience with digital tools for remote learning and teaching after the first period of school closures in spring 2020. The teacher-driven adaptation during the school year 2020/21 helped students acquire basic competencies in reading and mathematics (Hattie, 2021). Additional informal online learning also yielded learning gains (e.g., Spitzer and Musslick, 2021).

Yet, not all students succeeded in adapting to the new circumstances. Technological barriers such as lack of appropriate digital devices and lack of a proper learning environment at home made it difficult to participate in remote learning, especially for disadvantaged and low-achieving students (Huber and Helm, 2020; Werner and Woessmann, 2021). In the present analysis, low-achieving students had the largest learning losses over the course of the pandemic whereas the high-achieving students even exhibited learning gains (i.e., the best 5 percent of students in mathematics; RQ2).

In addition to these findings on the level of individual student data, the present study also provides insights on the school-level. Due to the differential length of school closures in the school year 2020/2021 across Baden-Württemberg, we could show that schools in districts with longer periods of school closures had (on average) larger learning losses than schools in districts with shorter periods of school closures. This result remains robust after controlling for confounding variables such as pre-pandemic achievement level and social-cultural background of students. Furthermore, we found that schools with a large proportion of students with migration background and with lower average socio-cultural capital, respectively, had larger average losses than other schools (RQ3). The correlations were rather weak, suggesting that other factors play a more important role in whether students’ learning was successful during the pandemic. In line with our findings, economically disadvantaged students and students with migration background in Netherlands were shown to have a larger average learning loss than their peers, as well (Haelermans et al., 2022).

Practical Implications

According to our interpretation, the partial rebound in students’ achievements suggests that teachers did a tremendous job during the second pandemic school year, juggling technology, learning, and often additional home schooling in their own family (see Hattie, 2021). Teacher surveys indicate that the majority of teachers increased their engagement in order to cope with the challenges of remote learning during school closures (Hilger et al., 2021; Jelińska and Paradowski, 2021; Trust and Whalen, 2021). Yet, there are other factors that could have been responsible for this finding. The efforts of students and their families likely played a crucial role during school closures. Successful remote learning during the COVID-19 pandemic also depends on students’ motivation and parental involvement (e.g., Weber et al., 2021).

Our data suggest that remote learning succeeded in teaching most students basic competencies in reading and mathematics that are crucial for their subsequent education. The main challenge after school closures is to ensure that low-achieving students and other at-risk students receive adequate support. Tailored coaching and tutoring seems to be necessary to make up for lost learning in groups of disadvantaged students. Educational assessments such as “Lernstand 5” at the beginning of secondary school can help to quickly identify individual students that need additional support.

Due to the pronounced learning loss of low-achieving students, remedial actions aimed at these at-risk students are advisable. However, despite many justified worries regarding the pandemic effects on the young generation, the fear of a lost generation of students appears to be unfounded based on achievement data in our and similar studies (e.g., Haelermans et al., 2021). The partial reversal of the initial downward trends suggests that early projections of substantial economic losses in the future due to school closures may have been too pessimistic (e.g., Azevedo et al., 2021).

Of course, schools are a place for learning not just reading and mathematics. It is probably easier to deal with mathematical problems remotely than it is to deal with socio-emotional problems. Remote learning can be perceived as less communal as it provides less direct communication and less informal conversations and actions. Furthermore, some students were not able or willing to even join online classes remotely (Werner and Woessmann, 2021). Therefore, the development of students’ well-being and social skills remains an important research topic. In light of these survey findings, schools might need to catch up on social events maybe even more urgently than on learning loss.

Strengths and Limitations

The majority of learning loss studies reports regression coefficients based on point estimates (e.g., Engzell et al., 2021; Haelermans et al., 2021), using samples with unclear selection processes due to voluntary participation or online access at school (e.g., Gambi and DeWitte, 2021; Kuhfeld et al., 2022). To augment the evidence for learning losses and learning gains during the pandemic, the present study presents one of the most complete cohort profiles of annual achievement. With an attrition rate of less than 1%, there is little room for selection bias in the study design. Established competence tests yielded reliable and valid achievement estimates (e.g., Schult and Lindner, 2019). Ceiling and bottom effects in the ability estimates were avoided by using plausible values, the same approach as the Programme for International Student Assessment (PISA). School context variables provided additional insights regarding students’ competencies before and during the pandemic. The findings were further validated by the school-level analysis of the relation between learning loss and length of school closure. In-person learning was associated with attenuated learning loss even after controlling for background variables such as socio-cultural capital and pre-pandemic achievement. Still, given that learning modes differed between schools (only) for up to 5 weeks, extrapolating these findings might be inappropriate.

The current analysis is based on the population of all incoming fifth-graders in Baden-Württemberg in Germany. However, the focus on one grade cohort is a limitation of the present study. Studies with additional grades (Haelermans et al., 2021; Kuhfeld et al., 2022) indicate that lower and higher grades differ slightly in terms of learning losses, although there seems to be no clear pattern across grades and subject domains. Due to legal restrictions, the “Lernstand 5” assessment has to be parsimonious with regard to personal information. On the student level, the assessment does not include personal identifiers, individual student context variables, and longitudinal measures. Thus, our data did not allow to identify individual student characteristics that were associated with individual learning gains throughout a school year and/or across years.

While large-scale assessments provide a clear picture of learning outcomes in main subjects, empirical studies of competencies in other subjects are at best scarce. In Germany, physical education and music lessons were, for example, suspended for prolonged periods and took place online and infrequently during the pandemic (Gunzenhauser and Saalbach, 2021). It is therefore possible that learning losses in these domains are substantially larger than in the main subjects.

A final limitation is the lack of information on learning processes as the assessments focus on learning outcomes after 4 years of elementary school. With insights from other studies of teacher and student behavior during the pandemic, conjectures can be made. Metacognitive skills and intrinsic motivation facilitate learning in general and even more in remote online settings (De Witte and Smet, 2021). This might explain why high-performing students suffered almost no learning loss (RQ2). Socially disadvantaged students spent less time learning during school closures and they were less likely to have access to digital learning infrastructure (Werner and Woessmann, 2021). This might explain why schools with less socio-cultural capital suffered larger learning losses (even after controlling for other confounding factors; RQ3). Of course, the pandemic affected more than just education. Pandemic-induced changes in the family environment, economic and health-related worries along with other pandemic circumstances could also have had an effect on learning that does not depend on whether learning takes place in-person or remotely. Again, we can merely provide educated conjectures regarding the processes behind the learning slides. Still, the present findings provide an important piece of knowledge in the theoretical framework of successful learning during the pandemic (see also Huber and Helm, 2020).

Conclusion

While in-person learning apparently remains the most beneficial type of learning in terms of cognitive achievement, our data indicate that teachers and students were better able to compensate for potential negative consequences of school closures during the second and third COVID-19 wave than during the first months of the pandemic in 2020. Overall, the learning losses we found appear to be less severe than initially feared (Azevedo et al., 2021), highlighting the resilience of the school system. However, the negative consequences of school closures seemed to hit disadvantaged and low-achieving student groups particularly hard. With regard to social equity, it seems therefore recommendable to make keeping schools open a priority in the overall pandemic management and to increase efforts to support the at-risk student groups in particular.

Data Availability Statement

The data analyzed in this study is subject to the following licenses/restrictions: Due to the nature of this research, participants of this study did not agree for their data to be shared publicly, so supporting data is not available. Requests to access these datasets should be directed to JS, Johannes.Schult@ibbw.kv.bwl.de.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent from the participants’ legal guardian/next of kin was not required to participate in this study in accordance with the national legislation and the institutional requirements.

Author Contributions

JS conceived the study design and wrote the manuscript. JS and NM performed the data analysis. NM, BF, and ML contributed to the theoretical and empirical framework of the study and provided substantial revisions to the original text. All authors read and approved the final manuscript.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^ The exact regulations changed almost every week. Information from the Baden-Württemberg Ministry of Education regarding the regulations is archived at https://km-bw.de/infoschreiben-archiv-corona. The Federal Statistics Office provides incidence rates from infas 360 and the Robert Koch Institute at https://www.corona-daten-deutschland.de/dataset/infektionen.
2. ^ During the school year 2020/21, schools had 4 days off on days of their choice (“bewegliche Ferientage”). Assuming that schools usually take these days off before Christmas or in February, the estimated length of school absence is reduced to 61 (of 188) school days.

References

Azevedo, J. P., Hasan, A., Goldemberg, D., Geven, K., and Iqbal, S. A. (2021). Simulating the potential impacts of COVID-19 school closures on schooling and learning outcomes: a set of global estimates. World Bank Res. Obs. 36, 1–40. doi: 10.1093/wbro/lkab003

CrossRef Full Text | Google Scholar

Birkelund, J. F., and Karlson, K. B. (2021). No evidence of a major learning slide 14 months into the COVID-19 pandemic in Denmark. SocArXiv 8, 1–34. doi: 10.31235/osf.io/md5zn

PubMed Abstract | CrossRef Full Text | Google Scholar

Bloom, H. S., Hill, C. J., Black, A. R., and Lipsey, M. W. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. J. Res. Educ. Effect. 1, 289–328. doi: 10.1080/19345740802400072

CrossRef Full Text | Google Scholar

de Ayala, R. J. (2009). The Theory and Practice of Item Response Theory. New York:NY: Guilford Press.

Google Scholar

De Witte, K., and Smet, M. (2021). Financing Education in the Context of COVID-19 (EENEE Ad Hoc Report no. 03/2021). Publications Office of the European Union. Available online at https://eenee.eu/wp-content/uploads/2021/06/EENEE_AHQ03_Financing-education-in-the-context-of-COVID-19-1.pdf.

Google Scholar

Donnelly, R., and Patrinos, H. A. (2021). Learning loss during COVID-19: an early systematic review. Prospects 10, 1–9. doi: 10.1007/s11125-021-09582-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Engzell, P., Frey, A., and Verhagen, M. D. (2021). Learning loss due to school closures during the COVID-19 pandemic. Proc. Natl. Acad. Sci. 118:e2022376118. doi: 10.1073/pnas.2022376118

PubMed Abstract | CrossRef Full Text | Google Scholar

Fischer, U., Merz, G., and Wagner, S. (2017). Kompetenzorientierung im Leseverstehensunterricht: Verknüpfung von Diagnose und Förderung in Lernstand 5. Zurich Switzerland: Schweizerisches Institut für Kinder- und Jugendmedien. 1–22. Available online at https://www.leseforum.ch/sysModules/obxLeseforum/Artikel/610/2017_3_Fischer_et_al.pdf

Google Scholar

Gambi, L., and DeWitte, K. (2021). The Resiliency of School Outcomes After the Covid-19 Pandemic: Standardised Test Scores and Inequality One Year After Long Term School Closures (Discussion Paper Series Dps21.12). KU Leuven. Available online at https://lirias.kuleuven.be/retrieve/636001. (accessed November 5, 202).

Google Scholar

Gunzenhauser, C., and Saalbach, H. (2021). Erste Schuljahre im Schatten der Pandemie: Was haben Grundschulkinder erlebt und was kommt nun auf sie zu? [first years of schooling during the pandemic: what did primary school students experience and what will they be facing now?]. Psychol. Erzieh. Unterr. 68, 280–286. doi: 10.2378/peu2021.art18d

CrossRef Full Text | Google Scholar

Haelermans, C., Korthals, R., Jacobs, M., de Leeuw, S., Vermeulen, S., van Vugt, L., et al. (2022). Sharp increase in inequality in education in times of the COVID-19-pandemic. PLoS One 17:e0261114. doi: 10.1371/journal.pone.0261114

PubMed Abstract | CrossRef Full Text | Google Scholar

Haelermans, C., Van der Velden, R., Aarts, B., Bijlsma, I., Jacobs, M., Smeets, C., et al. (2021). Balans Na Anderhalf Jaar: Vertraging Leergroei Het Grootst Voor Rekenen-Wiskunde (NCO Factsheet No. 8). Available online at https://www.nationaalcohortonderzoek.nl/sites/nco/files/media-files/factsheet08_nco_algemeen_0.pdf (accessed November 5, 2021).

Google Scholar

Halloran, C., Jack, R., Okun, J. C., and Oster, E. (2021). Pandemic schooling mode and student test scores: evidence from US states (NBER working paper 29497). Natl. Bureau Econ. Res. 1–29. doi: 10.3386/w29497

PubMed Abstract | CrossRef Full Text | Google Scholar

Hammerstein, S., König, C., Dreisoerner, T., and Frey, A. (2021). Effects of COVID-19-related school closures on student achievement—A systematic review. Front. Psychol. 12:4020. doi: 10.3389/fpsyg.2021.746289

PubMed Abstract | CrossRef Full Text | Google Scholar

Hattie, J. (2021). An Ode to Expertise: What Have We Learnt From Covid and How Can We Apply Our New Learning? Paper Presented at the Victorian education state principals conference. August. Available online at https://www.educationtoday.com.au/news-detail/An-Ode-to-Expertise-5409 (accessed January 20, 2022).

Google Scholar

Hilger, K. J., Scheibe, S., Frenzel, A. C., and Keller, M. M. (2021). Exceptional circumstances: changes in teachers’ work characteristics and well-being during COVID-19 lockdown. Sch. Psychol. 36, 516–532. doi: 10.1037/spq0000457

PubMed Abstract | CrossRef Full Text | Google Scholar

Huber, S. G., and Helm, C. (2020). COVID-19 and schooling: evaluation, assessment and accountability in times of crises—reacting quickly to explore key issues for policy, practice and research with the school barometer. Educ. Assess. Eval. Account. 32, 237–270. doi: 10.1007/s11092-020-09322-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Isphording, I. E., van Ewijk, R., Diederichs, M., and Pestel, N. (2021). Schools under mandatory testing can mitigate the spread of SARS-CoV-2 (Discussion Paper No. 14844). IZA. Available online at https://docs.iza.org/dp14844.pdf. (accessed November 16, 2021). doi: 10.26481/umagsb.2021018

CrossRef Full Text | Google Scholar

Jelińska, M., and Paradowski, M. B. (2021). Teachers’ engagement in and coping with emergency remote instruction during COVID-19-induced school closures: a multinational contextual perspective. Online Learn. 25, 303–328. doi: 10.24059/olj.v25i1.2492

PubMed Abstract | CrossRef Full Text | Google Scholar

Kempert, S., Edele, A., Rauch, D., Wolf, K. M., Paetsch, J., Darsow, A., et al. (2016). “Die Rolle der Sprache für zuwanderungsbezogene Ungleichheiten im Bildungserfolg [the role of language for migration-related inequalities in education],” in Ethnische Ungleichheiten im Bildungsverlauf, Eds. C. Diehl, D. Hunkler, and C. Kristen (New York, NY: Springer), 157–241. doi: 10.1007/978-3-658-04322-3_5

CrossRef Full Text | Google Scholar

Kuhfeld, M., Soland, J., and Lewis, K. (2022). Test Score Patterns Across Three Covid-19-Impacted School Years (EdWorkingPaper: 22-521). Rhode Island RI: Annenberg Institute at Brown University, doi: 10.26300/ga82-6v47

CrossRef Full Text | Google Scholar

Kuhfeld, M., Soland, J., Tarasawa, B., Johnson, A., Ruzek, E., and Liu, J. (2020). Projecting the potential impacts of COVID-19 school closures on academic achievement. Educ. Res. 49, 549–565. doi: 10.3102/0013189X20965918

CrossRef Full Text | Google Scholar

Lumley, T. (2019). Mitools: Tools for multiple imputation of missing data. R package version 2.4. Available online at https://CRAN.R-project.org/package=mitools. (accessed December 14, 2020)

Google Scholar

Mendez-Brito, S., El Bcheraoui, C., and Pozo-Martin, F. (2021). Systematic review of empirical studies comparing the effectiveness of non-pharmaceutical interventions against COVID-19. J. Infect. 83, 281–293. doi: 10.1016/j.jinf.2021.06.018

PubMed Abstract | CrossRef Full Text | Google Scholar

Panagouli, E., Stavridou, A., Savvidi, C., Kourti, A., Psaltopoulou, T., Sergentanis, T. N., et al. (2021). School performance among children and adolescents during COVID-19 pandemic: a systematic review. Children 8:1134. doi: 10.3390/children8121134

PubMed Abstract | CrossRef Full Text | Google Scholar

R Core Team. (2021). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. Available online at https://www.r-project.org (accessed December 14, 2020).

Google Scholar

Renaissance Learning. (2021). How Kids are Performing: Tracking the Midyear Impact of COVID-19 on Reading and Mathematics Achievement (Special Report Series: Winter 2020–2021 Edition). Available online at https://renaissance.widen.net/s/zvq8rnrp9t/r63370. (accessed November 26, 2021).

Google Scholar

Ritchie, S. J., and Tucker-Drob, E. M. (2018). How much does education improve intelligence? Psychol. Sci. 29, 1358–1369. doi: 10.1177/0956797618774253

PubMed Abstract | CrossRef Full Text | Google Scholar

Robitzsch, A., Kiefer, T., and Wu, M. (2021). TAM: Test Analysis Modules. R Package Version 3.7-16. Available online at https://CRAN.R-project.org/package=TAM. (accessed August 17, 2021).

Google Scholar

Schneider, R., Sachse, K. A., Schipolowski, S., and Enke, F. (2021). Teaching in times of COVID-19: The evaluation of distance teaching in elementary and secondary schools in Germany. Front. Educ. 6:282. doi: 10.3389/feduc.2021.702406

CrossRef Full Text | Google Scholar

Schult, J., and Lindner, M. A. (2019). Zur Messgüte von geschlossenen und offenen Antwortformaten in Lernstandserhebungen [psychometric properties of multiple-choice and constructed response formats in proficiency tests]. Psychol. Erzieh. Unterr. 66, 260–272. doi: 10.2378/peu2018.art31d

CrossRef Full Text | Google Scholar

Schult, J., Mahler, N., Fauth, B., and Lindner, M. A. (2022). Did students learn less during the COVID-19 pandemic? reading and mathematics competencies before and after the first pandemic wave. PsyArXiv 1–34. doi: 10.31234/osf.io/pqtgf

CrossRef Full Text | Google Scholar

Schult, J., and Wagner, S. (2020). VERA 8 in Baden-Württemberg 2020 (Beiträge zur Bildungsberichterstattung). IBBW. Available online at https://ibbw.kultus-bw.de/site/pbs-bw-km-root/get/documents_E1653873552/KULTUS.Dachmandant/KULTUS/Dienststellen/ibbw/Systemanalysen/Bildungsberichterstattung/Ergebnisberichte/VERA_8/Ergebnisse_VERA8_2020.pdf. (accessed October 2, 2020).

Google Scholar

Schult, J., and Wagner, S. (2021). Lernstand 5 in Baden-Württemberg 2020 (Beiträge zur Bildungsberichterstattung). IBBW. Available online at https://ibbw-bw.de/site/pbs-bw-km-root/get/documents_E386090764/KULTUS.Dachmandant/KULTUS/Dienststellen/ibbw/Systemanalysen/Bildungsberichterstattung/Ergebnisberichte/Lernstand_5/Ergebnisse_Lernstand%205_2020.pdf. (accessed July 9, 2021).

Google Scholar

Schulz, A., Leuders, T., and Rangel, U. (2017). “Arithmetische Basiskompetenzen am Übergang zu Klasse 5 – eine empirie- und modellgestützte Diagnostik als Grundlage für spezifische Förderentscheidungen [arithmetic competencies at the transition to grade 5 – an empirical and model-based diagnostic as base for specific support decisions],” in Handbuch Rechenschwäche, 3rd Edn, eds A. Fritz, S. Schmidt, and G. Ricken (Weinheim: Beltz), 396–417.

Google Scholar

Spitzer, M. W. H., and Musslick, S. (2021). Academic performance of K-12 students in an online-learning environment for mathematics increased during the shutdown of schools in wake of the COVID-19 pandemic. PLoS One 16:e0255629. doi: 10.1371/journal.pone.0255629

PubMed Abstract | CrossRef Full Text | Google Scholar

Trust, T., and Whalen, J. (2021). Emergency remote teaching with technology during the COVID-19 pandemic: using the whole teacher lens to examine educator’s experiences and insights. Educ. Media Internatl. 58, 145–160. doi: 10.1080/09523987.2021.1930479

CrossRef Full Text | Google Scholar

Weber, C., Helm, C., and Kemethofer, D. (2021). Are social and ethnic reading inequalities increasing during school closures?—The mediating role of parental involvement in distance learning. Front. Educ. 6:737064. doi: 10.3389/feduc.2021.737064

CrossRef Full Text | Google Scholar

Weirich, S., Hecht, M., and Becker, B. (2021). eatRep: educational assessment tools for replication methods. r package version 0.13.7. Available online at https://CRAN.R-project.org/package=eatRep. (accessed August 17, 2021).

Google Scholar

Werner, K., and Woessmann, L. (2021). The legacy of COVID-19 in Education (CESinfo Working Paper No. 9358). Available online at https://www.cesifo.org/DocDL/cesifo1_wp9358.pdf doi: 10.2139/ssrn.3945280

CrossRef Full Text | Google Scholar

Wickham, H. (2011). The split-apply-combine strategy for data analysis. J. Stat. Softw. 40, 1–29. doi: 10.18637/jss.v040.i01

CrossRef Full Text | Google Scholar

Wickham, H. (2016). ggplot2: Elegant Graphics For Data Analysis. New York, NY: Springer, doi: 10.1007/978-3-319-24277-4

CrossRef Full Text | Google Scholar

Wickham, H. (2021). tidyr: Tidy Messy Data. R Package Version 1.1.4. Available online at https://CRAN.R-project.org/package=tidyr (accessed August 17, 2021).

Google Scholar

Wu, M. (2005). The role of plausible values in large-scale surveys. Stud. Educ. Evaluat. 31, 114–128. doi: 10.1016/j.stueduc.2005.05.005

CrossRef Full Text | Google Scholar

Zierer, K. (2021). Effects of pandemic-related school closures on pupils’ performance and learning in selected countries: a rapid review. Educat. Sci. 11, 252. doi: 10.3390/educsci11060252

CrossRef Full Text | Google Scholar