Investigating the impact on learning outcomes through the use of EdTech during COVID-19: Evidence from an RCT in the Punjab province of Pakistan

The objective of the current study is to test the impact of low-tech solutions during COVID-19 school closures on maths, Urdu, and English scores for students in the underdeveloped district of Bahawalnagar, which is situated in the southern part of Punjab in Pakistan. The target population for this study is Grade 8 students attending private schools. Simple random sampling technique is adopted for the selection of participants in the study. We have tested the impact of three interventions, namely: Teaching at the right level (TaRL), Fortnightly assessments (FAS), and Digital teacher training sessions (DTS). Our findings show a significant and positive ‘Intention to Treat’ (ITT) impact on Urdu and English scores of the students in the TaRL treatment group. The students increased their English and Urdu scores by 0.56 SD. However, we found no significant impact of the intervention on maths scores in the TaRL treatment group. Fortnightly assessments and digital teacher training sessions were also found to contribute to higher English scores of the students. However, we found no ITT impact on the maths and Urdu scores for these treatment groups. The Local Average Treatment Effect (LATE) analysis revealed positive and significant improvement in Urdu and English scores of the students in the TaRL treatment group. Key stakeholders whom we interviewed suggested that redesigning the curriculum and incorporating TaRL within this approach could facilitate enhancement in learning outcomes in students in deprived areas. Our findings are important to help inform policymakers on the importance of designing and implementing cost-effective, low-tech solutions to help reduce learning gaps.


Figures
Distribution of treatment groups across 12 schools at baseline 19 Table 2. Details of training sessions 22 Table 3. Distribution of treatment groups in the set of 12 schools at baseline and endline 31 Table 4. Sample descriptives 31 Table 5. Intention to Treat estimates in regression framework (OLS) 40 Table 6. Average Treatment Effect on Treated in IV Regression Framework (2SLS) 42

Executive summary
Globally, the Covid-19 pandemic has deepened the learning disparities between students from disadvantaged backgrounds compared to those from more advantaged groups. The closure of schools due to the pandemic has perpetuated inequalities, given that students from more disadvantaged backgrounds have limited access to the technological devices needed for learning continuity. Surveys carried out during the school closures in Pakistan have found that the situation has created severe gender and wealth gaps in learning (⇡Crawfurd et al., 2021).
The objective of this study was to test the impact of using low-tech devices on the literacy and numeracy outcomes of secondary school students during school closures in Pakistan. Our target population was Grade 8 students attending private schools in Bahawalnagar District. This is an economically deprived area situated in the Southern part of Punjab province. We tested the impact of three interventions, which were administered online on 258 students in 12 schools. These three interventions were 1. Computer-assisted teaching at the right level (TaRL) interventions, 2. Fortnightly assessments (FAS)

Digital teacher training sessions (DTS)
The primary research questions which this study was interested in probing were: RQ1: What were the key issues regarding access, barriers, and usage of EdTech by students during school closures?
RQ2: What were the separate and combined effects of delivering TaRL, fortnightly assessments, and digital training sessions online on student learning outcomes in Urdu, English, and maths? online learning RQ3: What did stakeholders identify as the key barriers relating to access to technology for online learning during the school closures?
Our results show a significant and positive 'Intention to Treat' (ITT) impact on the Urdu and English scores of the students who were part of the computer-assisted TaRL treatment group. These students increased their English and Urdu scores by 0.56 SD. However, we found no significant impact on maths scores in the TaRL treatment group. We also found a positive impact of the fortnightly assessment and digital teacher training session interventions on students' English scores. However, we found no ITT impact on the maths and Urdu scores for these treatment groups. The Local Average Treatment 1. Introduction

Background to the study
Education plays a significant role in the socio-economic development of a country. It builds the capacity of individuals to increase productivity and efficiency, which in turn can put the economy on a path to sustainable development (⇡Ahmad et al., 2014). Within the context of Pakistan, the education system currently experiences two main challenges. The first relates to access, with Pakistan currently hosting the second largest number of out-of-school children. The second relates to poor learning for children even when they are in school (⇡Ahmad et al., 2013).
According to the most recent Annual Status of Education Report (ASER), Pakistan has the second largest number of out-of-school children in the world, with around 22 million learners of school-going age (ages 5-6) out of school. ⇡ASER (2019) also illustrates the problem of low learning outcomes, with many students at the primary education level being unable to read a story comprehensively in Urdu (the national language) and Sindhi or Pashto (provincial local languages). The learning crisis is further exacerbated by gender inequalities, with boys outperforming girls when it comes to literacy and numeracy learning outcomes. ⇡ASER (2019) finds, for example, that just 38% of girls can read words in English, Urdu, Sindhi, or Pashto, compared to 46% of boys.
The outbreak of Covid-19 has further aggravated issues relating to access, learning, and gender inequalities. In March 2020, in an attempt to contain the spread of the virus, the Government of Pakistan implemented a nationwide lockdown. This resulted in schools remaining closed from March 2020 to September 2020. Thereafter, intermittent school closures came into effect between November 2020 and August 2021. The lockdown meant that 46 million school-going children in Pakistan were required to stay at home, with learners from impoverished and underdeveloped areas of Pakistan particularly hard-hit by the school closures.

Purpose of this study
The purpose of the study was to explore what effect a set of EdTech interventions had on improving the learning outcomes of students. It specifically focused on learners living in the economically disadvantaged district of Bahawalnagar in Punjab (in Pakistan). The study was conducted over the period when intermittent school closures occurred due to the Investigating the Impact on Learning Outcomes of EdTech in Punjab, Pakistan Covid-19 pandemic. This study's core research focus had three stated objectives. These were: ■ Explore the types of technologies being used in the disadvantaged area where this study took place, and understand the constraints faced by the education system in using technology in such settings.
■ Investigate what role technology-based solutions could have in aiding the objective of 'personalised learning' during school closures.
■ Assess the impact of the selected interventions on learning outcomes for English, maths, and Urdu for learners in Grade 8 of secondary school.

Context of the study
Our study took place in the District of Bahawalnagar which is located in the southern part of the province of Punjab in Pakistan. Within this district, we selected 12 private schools across five administrative units (known as tehsils) of Bahawalnagar district. These were Haroon-Abad, Fort-Abbas, Minchin-Abad, Bahawalnagar, and Chistian. From each of the schools we worked in, we selected approximately 20 students to take part in the study. The students who were selected came from economically disadvantaged backgrounds where household incomes were low.

What this paper adds to the knowledge base
This research provides insights into what effect technology can have on learning outcomes, especially in the context of Pakistan. This study also highlights the particular barriers students from economically disadvantaged backgrounds face and considers these through the lens of gender. The inferences from the study can assist policymakers when it comes to forward-looking policy reforms which look specifically at the role of EdTech within education systems.

Research questions
The research questions this study addressed were as follows: RQ1: What were the key issues regarding access, barriers, and usage of EdTech by students during school closures?
RQ2: What were the separate and combined effects of delivering TaRL, fortnightly assessments and digital training sessions online on student learning outcomes when it came to Urdu, English and maths? RQ3: What did stakeholders identify as the key barriers relating to access to technology for online learning during the school closures?

Implications for policy and practice
The 'learning crisis' in Pakistan -further exacerbated by the effects of Covid-19 -means that there is an urgent need for the Government of Pakistan to consider how children can learn more effectively. One way to do this is to fundamentally shift how education systems are organised. This would mean organising children according to their learning levels, rather than age or grade -in other words, teaching at the right level. Using technology to do this, which is the focus of the study, is one approach government officials could consider.
In doing so, governments would need to consider the type of technology that would be appropriate for the context within which this is operationalised in Pakistan. The vastly different economic, social, and cultural barriers that exist when it comes to the use of technology are crucial for success, and these would need to be considered for students, parents, and teachers. Another important implication for policy in terms of organising education systems in this way is the training that teachers would need in order to execute these changes, as opposed to continuing with a 'business as usual' approach. Specifically, given that this incorporates the use of technology by both students and teachers, as proposed by ⇡Hennessey et al. (2021) this would mean: ■ using technology to train teachers; ■ training teachers in the use of technology to support their teaching, including the assessment of their students; ■ teachers being able to support students in the use of technology for their learning.

Structure
This report consists of six sections. The introduction provides the background, context, and objectives of this study and explains its significance. Section 2 reviews the existing literature and research in this area and positions how this study can contribute to the field of knowledge. Section 3 describes the research methodology, including the sampling procedure and sample size, data collection, and analysis. Section 4 presents the main findings. Section 5 discusses the policy implications of this study and outlines a set of recommendations that relate to technology and learning. Section 6 concludes the report.

Literature review
In this section, we provide a brief review of the literature to gain a better understanding of existing studies on the use of technology in low-and middle-income countries (LMICs) to promote personalised learning for better learning outcomes and cognition. The review is made up of 41 academic articles, together with 13 studies that fell under grey literature or non-academic criteria. This section is divided into two main sub-sections on technology-enhanced personalised learning and teaching at the right level (TaRL).

Technology-enhanced personalised learning
In a recent review undertaken on technology-supported personalised learning, ⇡Major & Francis (2020) found that this can not only enhance learning outcomes but also support TaRL. The key findings from the review appear to indicate that technology-supported personalised learning: ■ Can lead to a significant improvement in learning outcomes.
■ Is adaptive in nature, allowing students to learn at their own pace.
■ Can potentially help to close educational gaps for lower-attaining students.
■ Would enhance rather than decrease the importance of teachers. Technology has revolutionised personalised learning (⇡Andra, 2016). Students' one-to-one interactions with technologies such as tablets, netbooks, and mobile devices have provided more opportunities for personalised learning both in and outside of the traditional school environment. Students' interactions with learning platforms can generate data, which may then be used to learn about their knowledge, interests, and preferences. In turn, based on this information, instructional content for students can be tailored to their needs depending on the data generated. Data generated by student use of technology, for example, may be used to compare the learning behaviours they engage in, as well as whether such actions are similar to those of previous successful or failed students (⇡Penuel & Johnson, 2016).
⇡Henrie et al.'s (2015) study of personalised learning in schools, discovered that rather than simply adding devices and software onto existing instructional programmes, an integrated and coherent set of technological tools could be used to transform instruction and learning. The schools in their study utilised information management and productivity tools, computer-adaptive curricula and assessments, as well as digital media spaces to aid students in creative work. ⇡McHugh et al. (2020) similarly describe the process school leaders developed to produce an idiosyncratic technology ecosystem which is seen as critical to the implementation of personalised learning. Such a system would encompass the use of student-facing digital management systems and computer-adaptive technologies for instruction and assessment. There is a need for a transition from traditional in-class teaching and learning to technology-integrated in-class learning, especially in Pakistan, where resistance to using technology has to be addressed by vigorous research and training.
⇡McCarthy et al. (2020) compared the performance of students in the District of California in the USA who used EdTech with that of a virtual control group. They measured the impact of the EdTech intervention on maths, reading and language. Over the three-year period that the intervention was administered, the 1,911 students who were in the treatment group consistently outperformed their cohorts. The study also examined the ways personalisation of reading tasks influences learning outcomes. It found that learners who received adaptive text selection achieved greater gains in their comprehension skills 2 compared to students receiving random text selection, but only if they were less-skilled readers. They found no significant overall differences in performance and no differences in self-reported motivation or engagement. This suggests that there are heterogeneous impacts of the use of technology in education.
⇡Tauson & Stannard (2018) also describe utilising technology as either an addition to or a replacement for, traditional methods. The results of their thematic analysis, which are organised into four topics, offer further information (ibid. p. 33): 1. Improving access to education This looks at how technology-assisted personalised learning makes good educational resources more accessible, adjusts to learners' requirements by teaching at the correct level, extends learning, and potentially eliminates educational disparities for the most disadvantaged.
2. The role of teachers and their professional development This investigates the central role of teachers and teacher professional development in enabling technology-supported personalised learning.
3. Pedagogical and motivational affordances This gives an insight into the pedagogical benefits of technology-assisted personalised learning and how it affects student motivation.

Implementation difficulties and roadblocks
This deals with the financial and infrastructure consequences, as well as scalability and sustainability concerns.
Lessons learnt through EdTech are particularly important in the contemporary environment because EdTech has the ability to adjust to learners' requirements by teaching at the right level. Although there are few examples of research addressing the development of non-cognitive skills, the majority of studies focus on maths and science teaching.
The evidence base on teachers' use of technology in the classroom in Pakistan is still developing. While a number of studies argue that technology should be provided to students, less attention is paid to how EdTech will be used by teachers. Providing technology to teachers rather than students, on the other hand, is not only more cost-effective but also results in better learning outcomes. The results of two randomised controlled trials (RCTs) in the Punjab region were compared by ⇡Beg et al. (2019) to identify the differences between two methods of improving student achievement. The first method bypassed teachers by giving eLearn Tablets to youngsters instead of teachers. In the 3 second, teachers were supported and trained using eLearn Classrooms in the second method.
The authors concluded that the e-Learn Tablets reduced student achievement by roughly 0.4 SDs, a finding they attributed to the tablets diverting children's attention away from more important educational activities. In just four months, the eLearn classroom technology increased student achievement by approximately 0.3 SDs or 60% above the control group. This was partially due to the use of EdTech in conjunction with current pedagogies.
Finally, the authors (ibid) highlighted the relative cost-effectiveness and scalability of EdTech interventions at the teacher level, arguing that such programmes must function via the government school system, which teaches over 65% of Pakistani students, in order to be effective at scale. Teacher professional development (or training) is at the heart of nearly any pedagogical intervention (⇡Waqar & Bokhari, 2019).
3 Models of eLearn are brief, expert-led, curriculum-based videos integrated into the classroom experience under a government of Pakistan programme to improve student learning in government middle schools in maths and science. The two models, eLearn Classrooms and eLearn Tablets started from the premise that both students and teachers could benefit from high-quality explanations of concepts in the official science and maths curriculum.
Investigating the Impact on Learning Outcomes of EdTech in Punjab, Pakistan 14

Teacher's effectiveness and Teaching at the Right Level
Teaching at the Right Level is an evidence-based educational strategy that assists children at the primary and secondary levels to develop fundamental reading and arithmetic abilities, resulting in improved learning outcomes. Pratham, a renowned Indian organisation committed to increasing the quality of education, pioneered this technique in 2007, and it has since been adopted by students all around the world (⇡Lakhsman, 2019).
Teaching at the Right Level was created using a combination of hands-on experience, internal assessments, and research-based RCT evaluations. It has been tested in six randomised studies in India, with positive results (⇡Banerjee et al., 2007;⇡Banerjee et al., 2010). Since then, this intervention has been employed in countries across the world for students studying between Grades 3 and 10 to improve their reading and numeracy skills.
Teaching at the Right Level is an augmented learning method that attempts to improve the quality of education by separating pupils into learning levels rather than grades and ages. It was created by the Pratham Education Foundation (⇡Jagannathan, 2001) and encompasses the following key components for its teaching practice: There are two different models used in TaRL now, following decades of assessments and refinement: 1. The Directly Implemented Learning Camp Model. In this model, children are grouped according to their learning levels after pre-assessments; they study for 2-3 hours after school, for 1-2 months).
2. The Government Partnership Model, which engages teachers after training.
Teaching at the Right Level can be implemented in any of three ways (see Figure 2). Thus, personalised learning technology-based TaRL is an approach that uses technology to facilitate the TaRL pedagogical approach. An example of where such an approach has been applied comes from India. A study undertaken by ⇡Muralidharan et al. (2017) is one example of this approach. Here, the intervention in question focused on 215 students in Marathi-medium schools in India. The study found that technology-assisted TaRL is effective for learning and teaching. Students scored 100% in one of the eight phonic skills and learning abilities increased approximately tenfold.
⇡Perry & Steck (2015) tested the impact of an online TaRL intervention on student learning outcomes in the USA. Students underwent a pre-assessment and were segregated into groups. The treatment group was then given access to iPads, while the control group were administered traditional in-classroom teaching pedagogical approaches, which involved direct instruction, pen-and-paper and drill and practice methods of learning. The content used to teach both groups was the same. Post-intervention, the treatment group demonstrated greater student engagement, improved test scores, and increased self-efficacy compared to the control group. However, a number of studies conclude that merely integrating technology into teaching and learning is not enough to achieve desired outcomes.

Summary
The literature reviewed above recommends a shift towards personalised learning. During our search for literature on technology and personalised learning in Pakistan, we found limited rigorous evidence. This is supported by ⇡Zubairi et al. (2021) who conclude that studies focusing specifically on personalised learning and technology integration in Pakistan are limited, and where they exist they are of poor quality. Furthermore, the findings from the studies that do exist must be treated with caution given their unreliable research designs (⇡Rodriguez-Segura, 2021). This dearth in research is part of the motivation for our study, the detail of which is presented in the following section.

Methodology
This section details the methodological approach used in this study, which primarily focuses on an RCT approach that measures the effect of technological interventions on students learning outcomes. Section 3.1 starts by presenting the research questions addressed by this study. Section 3.2 provides an overview of the research methodology. Section 3.3 outlines the main research instruments that were used to answer each of the three research questions for this study. Section 3.4 sets out how the analysis for each of the instruments collected was done. Section 3.5 outlines the main ethical considerations for this study. Finally, Section 3.6 discusses the main challenges and limitations experienced during the course of undertaking this study.

Research questions
The research questions this study addressed were as follows:

RQ1:
What were the key issues regarding access, barriers, and usage of EdTech by students during school closures?
RQ2: What were the separate and combined effects of delivering TaRL, fortnightly assessments and digital training sessions online on student learning outcomes when it came to Urdu, English and maths?

RQ3
: What did stakeholders identify as the key barriers relating to access to technology for online learning during the school closures?

Sampling framework
We administered the interventions to 12 randomly selected private sector schools in the Bahawalnagar district which is one of the districts situated in 4 the Pakistani province of Punjab. Within the district of Bahawalnagar, there are 118 union councils within which there are 390 private schools. Since Bahawalnagar is a remote and underdeveloped district, the majority of the private schools were unable to deliver online education due to the unavailability of technological devices such as smartphones, tablets, and 4 All the public schools in Pakistan suspended educational activities during school closures caused by Covid-19. However, some private schools throughout Pakistan continued online delivery of instruction with varying degrees of technology employment. Since the core objective of the study was to investigate the impact of technology-aided instruction, assessment, and teacher training during school closures, we chose the private schools for this study.
laptops, etc. This was further compounded by internet costs being prohibitive for the majority of the schools in this district. Hence, only 25% of the private schools in the district delivered online instruction during school closures.
The 12 schools selected for this study were selected on the basis of: 1. Their ability to deliver online education.
2. The schools offering education to Grade 8 students.
3. Both boys and girls attending the schools as this study was interested in measuring the intervention's differential effect by gender.
All available students from Grade 8 were selected and, as far as possible, an equal number of boys and girls were selected from each school.
The 12 schools were selected from 12 different union councils to avoid spillover effects. Eight of these schools were used as treatment schools, while four schools were randomly selected to act as a control group. Table 1, below, shows the distribution of samples into different treatment groups. Parents of the 258 students who were chosen to be part of the treatment and control groups were then selected to collect information intended to capture socio-economic dynamics and the possible impact of Covid-19 on the students' learning outcomes.

Survey on socio-economic parameters
We collected quantitative data on the relevant socio-economic characteristics by conducting structured surveys with 246 parents (father / mother), 258 students, 36 teachers (class teachers and subject teachers), and 12 school principals. The aim was to collect the data required for the baseline. The dimensions covered in the surveys were as follows: 1. Socio-economic profiles of households, including household composition, monthly income, information on other assets, employment status, type of house, monthly internet expenses, number and type of available devices to access online education.
2. Access and barriers to EdTech and its use.
3. Perception of respondents regarding the effectiveness of educational technologies and online delivery of instruction.

Teaching at the Right Level
The study employed two types of tests to assess student learning. The first were standard tools that have been developed for the ASER Survey in 5 Pakistan. Language and maths tests administered for the ASER Survey are the primary performance measures ASER uses to test children's performance.
Since ASER tests measure basic numeracy and language skills to Grade 3 level, we observed a ceiling effect in the results of the majority of the Grade 8 students. Grade-appropriate tests were designed to match the grade-level competency of the students to avoid such ceiling effects. The tests were designed from the Grade 8 Punjab textbook board syllabus for maths, English, and Urdu. The Urdu and English tests covered vocabulary, grammar, and reading comprehension and maths tests measured grade-level skills such as number system, sets, LCM, HCF, ratios, etc. Tests were administered twice, once in baseline and then in endline surveys (see Annex A).
These tests, which took between 30 and 35 minutes to administer, tested the grade-appropriate competencies of the students.
The preliminary observation from our baseline survey of 12 schools located in Bahawalnagar district illustrated that the majority of the sampled students 5 The ASER reading test measures a child's reading ability in terms of the following classifications: • Beginner (cannot recognise letters).
The ASER maths test measures a child's maths level in the following classifications: beginner (cannot identify single-digit numbers), one-digit number recognition, two-digit number recognition, subtraction and division. In each of these assessments, children are marked at the highest level that they can comfortably attain.
Investigating the Impact on Learning Outcomes of EdTech in Punjab, Pakistan would only benefit from online education through the use of smartphones (WhatsApp messages) or basic phones (messages). This observation was based on the very limited ownership by households in our sample of either a laptop / PC or tablet.
The majority (approximately 90%) of the sample population used mobile data to access the internet. The speed and bandwidth of the mobile data were very low in the majority of the locations that we were conducting this study, thereby making the use of Zoom to conduct online teaching a fundamental challenge. Of the sample of schools that were worked with, only two schoolssituated in relatively more developed regions of Bahawalnagar district -used Zoom to conduct online teaching. Seven out of our sample 12 schools continued online education through the use of WhatsApp. With these factors in mind, we used WhatsApp to deliver online instruction of TaRL. 6 After selecting the four schools where the TaRL intervention would be implemented, we identified weak students who scored less than 50% in the grade-appropriate test that they took for the baseline survey. The students from all four schools were then grouped into English, maths, and Urdu groups. Hence, we formed three separate WhatsApp groups (one for each subject) consisting of 30 to 35 students in each group. The grouping was done separately for each subject, and a separate instructor / volunteer (with specialisation and relevant experience in that subject) was assigned to each subject group.
The instructors sent personalised learning material to each group that matched the learning level of that specific group every week. The learning material was related to the grade-appropriate skills tested at baseline. Students received a learning problem, a personalised recorded video for solving that problem, and a short quiz through WhatsApp. The students had to solve the problem during the week and send a picture of the solved problem to the instructor via WhatsApp. Online material on IXL and Khan 7 Academy for maths and English were also utilised and shared with the students where feasible. However, the instructor for Urdu language used 7 IXL Learning was founded in 1998 and was one of the first websites that allowed teachers and learners to create and share customised study materials. See: https://www.ixl.com/ 6 A similar model is used by HundrED.org, a not-for-profit organisation, which seeks and shares inspiring innovations in K12 education. In response to Covid-19, the Gabarone-based non-governmental organisation Young 1ove in Botswana (⇡HundrED.org, 2019) has developed and trialled a 'low-tech' solution that uses SMS messages and phone calls to provide educational instruction for students in 10,000 households across Botswana, Africa. Their results show that remote instruction by phone and simple SMS texts can improve children's learning at low cost and scale.
personalised videos for instruction, practice material, and quizzes due to the 8 unavailability of online material on IXL and Khan Academy.
After three months of instruction, the students were re-assessed, and an endline survey was administered to evaluate the impact of the TaRL intervention on their learning levels.

Digital training sessions
The digital training session intervention was designed to provide teacher training in four schools on using EdTech for instruction in and out of school.
The majority of the teachers in the selected schools had not received any training to deliver technology-assisted instruction before our proposed intervention. The training was intended to help teachers to use technology more competently to better understand whether this could positively affect student learning outcomes. Three teachers per school (teaching maths, Urdu, and English respectively in Grade 8) were selected from the four schools where the digital training session treatment was administered. In one school, where there were two Grade 8 classes, 6 teachers were trained in the use of technology. In total, 15 teachers from the 4 schools received training from an expert trainer. These sessions took place over Zoom on a fortnightly basis, with a total of 6 sessions delivered (see Table 2).

Fortnightly assessment
Student engagement is fundamental for student success (⇡Fredin et al., 2015). Student engagement is a concept that is approximately three-quarters of a century old and refers to how engrossed or attentive students seem to be in their learning (⇡Axelson & Flick, 2011).
Formative assessment is a tool to enhance student motivation and engagement in the learning process. Previous research shows that formative assessment conducted by teachers in the classroom has a positive impact on students' learning and motivation and may increase students' achievements, their understanding of how to learn, and control over their learning (⇡Black & Wiliam, 1998;⇡Brookhart, 2009). Students' success and achievement through assessment "is the essential fuel that powers the learning system for students" (⇡Stiggins, 2005, p. 19).
The intervention was designed to affect learning from the demand side of education by enhancing student involvement in the learning process. For this intervention, teachers of maths, Urdu, and English from selected schools were given two training sessions before the intervention. The sessions imparted skills to make and assign online assessments to students using the CK12 app and WhatsApp. These instructors assigned six online assessments / quizzes to their class on a fortnightly basis using CK-12 and WhatsApp as per the availability of devices such as smartphones, tablets, etc., and internet facilities for their students. Each quiz was created using the content taught in the two weeks prior to the delivery of the assessment in respective schools. Reassessment of students at the end of the intervention measured the impact of this intervention on student learning levels. We used the number of tests attempted by the students to capture the impact of this intervention on students' endline scores.

Key informant interviews and focus group discussions
To help answer the third and final research question, key informant interviews and focus group discussions were held.
The key informant interviews were held with institutions / individuals with deep insight into the existing landscape of education during the Covid-19 pandemic and the spectrum of education technology in use, especially in impoverished and deprived areas of Pakistan.
Our sampling approach for stakeholder selection was based on identifying respondents from different domains. It was essential to consider including the following.
■ A stakeholder familiar with the local landscape of secondary education where the study was being conducted i.e., Bahawalnagar district.
■ A stakeholder from the research and policy domain with professional expertise in education and EdTech for policy insights.
■ Stakeholders from the international development network, operating in Pakistan and facilitating the role of EdTech in education.
In this context, stakeholders from these areas were identified and semi-structured interviews of three key stakeholders were carried out. Given the sort of information we were interested in collecting, our key informant interviews were with stakeholders from: Apart from the face-to-face interview, which we held with the member of the Private School Association, the remaining interviews were conducted online. The interview schedule was divided into the following areas (see Annex E).
1. The landscape of education in Pakistan especially during the Covid-19 pandemic.
2. The adaptability of EdTech by the target group (students, teachers, and parents).
5. The role and response of the government and key players in relation to EdTech.
Focus group discussions were held with the teachers of the selected schools in the sample. These were held in person, and each focus group included 6-8 teachers teaching in Grades 7 and 8. Participants for the focus groups were selected on the basis of their suitability and were part of a sub-pool of teachers to whom we administered the digital training intervention. The focus groups included a mix of maths, English, and Urdu teachers.

Research analysis
Section 3.3, above, outlined the research tools we used to collect the necessary data required to help us answer the three research questions which were the focus of this study. This section focuses on our analysis of the data.

Exploratory data analysis
Exploratory data analysis (EDA) attempts to examine and display observed data in a relatively straightforward way. As a technique, EDA was considered an appropriate technique for obtaining thorough information on student motivation and teachers' skills. The EDA technique also provided us with evidence on the differential access girls have to technology.

Intention to Treat (ITT)
This study suffers from non-compliance in two of its interventions. The students included in TaRL were added to WhatsApp groups. However, around 15% of the original 78 students selected for the study left the groups and did not enrol and sign the consent form to participate in the study again. A further 20% of students left the groups in the middle of the intervention. This meant that only 65% (or 51 out of the 78 original students selected for the study) remained in the group till the end of the intervention. Similarly, for fortnightly assessments, a total of six tests were administered on a fortnightly basis during the three months of the intervention to 83 students. However, 37% of students (or 31 students) did not attempt any test in this intervention.
Hence, the estimated impact in the absence of full compliance in the treatment group is called 'Intention to Treat' (ITT). Here, we compare the groups that were randomly assigned to the treatment with the comparison group regardless of the lack of full compliance of the treatment group. The ITT is a weighted average of the outcomes of participants and non-participants in the treatment group compared with the average outcome of the comparison group (⇡Khandker et al., 2010).
The following specification was used for estimating the impact of interventions on test scores (ITT) in the first step: (1) Here, Yist1 is the normalised test score (normalised at µ = 0 and SD = 1) of the student i in school s at time t1 (endline). At the same time, yist0 is the student's test score in school s at time t0 (baseline). xis is a vector of student-and household-level factors, including gender, online class status, and wealth index. ist is the error term. Treatment is the variable of interest in this equation. Treatment is a categorical variable showing random assignment of sample units into different treatment groups or the control group. Treatment is assigned the value "1" if the student is in the 'combined treatment' group, "2" if the student is in a 'TaRL treatment' group, "3" for the 'fortnightly assessment' group, "4" for the 'digital training session' group, and "0" for the control group.
We ran two specifications, one without clustering and one with clustering standard errors at the level of schools. Further, all the specifications were controlled for baseline test scores. In addition, we also controlled for gender, online class status (a dummy assigned the value "1" if the school continued online education during school closure and "0" otherwise), and for the wealth index of the household. The wealth index is calculated by using the first factors from the polychoric principal component analysis. Household monthly income and different household assets are used for the construction of the wealth index. 9

Instrumental variable estimates of dose-response relationship
Next, we also needed to measure the impact of our programme on the group of individuals who were offered the programme and who participated. This estimated impact is called the 'Treatment on the Treated' (⇡Khandker et al., 2010). In case of incomplete compliance, we needed to estimate the Local Average Treatment Effect (LATE). To measure LATE i.e., the impact of treatment on those who attended the online instruction (TaRL) or assessments (FAS), we specify the instrumental variable (IV) equation as follows: (2) 1 = α + β Here Yist1, Yist0, Xisθ, and εist are defined as earlier. 'TaRL attendance days' are the number of days a student remained in the WhatsApp group formed for TaRL instruction. Fortnightly assessments (FAS Tests) are the number of tests a student attempted out of six tests administered during the three-month intervention period. The Local Average Treatment Effect estimated the dose-response relationship between attendance days, tests attempted, and value added.
Since the participation of the students in TaRL groups and FAS tests may be endogenous to the expected performance on test scores, we instrumented attendance days and FAS tests with the random allocation of a student to particular treatment groups following ⇡Muralidharan et al. (2017).
To test the endogeneity of the instrumental variable, we applied the Durbin and Wu-Hausman tests. The null hypothesis of both the tests is that the variable under consideration can be treated as exogenous.
Next, we applied Sargan and Basmann tests of overidentifying restrictions. The null hypothesis of these tests is that one or more instruments are invalid, or that the structural model is incorrectly specified.
We ran two specifications, one without clustering and one with clustering standard errors at the level of schools. Further, all the specifications were controlled for the baseline test scores. In addition, we also controlled for gender, online class status (a dummy assigned the value "1" if the school continued online education during school closures and "0" otherwise), and the wealth index of the household. The wealth index is calculated by using the first factors from the polychoric principal component analysis (PCA) (see Table 1). Household monthly income and different household assets are used for the construction of the wealth index.

Ethical considerations
Since this research focused on students who were under the age of 18, we took a number of measures to preclude any unintentional adverse effects on the participants and to ensure their participation was truly voluntary. We obtained permission from a parent or legal guardian in the household to acquire permission for the student to take part in the study. This was followed by a discussion with the student to obtain their consent to participate. The details of the project, its aim and objectives, potential outcomes, the participant's right to withdraw from the survey process at any time, and the scope of their participation were made clear before the respondent was allowed to participate.
The participants were also assured that no identifiable personal data would be shared with any other party and that their responses would not be used for any purpose other than the aims and objectives of this research. This guarantee of anonymity and the confidentiality of the respondents' data was provided and respected.
To ensure the cultural appropriateness of the survey, female enumerators surveyed females / girls and male enumerators surveyed males / boys. Surveys were then translated into Urdu and local enumerators were assigned to conduct interviews in Urdu.

Challenges and limitations
The primary challenge for this study related to the uncertainty caused by Covid-19, with the frequent school openings and closures leading to coordination challenges. As soon as our intervention started, a nationwide lockdown due to the third wave of Covid-19 sweeping the country meant that schools were closed. This negatively affected our interaction with students and teachers and coordination with them became more challenging. Explaining and implementing the interventions would have been a lot easier if students and teachers had been present at schools. Instead, however, information sharing and compliance had to be communicated over the phone.
A further challenge negatively affecting the study was the unwillingness of teachers and school administrations to take part in the study unless financial incentives were forthcoming. Moreover, school administrators were reluctant to run the interventions introduced for this study in their schools, arguing that this would be an extra burden both for teachers and students. Rather than seeing the positive contribution that the interventions for this study may bring, school administrators felt that the intervention proposed for this study would have meant students having less time for their routine homework and classwork. When it came to student participation, we found that students responded more proactively to their school teachers in comparison to the external volunteers. School coordinators had to be hired to ensure student participation in TaRL groups.
Another challenge we faced was that a number of the teachers and students who were part of the study frequently left the WhatsApp groups which had been specifically created to administer the interventions. This links to a major limitation of this study -its small sample size. Budget and time constraints meant that we were only able to focus this intervention on 12 schools and 258 students. The attrition of students leaving the intervention or else changing schools meant the original number of students who were part of the intervention fell further to 208 when we administered the endline survey.
Lastly, the study design was constrained in the sort of technological solutions we could employ, due to the lack of appropriate infrastructure to support these in the remote area of Pakistan where we were working. The digital training session intervention, for example, was constrained due to the lack of computer laboratories in schools sampled in this study, and teachers' unreadiness to use this technology. In addition, some teachers did not even own smartphones. This limited our capacity to go for high-tech instruction methods, which we believe could have contributed to more effective and interactive delivery of instruction.

Results
This section presents the findings of the study. We begin by presenting the demographic data of the participants targeted in this study, in Section 4.1. Section 4.2 presents our findings in relation to Research Question 1 and includes a summary of our findings on the type of education technologies used by the students in our study and the challenges that they faced during school closures. Section 4.3 presents the findings from our regression analysis, which primarily explores the effect the separate and combined sets of our three interventions had on the test scores for students in English, maths and Urdu. Finally, Section 4.4 presents some of the main themes emerging from the key informant interviews conducted with key stakeholders that we interviewed, together with the focus group discussions that teachers participated in.

Data description
Our intervention targeted 258 students who took the baseline tests pre-intervention. Of these, 139 (54%) were female and 119 (46%) were male. The control group included 43 males and 44 females, while the treatment group included a total of 96 females and 75 males. Ultimately, 81% (or 208) of the original sample took the endline test (see Table 3).
While there was 100% compliance for the digital training session intervention, the issue of non-compliance arose when it came to the TaRL and fortnightly assessment interventions. For the TaRL intervention, for instance, the mean attendance days in WhatsApp groups was 25 days. The 'complier's' mean 10 attendance rate was 28% (25 days out of 90 maximum possible days) in TaRL groups. For the fortnightly assessments, a mean of 4.9 Urdu tests out of the maximum of 6 tests was attempted by complying students. The equivalents for English and maths were 5.2 and 4.9 respectively.
We found no significant difference between treatment and control groups in mean student characteristics (age, gender, wealth index, or baseline test scores) of those who attended both baseline and endline tests and who comprised our main study sample (Table 4).  11 Note: ***p < 0.01, **p < 0.05, *p < 0.1. Treatment here refers to groups who were in schools randomly assigned to receive a treatment. Control refers to the group who were in schools randomly picked to act as a comparison group. Variables used in this table are from the baseline data collection in March 2021. The data collection consisted of three parts: (a) a self-administered student survey and parent survey, from which demographic characteristics, details of schooling, and online classes are taken and (b) assessment of skills in maths, Urdu, and English, administered using pen-and-paper tests. Tests were designed to cover a wide range of achievement and to be linked between baseline and endline assessments, using common items. Scores are standardised to have a mean of zero and a standard deviation of one in the baseline. The Wealth Index refers to a wealth index generated using the first factor from a Principal Components Analysis (Polychoric) consisting of indicators for ownership of various consumer durables and services in the household and monthly income.

Results from exploratory data analysis
The exploratory data analysis (EDA) was intended to capture issues relating to the access, use, and barriers students faced in using EdTech during school closures. This was explored in relation to the socio-economic characteristics of the households. These characteristics were captured through administering a survey to the parents of students who were selected to take part in the intervention.
The main mechanisms through which students accessed learning content during school closures were WhatsApp, SMS, and textbooks. Nearly 93% of students used WhatsApp and SMS as a mechanism with which to continue their learning during the Covid-19 pandemic, while 88% used textbooks to do the same. Comparatively fewer students accessed learning through smartphone applications and web platforms (37%), and recorded videos (35%).
When it came to online classes, less than 15% of students reported learning through this mechanism during school closures. On the other hand, 50% of students resorted to in-person tuition, which became a dominant source of learning.
In terms of access and type of technology, the majority (98%) had access to some form of technology in their homes. Mobile phones turned out to be the most prevalent type of technology (96%) in households, followed by TV (72%). By contrast, laptop ownership was comparatively lower (31%). Most mobile phones owned by households were smartphones (74%), while 19% owned basic phones. The high level of smartphone ownership means there is potential to explore the access of educational apps by students through the use of smartphones. Our survey found that nearly 56% of students had access to mobile phones during school closures, while nearly half had limited access or no access at all (Figure 3, below).   The majority of the students (71%) confirmed that online lessons took place during school closures ( Figure 4). The main mechanisms for the delivery of these online lessons were WhatsApp or SMS messages. Online lessons using Zoom, Teams, and Google Classroom, on the other hand, were reported to be less well-utilised with less than one-eighth of the sample indicating these mechanisms were used.
When it came to online assessments, close to 62% of students surveyed reported taking these during the period of school closures. The type of online assessments varied depending on the school. Multiple-choice questions (32%) and short-answer questions (35%) were the main formats of online assessments. Nearly one-fifth of students attempted True / False assessments (17.5%), while a small percentage attempted fill-in-the-blanks (8.5%) and problem-solving questions (7%). A substantial percentage of students (42%) used WhatsApp for taking online assessments during school closures.
Students who faced barriers in accessing online education during school closures reported the unavailability of devices (laptops / tablets / smartphones) as the main reason for this (47%), while 19% reported the unavailability of the internet as a barrier to accessing online education. A total of 34% of students could not access online education because their school either did not offer any online options or access to devices or the internet ( Figure 5).
Students had mixed views about the effectiveness of EdTech in terms of learning. While the majority of students (48%) indicated that EdTech proved to be effective and that they had learned new concepts through EdTech, a significant number of students disagreed (30%), while 21% remained neutral. The consensus among students was that nothing could replace the in-class learning experience. When questioned about monthly internet costs over the period schools were closed, 37% of the parents surveyed reported that they spent less than Rs 1,000 per month (this is equivalent to USD 5.8). For 35% of parents surveyed, the cost was the equivalent of between Rs. 1,000-2,000 ( between USD 5.8 and USD 11.7). For 6.5% of parents surveyed, the cost exceeded Rs 2,000 (USD 11.7).
When asked about the time parents spent supporting their children with online education, the majority of the parents (87%) reported allocating 1-2 hours per day to facilitate their children's access to online education. Nine per cent spent 2-3 hours per day helping their children with online learning during school closures.
Of the 246 parents asked whether preferential access was granted to children based on their gender, 130 parents (or 53% of the total) indicated there was no gender bias. In contrast, 54 parents (or 22% of the total) indicated that they gave boys preferential treatment over girls when it came to accessing devices needed for education, while 29 (or 12% of the total) parents gave preference to girls over boys (Figure 7). Similarly, while most parents (67% of the total surveyed) permitted their daughters to access and use various devices to attain online education, 72 parents (29%) did not permit girls to access devices needed to access education during the school closures ( Figure 6). Among households where girls were prohibited from accessing devices, the reasons ranged from cultural or religious to financial ones.

Regression analysis results
This section presents the findings we sought to answer Research Question 2 using regression analysis to understand the effects on student learning outcomes of the three interventions designed for this study. A summary of the main results found the following: 1. Teaching at the Right Level significantly increased Urdu and English scores but there was no impact on maths scores.

A significant and positive Intention to
Treat effect was noted for the TaRL treatment group. Students' who were part of this treatment group increased their Urdu and English scores by 0.56 SD.
3. The Local Average Treatment Effects on Treated analysis found positive and significant improvements in Urdu and English scores. Students who were part of the TaRL treatment group increased their Urdu scores by 0.02 SD and English scores by 0.03 SD.
4. Fortnightly assessments and digital training sessions also indicated improvements in the English scores of the treatment groups. However, the number of tests attempted under fortnightly assessments did not increase student scores significantly in any of the subjects.
The grade-appropriate tests administered at baseline and endline provide us data for the assessment of grade-level competence of the students and improvement in their scores as a result of interventions administered in our sample. Before discussing the regression estimates, a simple visualisation of the baseline and endline scores through box plots (Figures 9, 10, 11) depicts an improvement in the student's endline scores in the TaRL and combined treatment groups for all three subjects in contrast to the control group. However, we observed no positive change in students' endline scores in the fortnightly assessment and digital training session treatment groups at this stage. Section 4.3.1 and 4.3.2 below present regression results to provide greater detail on the exact impact of the interventions on students' performance.   Investigating the Impact on Learning Outcomes of EdTech in Punjab, Pakistan Table 5, below, shows the impact of our different interventions on the standardised scores achieved by the students. We found that the students randomly assigned to the TaRL group increased their English and Urdu scores by 0.56 SD compared to the control group following the three months of the intervention (see columns 1, 2, 4 and 5, row 2). However, our results found no significant impact of TaRL on maths scores. This is in contrast to what other studies evaluating the impact of computer-assisted learning in under-resourced contexts have found where observable significant and positive effect sizes on student academic achievement of between 0.10 SD and 0.35 SD have been noted (⇡Banerjee et al., 2007;⇡He et al., 2008;⇡Mo et al., 2015). However, our data on Urdu and English scores are consistent with these studies. Our findings show that the combined treatment effect led to a 0.32 SD higher score in English compared to the control group (column 2 and 5, row 1).

Intention to Treat (ITT) effects
The interventions relating to fortnightly assessments and the digital training sessions also demonstrate an improvement in the English scores of the groups receiving this. Fortnightly assessments resulted in a 0.33 SD higher score while digital training sessions increased the English score of the treatment group by 0.58 SD. These interventions, however, had no impact on Urdu and maths scores (Columns 2 & 5, Row 3 & 4).
Investigating the Impact on Learning Outcomes of EdTech in Punjab, Pakistan 12 Note: Robust standard errors in parentheses. *** p < 0.01, ** p < 0.05, * p < 0.1. Combined is a dummy variable indicating a randomly assigned allocation for inclusion in the treatment group of all three interventions in one leg. Teaching at the Right Level is a dummy variable indicating a randomly assigned allocation for inclusion in the treatment group for TaRL. Fortnightly assessment is a dummy variable indicating a randomly assigned allocation for inclusion in the treatment group of fortnightly assessment. The digital training session is a dummy variable indicating a randomly assigned allocation for inclusion in the treatment group of the digital training sessions. Tests in Urdu, English, and maths were designed to cover wide ranges of achievement and linked between baseline and endline assessments, using common items. Scores are standardised to have a mean of zero and a standard deviation of one in the baseline. All equations are controlled for standardised baseline subject scores in the baseline.

Robustness of (ITT) OLS estimates
Tables C1, C2, and C3 in Annex C show our specification results with the addition of three important covariates as robustness checks. The first covariate is the wealth index of the households constructed by using the first factors of polychoric principal component analysis (see Table 1). The second covariate is a dummy for gender, and the third is the dummy for showing if the school continued online education during school closures due to Covid-19. These additional covariates did not change our previous findings, with the TaRL intervention contributing to a positive and significant improvement on the Urdu and English scores of students. Table C1 presents the impact of the wealth index on students' standardised endline scores. We found that students with a higher score on the wealth index scored 0.18 SD and 0.14 SD points higher in English and maths tests while 0.2 SD lower in the Urdu tests after three months.
These findings may be supported by the fact that the students from well-off families usually go to English-medium schools where greater emphasis is given to English and maths subjects. The students from higher-income brackets usually have good English language skills compared to Urdu. We can confirm it from the fact that the wealth index and baseline scores of students were positively associated with maths (Correlation coefficient = 0.16) and English scores (Correlation coefficient = 0.27) while negatively correlated with the Urdu scores (Correlation coefficient = -0.05).
Gender did not significantly affect student scores (Table C2). However, for students who continued to receive online education over the course of the school closures, standardised maths and English scores improved at the endline (Table C3). At the same time, however, we found no impact of online classes on the Urdu scores of the students.
Again, in all specifications, TaRL significantly improved Urdu and English scores with no impact on maths scores. Recent research on the effectiveness of computer-assisted learning has shown that insignificant or small impacts may be linked to a lack of compliance with the intervention (⇡Mo et al., 2020; ⇡Tang et al., 2018). The next section takes non-compliance into account and estimates the local average treatment effect on the treated.

Local Average Treatment Effects on Treated (LATE)
Using the 2SLS technique, IV estimates appear to indicate that an extra day of attendance within the WhatsApp groups formed for TaRL instruction increased Urdu scores by 0.03 SD and English scores by 0.02 SD. See columns 1, 2, 4, and 5 in Table 6, below.
However, the number of tests attempted under fortnightly assessments did not increase student scores significantly in any of the subjects. Since there was 100% compliance for the digital training sessions treatment, we only tested the dose-response value-added impact for TaRL and fortnightly assessment interventions. Rows 8,9,10,and 11 in Table 6 present the test statistics for overidentifying restrictions and endogeneity. The P-value of both Sargan and Basmann test statistics is greater than 0.1; hence we reject the null hypothesis of overidentification and misspecification of our structural model. Durbin and Wu-Hausman test statistics are highly significant, so we reject the null of exogeneity; we must continue to treat endline test scores as endogenous. Again a day's increase in attendance in the WhatsApp TaRL group led to a 0.04 SD increase in Urdu and a 0.03 SD increase in English scores. Hence, attendance in TaRL groups turned out to be consistent and significant in this specification as well. Online classes increased students' scores by 0.49 SD in English and 0.51 SD in maths. We found a significant improvement in the English and Urdu standardised scores in the TaRL treatment group when using the ITT and LATE approaches. However, the fortnightly assessment intervention did not show any improvement in test scores at the endline. However, we found no significant impact of TaRL on maths scores.
An important aspect of computer-assisted learning at the right level is the differential role of teachers in school-or home-based learning. ⇡Straub (2009) suggested that adoption and compliance of computer-assisted learning interventions are significantly dependent on the teacher, even if the end-user is the student. Since greater teacher-student interaction is required for effective maths understanding, our intervention of TaRL, where video lectures were shared through WhatsApp messages when the schools were closed, could not have a significant impact.

Stakeholder interviews and focus group discussions
To answer our third and final research question, we undertook interviews to help us identify the key barriers in relation to technology in the context in which we were working in Pakistan. Beyond access to technological devices, which we discussed in Section 4.2. as part of the EDA analysis, the following section presents a summary of some of the key issues identified by the stakeholders we interviewed.

Household resistance to the use of technology
Online education requires the cooperation of parents but, as the feedback from the focus groups and key informant interviews illustrated, this was not forthcoming. A number of reasons for the lack of parental cooperation were identified.
The low literacy and educational attainment levels of the parents of the student population we were working with meant that few parents were technologically literate. This translated into two specific barriers. First, parents' technological illiteracy prevented them from supporting their children when accessing online education. Second, it led to parents' reluctance in allowing their children access to technological devices. Rather than seeing technology as a medium to access educational content, technology was instead perceived by parents to be a mechanism through which children were idly spending their time surfing the internet. In other words, in the eyes of many parents, the sole purpose of technology is about providing entertainment rather than it being seen as a medium for accessing education.
In terms of accessing education online, gender dimensions also reflected some of the restrictions imposed in households, specifically when it came to girls' use of technology. In the EDA analysis, 80% of respondents indicated that they gave preference to boys over girls for using mobile phones. The focus group discussions with teachers indicated that often girls were not allowed to use mobile phones, or else indicated that the uneven distribution of household chores contributed to girls' lack of participation in online classes.

Lack of training for teachers in the use of technology
Aside from barriers relating to households, the lack of teacher training in how to use technology to conduct online classes was identified as a barrier. In the context of the study, teachers were ill-prepared to teach remotely using technology and were not given the tools needed to do so. While teachers required time to acclimatise to this new normal, the reality was that several stakeholders identified that teachers were not given the support they needed to switch from face-to-face to online teaching Additionally, the interviews raised the point that there was a lack of transparency as to how students should be assessed by teachers during remote learning. Instead, decisions were made in an ad hoc manner, with teachers often having to decide how this should be done.

Policy implications
This study has revealed a number of important findings. Some of these relate specifically to the use of technology-assisted TaRL, while other insights are more generally related to the use of technology to support education. Specifically, TaRL is a promising approach to teaching, which has demonstrable positive effects on learning outcomes. However, there are general challenges relating to technology adoption that must be overcome before technology-enabled TaRL can be incorporated more widely.
This section presents some of the main issues that policymakers need to take into account when designing a technology-assisted TaRL programme in Pakistan.

Actively involving school actors in the design of a technology-assisted TaRL intervention is important for the long-term buy-in of teachers and students.
One of the challenges that this study faced was the resistance of school actors in implementing this intervention. By extension, this led to challenges of non-compliance among students selected to take part in the programme. Where school teachers were supportive of our intervention, students were more proactive in the WhatsApp groups that were being administered by us. The opposite was also true with a lack of teacher support resulting in less student compliance. Part of the resistance stemmed from school officials believing the intervention created more work for them and the students. Therefore, any TaRL intervention should be integrated into existing processes. It must also avoid overburdening both students and teachers in order to get adequate buy-in.

Understanding the complexities involved when it comes to implementing technology-assisted TaRL interventions and how this can impact their level of success is key.
For the purposes of this study, the design was heavily reliant on hiring specialist volunteers to administrate the WhatsApp groups through which interventions were administered. Similarly, due to the school closures, the intervention was largely implemented out of school and at home. Lastly, the interventions were made in addition to the instructional time being offered by schools. Each of these factors affected student and teacher compliance. For example, interventions were administered at the household level, despite our interviews appearing to conclude that the households in the context we were working in appeared to harbour largely negative attitudes towards technology. Policymakers would therefore need to consider the best mix of approaches when designing a technology-assisted TaRL intervention. While the interventions that we administered for this study did not provide support to any of the households, given the importance of such support we would also recommend that any programme carefully look at what support households could be given to facilitate students' learning.
3. Considering factors relating to access and use of devices beyond device ownership alone is essential. The design of our study was to a large extent influenced by the types of devices households in the poor rural contexts we were working in had access to, namely, mobile phones. We found, however, that even where household ownership of mobile phones was high, internet costs associated with accessing Whatsappthe main medium through which our intervention was rolled out -was a big challenge in terms of engaging students within our treatment groups. This supports ⇡Muralidharan et al. (2017), who found that any technology-aided instruction involving even a small cost limits the ability of low-income students to benefit from the programme. In such contexts, policymakers must consider costs relating to hardware, alongside costs that end-users may face. Policymakers may wish to consider whether and how some of these costs can be subsidised.

4.
Community perceptions of technology must be factored into the planning of any technology-supported education programmes. Lack of implementation fidelity was a significant challenge in delivering this research. There were significant levels of non-compliance among participants. Our research identified that this non-compliance largely stemmed from limited pre-existing exposure to technology among the participants' families and teachers. Where technology use was more common it was generally used for entertainment purposes, rather than education. These kinds of cultural norms and reluctance to use technology to support learning must be addressed before EdTech can support learning among the most marginalised.

Conclusion
The objective of this study was to test the impact of a technology-assisted TaRL intervention on student learning outcomes during school closures in the low-income district of Bahawalnagar in Pakistan. This was undertaken using low-tech solutions. Our findings showed a significant and positive impact on the Urdu and English scores of students who were part of the TaRL treatment group. However, no significant impact of the TaRL intervention was found for maths scores. The digital training session and fortnightly assessment treatment groups were found to have positive and significant effects only on English scores.
The impact of the intervention was found to be linked with both the fidelity of user uptake, as well as cultural norms surrounding the use of technology among the beneficiary communities. Interestingly, the study found that gender did not seem to be a determining factor in the learning outcomes generated. However, it was observed that students from higher-income households generally had higher levels of engagement with the intervention, and therefore, income was linked to better learning outcomes of students.
This study concluded with a set of key policy suggestions based on the findings emerging from the study. This includes policy suggestions relevant to the deployment of technology-enabled TaRL solutions, as well as the deployment of technology-facilitated learning interventions more generally. These recommendations include ensuring that school actors (i.e., teachers and students) are involved in the design of technology-assisted TaRL solutions to foster buy-in, applying due consideration to how the implementation modalities may influence the uptake of the TaRL tools, expanding access considerations well beyond device ownership and considering cultural attitudes to technology during programme design. Considering these important areas during the design and implementation of both technology-enabled TaRL programmes and EdTech interventions more broadly, are likely to significantly improve the success of future similar initiatives. 3. Identify and write the present, past and future verbs in the following sentences: a. Jan called his teacher b. They will sing hard c. He loves his father 4. Identify common and proper common nouns in the following sentences and write in the table.

Sentences Common Noun Proper
I love my mother.
Ali is playing football.

Subject pronoun Object pronoun
She gave me some chocolates.
Ahmad plays football with his brother.

Punctuate the following sentences.
will you go now I don't know Ali wants me to visit him in lahore.

The History of Chocolates
One of the most popular foods of all times is chocolates. People nowadays eat chocolates in many different forms. We eat chocolate candy, and we drink hot and cold chocolate drinks. The Chocolate we eat today is made from a lot of different ingredients, but the most important ingredient is cacao beans.
The story of the cacao bean and its long journey to stores and supermarkets all over the world started hundreds of years ago in Mexico. Cacao trees need hot and humid weather, and they originally grew in the Yucatan Peninsula.
The Maya were the first people to eat cacao beans. They picked cacao beans from wild trees and cleared land to cultivate their own trees. They made a drink from cacao beans and exchanged the beans for other goods. They also used cacao beans for religious ceremonies.
Mayan merchants traveled to the north and introduced cacao beans to the Aztec people. Soon the cacao bean was part of the Aztecs' lives. They used it as drinks, as part of religious ceremonies, and as money. With ten beans, you could buy a rabbit. With ten beans, you buy a slave, one of the Aztec Gods. The Aztecs could not grow cacao trees because of the dry climate. When the Aztecs conquered the Maya, they asked for cacao beans as tribute.

7.A Read and circle the mistakes in the sentences.
I. Chocolate isn't popular nowadays. II. The Cacao trees grow in Europe. III. The Maya made bread from cacao beans.

7.B Read the story and answer the questions.
I. What is the most important ingredient in chocolate? II. Where did cacao beans originally grow? III. Who were the first people to eat cacao beans?
IV. Who introduces Cacao beans to the Aztecs? V. What did the Aztecs use cocoa beans for? VI. What did the Maya give Aztecs as a tribute? Infection An act of using violence to try to hurt or kill somebody.
Intolerance A substance that is put into the blood and that protects the body from a disease.

Vaccine
The fact of not being willing to accept ideas 2. Choose and write the verbs in the following sentences.
He goes to the party. Oscar likes to play soccer.
Maria sleeps for eight hours.

Identify and write the present, past and future verbs in the following sentences.
She'll write the email after lunch.

Bill writes the letters
Angela watched TV all night.

Sentences Common Noun Proper
Alex is a wonderful player.
He moved to Australia when he was young.

Sentences Subject pronoun
Object pronoun Felix stunned the dragon with a ray gun.
Gus smashed the electric guitar with a sledgehammer.
she said I'm sorry Happy birthday Annie my sisters name is Lisa 7. Read the passage. Then answer the questions. Once upon a time, dogs, cats, and mice were friends. Then, one day, the dogs had to go away on a trip. "We have some important papers, "they told the cats. "Will you look after them for us?" "Yes," said the cats. "We will keep them safe for you." When the dogs had gone, the cats said, "We have nowhere to keep the papers. Let's give them to the mice." "Will you look after these papers for us?" the cats asked the mice. "Yes," said the mice. "We'll look after them for you." Winter came. The mice were cold. They nibbled at the papers and made little nests to keep themselves warm. Then, one day, the dogs came back. "We've come for our papers," they said to the cats. "We had nowhere safe to keep them," said the cats. "We gave them to the mice." The cats went to the mice. "The dogs are back," they said. "We've come for the papers."When they saw what the mice had done, the cats were very angry. They chased every mouse they saw. The cats went back to the dogs. "The mice did not look after your papers," they said. "The mice were cold, so they made them into nests." The dogs were very angry with the cats. They chased every cat they saw. From that day on, dogs have chased mice. They have never been friends again!           Fortnightly assessment, and Digital training sessions are defined as in Table 5. The wealth index is generated using the first factors from Polychoric PCA applied to household assets and monthly income of the households. Tests in Urdu, English, and maths were designed to cover wide ranges of achievement and to be linked between baseline and endline assessments, using common items. Scores are standardised to have a mean of zero and a standard deviation of one in the baseline. All equations are controlled for standardised baseline subject scores in baseline  Table 5 . Gender is a dummy variable assigned the value "1" if male and "0" otherwise. Tests in Urdu, English, and maths were designed to cover wide ranges of achievement and to be linked between baseline and endline assessments, using common items. Scores are standardised to have a mean of zero and a standard deviation of one in the baseline. All equations are controlled for standardised baseline subject scores in baseline. Fortnightly assessment and Digital training session are defined as in Table 5. Online class is a dummy variable assigned values "1" if the students received online education during school closure and "0" otherwise. Tests in Urdu, English and maths were designed to cover wide ranges of achievement and to be linked between baseline and endline assessments, using common items. Scores are standardised to have a mean of zero and standard deviation of one in the baseline. All equations are controlled for standardised baseline subject scores. The wealth index is generated using the first factors from polychoric PCA applied to household assets and monthly income of the households. Tests in Urdu, English, and maths were designed to cover wide ranges of achievement and to be linked between baseline and endline assessments, using common items. Scores are standardised to have a mean of zero and a standard deviation of one in the baseline. All equations are controlled for standardised baseline subject scores in the baseline.  22 Note: Robust standard errors in parentheses. *** p < 0.01, ** p < 0.05, * p < 0.1. Attendance days is the number of days a student remained in the WhatsApp group formed for TaRL instruction. Tests Attempted is the number of tests attempted out of 6 total tests by the students in the FAS treatment group. Tests in Urdu, English and maths were designed to cover wide ranges of achievement and to be linked between baseline and endline assessments, using common items. Scores are standardised to have a mean of zero and standard deviation of one in the baseline. All equations are controlled for standardised baseline subject scores. 23 Note: Robust standard errors in parentheses. *** p < 0.01, ** p < 0.05, * p < 0.1. Attendance days is the number of days a student remained in the WhatsApp group formed for TaRL instruction. Tests Attempted is the number of tests attempted out of 6 total tests by the students in the FAS treatment group. Tests in Urdu, English and maths were designed to cover wide ranges of achievement and to be linked between baseline and endline assessments, using common items. Scores are standardised to have a mean of zero and standard deviation of one in the baseline. All equations are controlled for standardised baseline subject scores. 24 Note: Robust standard errors in parentheses. *** p < 0.01, ** p < 0.05, * p < 0.1. Attendance days is the number of days a student remained in the WhatsApp group formed for TaRL instruction. Tests Attempted is the number of tests attempted out of 6 total tests by the students in the FAS treatment group. Tests in Urdu, English and maths were designed to cover wide ranges of achievement and to be linked between baseline and endline assessments, using common items. Scores are standardised to have a mean of zero and standard deviation of one in the baseline. All equations are controlled for standardised baseline subject scores. 25 Note: Robust standard errors in parentheses. *** p < 0.01, ** p < 0.05, * p < 0.1. Attendance days is the number of days a student remained in the WhatsApp group formed for TaRL instruction. Tests Attempted is the number of tests attempted out of 6 total tests by the students in the FAS treatment group. Tests in Urdu, English and maths were designed to cover wide ranges of achievement and to be linked between baseline and endline assessments, using common items. Scores are standardised to have a mean of zero and standard deviation of one in the baseline. All equations are controlled for standardised baseline subject scores.

4.
As a researcher or policymaker, how can we bridge the gap between government and other key players (donors, school associations etc.) who collectively play a pivotal role in the education landscape?

5.
What type of human and economic resource readiness is important for technology integration in Pakistan?

6.
Although we face a capacity constraint and have limited resources in our schools, the little that we have available is also not utilised to its optimum. How can we confront this issue?

7.
What are the challenges faced by researchers to create impactful research given the socio-economic conditions in Pakistan?

8.
Considering the economic disadvantages and the landscape of South Punjab, do you suggest RCT to be a precise technique to employ? If yes, how can we curtail the challenges related to it?

9.
Should Teaching at the Right Level (TaRL) be adopted at macro level to compensate for the learning level deficit in Pakistan?

10.
Do you think this pandemic is a gateway to technology integrated education or to hybrid education in the case of Pakistan? Are we able enough to utilise this opportunity? Also, comment on learning losses from covid-19.

5.
The FCDO's GEC Transition (GEC-T) and Leave No Girl Behind (LNGB) programs under the umbrella of Girls Education Challenge (GEC) global initiative comprises interventions for the highly marginalised to attain and resume education. How do you see the intervention of education technology in this capacity? Possibilities of scope, size and scaling up of EdTech in this context?

6.
PM Boris Johnson has termed supporting quality school education as "one of the smartest investments we can make as the world recovers from Covid-19" and challenged the world's richest countries to prioritise school education at the G7 by pledging £430 million of new development aid for the World Bank-backed Global Partnership for Education (GPE), which provides funds for schools across developing nations. How do you see promotion/enhancement of EdTech under this mega initiative? What developments do you see happening for Pakistan through FCDO, in this regard? Following insights were expected from the key stakeholder: 1.What is the general perception of the community, especially the economically weak households (as they constitute the majority) towards online education and EdTech?

2.
How readily were you able to embrace this change and what were the challenges you faced? How did you strategise it?

3.
Were the parents supportive towards this new mode of education? Also, was there any gender discrimination in terms of the use of gadgets? Any preferential behaviour of parents regarding it?

4.
How investing in educational technology can benefit underprivileged kids directly?