MindPlay Virtual Reading Coach: Does It Affect Reading Fluency in Elementary School?

Kloos, Heidi; Sliemers, Stephanie; Cartwright, Macey; Mano, Quintino; Stage, Scott

doi:10.3389/feduc.2019.00067

ORIGINAL RESEARCH article

Front. Educ., 10 July 2019

Sec. Digital Education

Volume 4 - 2019 | https://doi.org/10.3389/feduc.2019.00067

MindPlay Virtual Reading Coach: Does It Affect Reading Fluency in Elementary School?

HK
Heidi Kloos ¹^*
SS
Stephanie Sliemers ¹
MC
Macey Cartwright ¹
QM
Quintino Mano ¹
SS
Scott Stage ²

1. Department of Psychology, University of Cincinnati, Cincinnati, OH, United States
2. Department of Psychology, North Carolina State University, Raleigh, NC, United States

Abstract

Important strides have been made in the science of learning to read. Yet, many students still struggle to attain reading proficiency. This calls for sustained efforts to bridge theoretical insights with applied considerations about ideal pedagogy. The current study was designed to contribute to this conversation, namely by looking at the efficacy of an online reading program. The chosen reading program, referred to as MindPlay Virtual Reading Coach (MVRC), emphasizes the mastery of basic reading skills to support the development of reading fluency. Its focus on basic skills diverges from the goal of increasing reading motivation. And its focus on reading fluency, vs. broad literacy achievement, offers an alternative to already existing reading enrichment. In order to test the efficacy of MVRC, we recruited three school districts. One district provided data from elementary schools that used the MVRC program in Grades 2 to 6 (N = 2,531 total). The other two districts participated in a quasi-experimental design: Six 2nd-grade classrooms and nine 4th-grade classrooms were randomly assigned to one of three conditions: (1) instruction as usual, (2) instruction with an alternative online reading program, and (3) instruction with MVRC. Complete data sets were available from 142 2nd-graders and 172 4th-graders. Three assessments from the MVRC screener were used: They assessed reading fluency, phonic skills, and listening vocabulary at two time points: before and after the intervention. Results show a clear advantage of MVRC on reading fluency, more so than on phonics or listening vocabulary. At the same time, teachers reported concerns with MVRC, highlighting the challenge with reading programs that emphasize basic-skills mastery over programs that seek to encourage reading.

Highlights

- We assessed the efficacy of the online reading program MVRC on the reading fluency of children in Grades 2–6.
- Correlational analyses reveal a significant effect of MVRC time on improvements in reading fluency in Grades 2, 3, 5, and 6.
- MVRC time affected improvements in reading fluency more than in phonics or in listening vocabulary.
- Two randomized control trials (Grade 2; Grade 4) revealed a strong effect of MVRC on reading fluency, compared to instruction as usual and compared to an alternative online reading program.
- Teachers had some concerns with MVRC. When given a choice, none of the teachers interviewed were interested in adopting MVRC for their classroom.

Introduction

Attaining reading proficiency remains a challenge for many students. For example, the recent National Assessment of Educational Progress report found that only 37% of 4th-graders could read proficiently (National Center for Education Statistics, 2017). Online programs are a possible solution to this problem, variously referred to as computer-based teaching (CBT), computer-assisted instruction (CAI), computer-managed instruction (CMI), computer-assisted learning (CAL), digital game-based learning, or the like (Peterson et al., 1999; Kozma, 2003; Fenty et al., 2015; Jamshidifarsani et al., 2019). In the current paper, we test the efficacy of such a program: the MindPlay Virtual Reading Coach (www.mindplay.com). This program is designed to improve reading fluency via the mastery of basic reading skills.

Reading Fluency and the Importance of Basic Reading Skills

The pace at which an individual can read matters centrally for general literacy achievement (Alvermann et al., 2013; Silverman et al., 2013). A student needs to be able to read a minimum number of words per minute in order to be able to interpret and analyze a text. Yet, it is still unclear how to best raise a student's reading fluency. To illustrate this gap in our understanding, we briefly review the research listed in the What Works Clearinghouse (WWC; U.S. Department of Education¹). This data base summarizes research studies that report on the efficacy of a pedagogical tool. While the details of the pedagogical tools differ, all studies in the WWC data base use a research design that provides conclusive evidence for effective pedagogy.

For the current purposes, we looked specifically at WWC research that assessed reading fluency in elementary school. This filter returned 16 research papers, covering the following literacy programs: SpellRead™, READ180, Success for All®, Earobics®, Leveled Literacy Intervention, Fast ForWord®, Peer-Assisted Learning Strategies, Start Making a Reader Today®, Read Naturally®, Achieve3000®, and Repeated Reading. Of these programs, 38% involved an online component, 31% involved small-group pedagogy, and 12% involved a training that went beyond reading (e.g., training in socio-emotional skills). Importantly, despite this diversity, improvement in reading fluency was rare – documented in less than 20% of the studies (see Table 1 for details). Thus, a large majority of existing tools did not lead to improved reading fluency (Macaruso et al., 2006; Ross et al., 2010; Balu et al., 2015).

Table 1

Name of the program (with brief description)	Description of Sample	Stat. sign. improvement?
SpellRead™ integrates the auditory and visual aspects of the reading process and emphasizes specific skill mastery through systematic and explicit instruction. Students are taught to recognize and manipulate English sounds; to practice, apply, and transfer their skills using texts at their reading level; and to write about their reading.
Rashotte et al. (2001)	N = 33 Grade 5	No
Torgesen et al. (2006)	N = 104 Grade 5	No
READ 180® combines online and direct instruction, student assessment, and teacher professional development. READ 180® is delivered in 90-min sessions that include whole-group instruction, three small-group rotations, and whole-class wrap-up. Small-group rotations include individualized instruction using an adaptive computer application, small-group instruction, and independent reading.
Fitzgerald and Hartry (2008)	N = 297 Grades 4–6	No
Kim et al. (2010)	N = 264 Grades 4–6	Yes: 4% Improvement
Success for All® is part of a whole-school reform model that includes a literacy program, quarterly assessments of student learning, a social-emotional development program, computer-assisted tutoring tools, family support teams for students' parents, a facilitator who works with school personnel, and extensive training for all intervention teachers. The literacy program emphasizes phonics for beginning readers and comprehension for all students. Teachers provide reading instruction to students grouped by reading ability for 90 min a day, 5 days a week. In addition, certified teachers or paraprofessionals provide daily tutoring to students who have difficulty reading at the same level as their classmates.
Tracey et al. (2014)	N = 737 Grade 1	No
Earobics® provides students with individual, systematic instruction in early literacy skills as students interact with animated characters. It builds students' skills in phonemic awareness, auditory processing, and phonics, as well as the cognitive and language skills required for comprehension. Each level of instruction addresses recognizing and blending sounds, rhyming, and discriminating phonemes within words, adjusting to each student's ability level. The software is supported by music, audiocassettes, and videotapes, and includes picture/word cards, letter–sound decks, big books, little books, and leveled readers for reading independently or in groups.
Gale (2006)	N = 24 Grade 1	No
Cognitive Concepts Inc (2003)	N = 74 K-Grade 3	No
Leveled Literacy Intervention is a short-term, supplementary, small-group literacy intervention designed to help struggling readers achieve grade-level competency. It provides explicit instruction in phonological awareness, phonics, fluency, vocabulary, reading comprehension, oral language skills, and writing. It also helps teachers match students with texts of progressing difficulty and deliver systematic lessons targeted to a student's reading ability.
Ransford-Kaldon et al. (2010)	N = 281 Grades 1-2	Yes: 19% Improvement
Fast ForWord® is a computer-based reading program intended to help students develop and strengthen the cognitive skills necessary for successful reading and learning. It is designed to be used 30 to 100 min a day, 5 days a week, for 4 to 16 weeks.
Scientific Learning Corporation (2008)	N = 308 Grade 2	No
Peer-Assisted Learning Strategies is a peer-tutoring program that aims to improve student proficiency in several disciplines. During the 30 min peer-tutoring sessions, students take turns acting as the tutor, coaching and correcting one another as they work through problems. The designation of tutoring pairs and skill assignment is based on teacher judgement of student needs and abilities, and teachers reassign tutoring pairs regularly.
McMaster et al. (2005)	N = 41 Grade 1	No
Start Making a Reader Today® is a volunteer tutoring program designed to be a low-cost, easy-to-implement intervention. Volunteer tutors go into schools where at least 40% of students are eligible for free or reduced-price lunch and read one-on-one with students twice a week for 30 min. Typically, one volunteer works with two students on four types of activities: reading to the student, reading with the student, re-reading with the student, and asking the student questions about what has been read. The program also gives each student two new books a month to encourage families to read together.
Baker S. et al. (2000)	N = 84 Grades 1–2	No
Read Naturally® is a supplemental reading program designed to improve reading fluency using a combination of books, audiotapes, and computer software. The program has three main strategies: repeated reading of text for developing oral reading fluency, teacher modeling of story reading, and systematic monitoring of student progress by teachers and the students themselves. Students work at a reading level appropriate for their achievement level, progress through the program at their own rate, and, for the most part, work on an independent basis.
Arvans (2009)	N = 82 Grades 2–4	No
Hancock (2002)	N = 94 Grade 2	No
Kemp (2006)	N = 158 Grade 3	No
Achieve3000® is a supplemental online literacy program that provides nonfiction reading content and focuses on building phonemic awareness, phonics, fluency, reading comprehension, vocabulary, and writing skills. Teachers use the program with an entire class but the assignments are tailored to each student's reading ability level. For example, teachers assign an article and related activities to an entire class; the program then tailors the version of the article to each student by automatically increasing the difficulty of text when a student is ready for more challenging text. Achieve3000® provides lessons that follow a five-step routine: (1) respond to a Before Reading Poll, (2) read an article, (3) answer activity questions, (4) respond to an After Reading Poll, and (5) answer a Thought Question. Progress reports and student usage data, provided by the online tool, enable teachers to track both whole-class and individual student progress.
Hill and Lenard (2016)	N = 12,851 Grades 2–5	Yes: 11% Improvement
Repeated Reading aims to increase oral reading fluency of students who have developed initial word reading skills but demonstrate inadequate reading fluency for their grade level. During repeated reading, a student sits in a quiet location with a teacher and reads a passage aloud at least three times. Typically, the teacher selects a passage of about 50 to 200 words in length. If the student misreads a word or hesitates for longer than 5 seconds, the teacher reads the word aloud, and the student repeats the word correctly. If the student requests help with a word, the teacher reads the word aloud or provides the definition. The student rereads the passage until he or she achieves a satisfactory fluency level.
Wexler et al. (2010)	N = 62 Grades 9–12	No

IES what works clearinghouse (WWC) programs.

All information stems from the WWC data base. The filter applied to the data base was such that (1) research papers reported reading-fluency effects, and (2) the program is designed for at least one elementary-school grade. Statistically significant improvement pertains to reading fluency only.

In order to understand how to improve reading fluency, two extremes can be considered: Students either lack the required skills, or they lack the required motivation (Thomas, 2013). There is evidence for both extremes. For example, research has demonstrated that foundational skills of phonics are central to reading fluency (Ehri et al., 2001), even for adult readers (e.g., Van Orden et al., 1990). In line with these findings, the National Reading Panel recommends that reading lessons should focus on the fundamentals of phonemic awareness, phonics, fluency, vocabulary, grammar, and comprehension strategies (National Reading Panel, 2000). Early reading skill was even found to predict later reading motivation, further highlighting the relevance of fundamental reading skills (e.g., Chapman and Tunmer, 1997; Poskiparta et al., 2003).

On the other hand, researchers have proposed that poor motivation may be a defining factor of reading failure (e.g., Baker L. et al., 2000; Pressley, 2002; Wang and Guthrie, 2004; Lepola et al., 2005; Sideridis et al., 2006; Morgan et al., 2008). The argument is that the development of reading skills requires frequent reading practice, which, in turn, depends on perceiving reading as valuable and fun (Echols et al., 1996; Griffiths and Snowling, 2002; Meece et al., 2006; Froiland and Oros, 2014). In support of this argument, children's early reading motivation was found to predict later reading ability (Chapman et al., 2000; Onatsu-Arvilommi and Nurmi, 2000).

It should be noted that an ideal reading pedagogy is not an either-or proposition (Madden et al., 1993; Mano and Guerin, 2018; Prescott et al., 2018). Basic skills of decoding, phonics, and grammar are not detached from the communicative aspects of an interesting read (e.g., Fountas and Pinnell, 2006; Wallot, 2014). However, recent years have seen an increased emphasis on the communicative aspect of reading, as illustrated by the recommendation from the American Academy for Pediatrics to read with children at home (American Academy of Pediatrics News, 2014). This might have inadvertently pushed the pendulum too far into the camp of increasing reading motivation, at the expense of the development of basic phonics and grammar skills (Wise et al., 2000; e.g., Bosman et al., 2017).

MVRC has the potential to re-adjust the pendulum of pedagogy, away from an over-emphasis on content-rich activities of literacy, and toward a focus on the mechanics of decoding. MVRC is based on the idea that training in foundational skills can improve reading fluency more so than mere reading practice (Bosman and Van Orden, 1997; Mellard et al., 2010; Huo and Wang, 2017; Cordewener et al., 2018). Given this theoretical commitment, MVRC does not consider students' preferences about what to read, and there are no choices about lessons and practice activities. Rather than seeking to entice students to read, MVRC rolls out a learning regime that targets identified gaps in foundational reading skills.

Details About MVRC

MindPlay Virtual Reading Coach is a commercially available educational software geared toward improving reading fluency in an individualized learning environment. According to their webpage, lessons are provided by virtual reading specialists and speech pathologists, followed by practice that includes immediate and specific feedback. Depending on the proficiency of the student, the program covers phonological awareness, phonics skills, vocabulary, grammar, silent reading fluency, and comprehension. A unique and differentiated syllabus is automatically developed and then adapted continuously to fit the needs and emerging skills of individual students.

More specifically, MVRC combines three facets of pedagogy: a comprehensive diagnostic tool, a lesson/practice pairing that is calibrated to fit the proficiency gaps of the student, and a flow-chart structure of activities designed to support mastery of foundational skills. The diagnostic tool, known as the universal screener, is administered when the student first logs on. It consists of several elements, each of which is normed internally (MindPlay Universal Screener™, 2018). For Grades 2 and older, the first part of the screener is designed to determine the student's reading fluency. Subsequent parts pertain to a visual scanning test, a listening vocabulary test, a phonics test, and a letter-discrimination test.

Depending on the screener's outcome profile of a specific student, MVRC determines lesson/practice pairings that precisely fit the skills and gaps of the student (MindPlay Virtual Reading Coach, 2017). The lessons pertain to phonemic awareness, phonics, fluency, vocabulary, grammar for meaning, and comprehension. Lessons are delivered by an online reading coach, requiring minimal involvement of the teacher. Each lesson is followed by pertinent practice activities to enforce the material covered in a lesson. Importantly, the amount of practice is adjusted continuously, depending on a student's success rate during the practice activities. Thus, the speed at which students move through the lessons is determined by their error rate and types of errors.

An underlying flow-chart structure defines the order in which lessons and practice activities are presented. Central to this structure is the gradual increase in difficulty. For example, in phonics, the initial lessons involve single letters that have relatively straight-forward phonics (e.g., letters m, b, t, s and a), followed by letter groups that have more complex phonics (e.g., th, sh). A minimum of 90% mastery is required before the student moves on to a new lesson set. If this level of mastery is not achieved on the first try, the lesson content is revisited in several different ways so that mastery becomes attainable without excessive repetitions. The expected outcome is an improvement in reading fluency.

Preliminary research with MVRC was carried out in several dissertations (Jensen, 2015; Sherrow, 2015; Kealey, 2017; Mann, 2017; Reiser, 2018). There is also published research, for example with college-students placed in a remedial reading course (Bauer-Kealey and Mather, 2018), with 2nd-graders in a diverse school district (Schneider et al., 2016), with 2nd-grade second-language learners (Vaughan et al., 2004), and with 8th-graders enrolled in a summer program (Chambers et al., 2013). Results are promising: For example, 2nd-graders who logged in for an average of 44 MVRC hours improved in their reading fluency more than students who did not take part in the intervention (Schneider et al., 2016). The current study was designed to substantiate these findings, looking specifically at the impact of the MVRC intervention on children's reading fluency.

Overview of the Current Study

In order to investigate the effect of MVRC on reading fluency, we focused specifically on elementary-school students. These children are old enough to complete the reading fluency test, and they often have gaps in phonics and basic grammar skills. Thus, these grades are ideal to investigate the link between foundational skills and reading fluency. Three Midwestern public-school districts participated. District 1 provided data from elementary schools that used the MVRC program in Grades 2 to 6. Districts 2 and 3 participated in a quasi-experimental design: Six 2nd-grade classrooms and nine 4th-grade classrooms were randomly assigned to one of three conditions: (1) instruction as usual, (2) instruction with an alternative online reading program, and (3) instruction with MVRC.

For each district, we used three assessments from the MVRC screener, administered to all children before and after the intervention, independently of grade, school, or condition: reading fluency, phonics, and listening vocabulary. Reading fluency was considered the target measure, given MVRC's explicit purpose to improve reading fluency. In contrast, the measures of phonics and listening vocabulary were considered control measures. The decision to treat phonics and listening vocabulary as control measures (vs. as mediating or moderating variables) allowed us to account for idiosyncratic aspects of the assessment and thus provide a strong test of the claim that MVRC affects reading fluency.

District 1: Correlational Design With Grades 2-6

Our first question was whether children's gain in reading fluency is related to the amount of the time they spend actively using the MVRC program.

Method

Description of the School District, Schools, and Students

District 1 serves about 20,000 students, approximately 60% of whom identify as Caucasian. For the year in which our data were collected, over two thirds of the district's schools were rated as “D” or “F” schools by the state's Department of Education. Data were obtained from a total of 29 schools (N = 2,531 students). No information was provided about the gender, race, ethnicity, or economic level of the students from these schools.

Description of the Measures

Three measures were of specific interest, all of which are returned by the MVRC screener. Reading Fluency is the target measure: It is assessed in two steps, each time involving a text and a multiple-choice comprehension test. Texts are chosen randomly from an assortment of stories of a given Lexile (MetaMetrics, Inc.). The first text is presented one page at a time, with words disappearing at a predetermined rate. This forces students to read faster than the rate at which the words disappear. Depending of students' comprehension score obtained for this first text, the second text is presented either in meaningful chunks (if students performed poorly on the first comprehension test), or it is presented in its entirety and then removed after a certain amount of time. Students' comprehension scores are used to determine their effective reading rate, transformed into a grade-equivalence score.

Phonics is our first control measure: The assessment examines the student's knowledge of English spelling rules. Nonsense words are enunciated by a virtual speech pathologist, and the student is asked to type part or all of the word on the keyboard. Listening Vocabulary is the second control measure. The assessment consists of a series of unrelated questions (e.g., “Which of these would you do on a ship?”). For each question, students have to choose among several answer options. For the example question above, answer options include “sail,” “cut,” and “roar.” Both the phonics test and the listening vocabulary test are adaptive: The difficulty level of items is adjusted based on the students' ongoing level of success during the assessment.

Procedure

Students were given access to MVRC for an entire school year. The pre-test assessment took place during the Fall semester (September or October), and the post-test assessment took place during the Spring semester (April or May, depending on school). All of the students with these two data points were included in the analysis.

Results and Discussion

All results were analyzed by grade level to account for grade-relevant discrepancies in assessments (e.g., Lexile band). Table 2 provides the descriptive statistics of pre-test data, including number of students, separated by grade level. Note that there were more 5th- and 6th-graders in our sample than students of lower grades. Note also that the average reading fluency at pre-test was below the students' actual grade level. Specifically, 3rd-graders were approximately one grade level behind, on average, and students in all other grades were more than two grade levels behind, on average. This finding is consistent with the low state ratings of the schools.

Table 2

Grade	N	Pre-test scores
		Fluency^†	Phonics	Listening vocabulary
2	242	0.46 (0.75)	3.98 (1.07)	4.69 (2.05)
3	290	1.05 (0.92)	4.60 (1.23)	6.47 (2.07)
4	316	1.73 (1.36)	4.62 (1.38)	7.47 (2.09)
5	880	2.62 (1.60)	5.27 (1.48)	8.50 (1.95)
6	803	3.90 (1.99)	5.60 (1.54)	9.32 (1.79)

Average pre-test scores obtained for District 1, separated by grade.

The three measures were obtained from the MVRC Screener. Values in parentheses reflect standard deviations.

†

The fluency measure returns a score that reflects grade equivalence.

For each grade level, we turned the continuous measure of MVRC time into discrete quartiles: Children in Quartile 1 spent the shortest time on MVRC, and children in Quartile 4 spent the longest time on MVRC. This made it possible to account for distributional inconsistencies in MVRC time and to obtain a more detailed picture of the effect of MVRC time on children's performance (e.g., non-linear patterns). Absolute improvement in fluency, phonics, and listening vocabulary served as dependent variables (post-test performance minus pre-test performance)². Table 3 provides the average number of hours on MVRC for each quartile. It also provides the average improvement in phonics, the average improvement in listening vocabulary, and the average fluency at pre-test (separated by quartile).

Table 3

	Quartile 1	Quartile 2	Quartile 3	Quartile 4
2nd-Grade (N = 60 or 61 per column)
MVRC Hours	35 (7.90)	48 (3.08)	59 (3.40)	70 (4.85)
Δ Phonics	0.62 (1.08)	0.70 (1.39)	0.75 (1.25)	1.00 (1.07)
Δ Listening vocabulary	0.88 (2.35)	1.13 (2.61)	0.92 (2.35)	1.35 (2.08)
Pre-test Fluency^†	0.17 (0.49)	0.44 (0.82)	0.43 (0.64)	0.80 (0.84)
3rd-Grade (N = 72 or 73 per column)
MVRC Hours	30 (5.85)	43 (3.71)	55 (4.65)	75 (8.12)
Δ Phonics^*	0.11 (1.52)	0.08 (1.73)	0.36 (1.50)	0.86 (1.15)
Δ Listening vocabulary	−0.13 (2.34)	0.34 (2.35)	−0.40 (2.76)	0.63 (2.50)
Pre-test Fluency^†	0.88 (0.95)	0.85 (0.77)	0.97 (0.80)	1.51 (1.71)
4th-Grade (N = 79 per column)
MVRC Hours	25 (6.09)	41 (3.95)	55 (4.83)	74 (9.07)
Δ Phonics	0.44 (1.56)	0.29 (1.70)	0.29 (1.59)	0.70 (1.62)
Δ Listening vocabulary	−0.86 (2.83)	−0.25 (2.42)	−0.81 (2.99)	−0.48 (2.37)
Pre-test Fluency^†	1.54 (1.41)	1.81 (1.38)	1.54 (1.30)	2.99 (1.37)
5th-Grade (N = 220 per column)
MVRC Hours	17 (6.30)	35 (3.88)	46 (2.99)	60 (7.20)
Δ Phonics^*	−0.09 (1.72)	0.08 (1.68)	−0.06 (1.75)	0.50 (1.43)
Δ Listening vocabulary^*	−0.70 (2.63)	−0.55 (2.70)	−0.53 (2.47)	0.07 (2.11)
Pre-test Fluency^†	2.40 (1.63)	2.65 (1.62)	2.75 (1.61)	2.68 (1.53)
6th-Grade (N = 200 or 201 per column)
MVRC Hours	26 (7.43)	40 (2.67)	49 (2.19)	60 (6.71)
Δ Phonics	−0.12 (1.87)	−0.02 (1.96)	−0.02 (1.84)	0.34 (1.59)
Δ Listening vocabulary	−0.65 (2.58)	−0.39 (2.27)	−0.57 (2.31)	−0.16 (2.20)
Pre-test Fluency^†	3.51 (2.22)	4.05 (1.93)	4.00 (1.89)	4.04 (1.88)

District 1 averages separated by grade, quartile, and type of measure.

Averages pertain to number of hours students were logged in, the difference between pre-test and post-test (Δ) in phonics and listening vocabulary, and pre-test fluency. Quartiles were determined on the basis of number of MVRC hours. Values in parentheses reflect standard deviations.

Significant difference among quartiles (two-tailed one-factor ANOVA; p < 0.05).

†

The fluency measure returns a score that reflects grade equivalence.

Second-Grade Findings

Figure 1 illustrates the 2nd-grade improvements in reading fluency, separated by quartiles of data (N = 60 or 61 per quartile). A one-way ANOVA revealed a significant effect of MVRC time on reading fluency, F_{(3, 238)} = 3.38, p < 0.05, η² = 0.04. Bonferroni's post-hoc pair-wise comparisons reveal a significant difference between the first quartile (M_{MVRC time} = 36, M_{Fluency Improvement} = 0.83) and the last quartile (M_{MVRC time} = 71, M_{Fluency Improvement} = 1.38), p < 0.05. By comparisons, improvement in neither of the two control measures (phonics, listening vocabulary) was affected by MVRC time, ps > 0.34.

Figure 1

Third-Grade Findings

Figure 2 illustrates the 3^rd-grade improvements in reading fluency, separated by quartiles of data (N = 72 or 73 per quartile). Mimicking our findings with 2nd-graders, a one-way ANOVA revealed a significant effect of MVRC time on improvements in reading fluency, F_{(3, 286)} = 4.68, p < 0.01, η² = 0.05. The effect stems from significant differences between the first (M_{MVRC time} = 30, M_{Fluency Improvement} = 1.03) and last quartile (M_{MVRC time} = 75, M_{Fluency Improvement} = 1.69), as well as between the third (M_{MVRC time} = 55, M_{Fluency Improvement} = 1.18) and last quartile, Bonferroni's ps < 0.05. MVRC time had a significant effect on phonics improvements, F_{(3, 286)} = 4.21, p < 0.01, η² = 0.04. However, MVRC time did not affect listening vocabulary, p > 0.06, η² = 0.03.

Figure 2

Fourth-Grade Findings

Figure 3 illustrates the 4th-grade improvements in reading fluency, separated by quartiles of data (N = 79 per quartile). Unlike what we found for 2nd- and 3rd-graders, a one-way ANOVA revealed no significant effect of MVRC time on reading fluency, p > 0.65, η² < 0.01. This result did not change when we implemented the statistically more powerful linear contrast. Thus, for 4th-grade, time on MVRC did not have the expected effect. Neither phonics nor listening vocabulary changed as a function of MVRC time, ps > 0.35.

Figure 3

Fifth-Grade Findings

Figure 4 illustrates the 5th-grade improvements in reading fluency, separated by quartiles of data (N = 220 per quartile). Mimicking our findings with 2nd- and 3rd-graders, a one-way ANOVA revealed a significant effect of MVRC time on fluency improvement, F_{(3, 876)} = 15.64, p < 0.01, η² = 0.05. Bonferroni's post-hoc pair-wise comparisons reveal significant differences between the first (M_{MVRC time} = 18, M_{Fluency Improvement} = 1.02) and second quartile (M_{MVRC time} = 35, M_{Fluency Improvement} = 1.49), as well as between the second and third quartile (M_{MVRC time} = 47, M_{Fluency Improvement} = 1.89), ps < 0.05. The first quartile also differed from the last quartile (M_{MVRC time} = 60, M_{Fluency Improvement} = 1.77), p < 0.05.

Figure 4

For 5th-graders, MVRC time also affected changes in the two control measures. Specifically, in phonics, F_{(3, 876)} = 6.07, p < 0.01, η² = 0.02, the first quartile (M_{PhonicsImprovement} = −0.09) and the second quartile (M_{PhonicsImprovement} = 0.08) differed significantly from the last quartile (M_{PhonicsImprovement} = 0.50), Bonferroni ps <0.05. And in listening vocabulary, F_{(3, 876)} = 4.12, p < 0.01, η² = 0.01, the first quartile (M_{VocabImprovement} = −0.70) differed significantly from the last quartile (_{VocabImprovement} = 0.07), Bonferroni ps < 0.05.

Given the high number of 5th-graders, we were able to analyze improvement in reading fluency as a function of school. Five of the schools had sufficiently high numbers of students (N > 100). Of these five school, we decided against including two of them, given knowledge about special circumstances (e.g., one was a summer school). Figure 5 illustrates the improvements in reading fluency for the three remaining schools. Interestingly, improvement in reading fluency differed by school, even though we controlled for the amount of time students spent on the MVRC program, one-way ANCOVA F_{(3, 376)} = 10.94, p < 0.01, η² = 0.08. Thus, even though students were at the same grade level, and presented with the exact same online reading tool for similar durations, the degree to which they benefited from it differed (see Table 4 for additional information).

Figure 5

Table 4

Measure	School A	School B	School C	F
	N = 110	N = 132	N = 138
Δ Phonics	0.26 (1.67)	0.40 (1.40)	0.05 (1.59)	1.76
Δ Listening vocabulary	−0.34 (2.57)	−0.22 (2.35)	−0.36 (2.74)	0.12
Pre-test Fluency^†	2.42 (1.68)	2.58 (1.51)	3.11 (1.54)	6.81^*

Averages obtained for District 1 5th-graders, separated by measure and school.

Δ = Difference between pre-test and post-test. Values in parentheses reflect standard deviations.

p < 0.05.

†

The fluency measure returns a score that reflects grade equivalence.

Sixth-Grade Findings

Figure 6 illustrates the improvements in reading fluency found for 6th-grade, separated by quartiles of data (N = 200 or 201 per quartile). Effects of MVRC time on reading fluency were comparable to those obtained for 5th-graders, F_{(3, 799)} = 4.26, p < 0.01, η² = 0.02. Bonferroni's post-hoc pair-wise comparisons reveal significant differences between the first (M_{MVRC time} = 24, M_{Fluency Improvement} = 1.21) and second quartile (M_{MVRC time} = 40, M_{Fluency Improvement} = 1.48), as well as between the first and third quartile (M_{MVRC time} = 49, M_{Fluency Improvement} = 1.69), ps < 0.05. The first quartile also differed from the last quartile (M_{MVRC time} = 60, M_{Fluency Improvement} = 1.65), p < 0.05. MVRC time did not affect improvements in phonics or listening vocabulary ps > 0.06.

Figure 6

In sum, we found robust effects of MVRC time on improvement in reading fluency, consistently more so than on improvements in phonics or in listening vocabulary. For example, while MVRC time affected reading fluency in Grades 2, 3, 5, and 6, it affected listening vocabulary only in Grade 5. This demonstrates that the time of MVRC practice operates as designed, affecting primarily students' reading fluency. However, specifics of the school appear to matter. Incidentally, the school that benefited the most from MVRC practice was rated as “D” school by the state's Department of Education. In contrast, the two schools that benefited least from MVRC were rated as “F” school.

District 2: Quasi-Experiment With Grade 2 Students

Our next question was whether the positive effect of the MVRC program holds up in a randomized control trial. District 2 made such a research design possible with 2nd-grade classrooms. At this grade level, some children might have made the transition to reading for comprehension, while other children might still be focused on decoding and the mechanics of reading. Thus, we can shed light on the effect of the MVRC during this transitional time.