Abstract
Introduction:
Utility value interventions are assignments designed to increase student motivation by helping them find personal value in what they are learning. Previous studies have found utility value interventions to lead to improved student outcomes. The purpose of this study was to investigate the impact of a utility value (UV) intervention on the academic outcomes of undergraduate students in online courses, compared to students in a control condition.
Methods:
A total of 1,243 students in upper-level online biology courses participated and were randomized either to UV (n = 589) or control (n = 654) conditions.
Results:
The intervention had a positive effect, as UV participants engaged with and participated in their courses significantly more than students in the control condition. Utility value participants also wrote significantly longer essays, and used more pronouns, social process words, and cognitive process words than students in the control condition. However, UV participants did not have higher course grades than students in the control condition. Additionally, underrepresented minority (URM) status was a significant predictor of course grade; non-URM students had significantly higher course grades than URM students.
Discussion:
The utility value intervention did have a positive effect on student course engagement but, contrary to most previous studies, did not lead to higher student course grades, perhaps because these classes were online, because they were upper-level instead of introductory, or because the study was conducted during the COVID-affected spring 2020 quarter. Findings from this study can help institutions decide whether to adopt UV interventions in online courses.
Introduction
How can educators best help their students learn? One approach has been to try to understand at a fundamental level how people learn (i.e., change their concepts), with the hope that this understanding will give insights into how to most effectively facilitate learning. suggested that conceptual change can happen when an individual becomes dissatisfied with the current concept and learns about a potential replacement concept that is plausible, intelligible, and fruitful. Conceptual change in this model is viewed entirely as a cognitive process, and this model has thus been called “cold” conceptual change. Subsequently, researchers recognized that psychological and social factors can play important roles in facilitating or preventing conceptual change, leading to “warm” models of conceptual change (; ; ). For example, in the recently proposed Dynamic Model of Conceptual Change (), social and psychological factors are postulated to affect a learner's motivation to engage with a new concept; a learner's deep engagement with a concept in turn is essential for real conceptual change to take place. Situated Expectancy-Value Theory proposes that a learner's motivation to engage in an academic task is determined by a combination of the learner's expectancy for success (self-efficacy) and the task value that the learner places on the subject (; Wigfield and Eccles, 2000). Task value is differentiated into four components: utility value (usefulness for near- or long-term goals), attainment value (importance of a task because it is aligned with one's identity), intrinsic value (interest and enjoyment), and cost (especially the effort involved).
Researchers have empirically tested warm models of conceptual change by investigating whether targeting social and psychological processes with brief interventions can facilitate learning. These social-psychological interventions have in general shown good success in improving student outcomes in both K-12 and higher education contexts (; ; ; ). For example, a meta-analysis of 92 intervention studies found an average effect size, including both academic and psychological effects, of 0.49 or about half a standard deviation (). This compares with an average effect size on student academic performance of the adoption of active learning practices of 0.47 (based on a meta-analysis of 158 studies; ). The effects of social-psychological interventions are thus relatively large by the standards of educational interventions, especially given that they generally require modest investments of time and effort compared to many other types of educational reforms (). The strongest effects of the interventions are often among disadvantaged students such as underrepresented minorities and first-generation college (; ), a differential effect also seen for active learning ().
Social-psychological interventions can be classified according to two parameters (): (1) The psychological processes targeted in students: how students value academic tasks (task value interventions), how students frame academic challenges (framing interventions), or students' personal values (personal values interventions); and (2) The levels at which these interventions work: a specific course, a field of study, or school general. Utility value interventions are a type of task value course-specific intervention designed to increase the task value that students place on a course or subject. Several forms of the intervention have been tested, including telling students why a subject may be valuable (), assigning students to write essays about how the subject may be valuable personally or to others (), or assigning them to evaluate and respond to quotations that describe reasons a subject may be valuable (; ). Tests of the essay-based utility value intervention (the most frequently-used format) have also attempted to define the optimum delivery parameters by varying the frequency and timing of the assignment (), varying the focus of the essay (value to self vs. value to others; ), testing the effects of student choice and variety in the focus of the assignments (; ), and testing the effects of increasing the frequency of connections students make between the material and their lives (). The somewhat complicated results of these studies, especially when the effects on different groups of students are considered, lead to the overall recommendation that three administrations of the utility value intervention in a semester, beginning with a self-focused essay followed by an other-focused essay and with some choice, is likely to give the best outcome for the greatest number of students. In most tests of the utility value intervention, students with low prior academic achievement (in the course, or in high school) do better academically in the course after receiving the intervention. The observation that a single utility value intervention can often improve a student's grade point average (GPA) and retention in a STEM major up to three years later (; ) suggests that success in the course can in turn increase a student's sense of self-efficacy and motivation, and thereby increase their success in subsequent classes, creating a virtuous cycle. Indeed, many studies that have directly measured various psychological factors have found increases in interest in a subject, career motivation, and competence-related beliefs among students who received a utility value intervention, especially lower-performing students or those with lower performance expectations (; ; ; ; ; ; ; ).
Despite the many studies that report encouraging and even impressive results for social-psychological interventions, there is some evidence in the literature that outcomes can be variable and sensitive to context. For example, one study of > 7,000 students at a flagship state university who were given either a growth mindset or social belonging intervention found that both interventions led to increased full-time enrollment through the entire first year for students from socially and economically disadvantaged backgrounds (Yeager et al., 2016). By contrast, a study of over 6,000 incoming first year students at a different major state university given either a growth mindset or social belonging intervention found that the social belonging intervention had no effect on first year GPA for any group and that the growth mindset intervention resulted in an improvement in GPA only for Latino/a students and not for African American students (). Additionally, administered a social belonging intervention to 26,911 students across 22 post-secondary institutions in the US and found that the intervention improved students' progress in university, especially among those who belonged to groups that historically progressed at lower rates and at institutions perceived as providing a supportive environment. As another example, a study of 798 students in introductory biology given two “doses” of a values affirmation intervention found that the gap between first generation and continuing generation students in course grades and semester GPA was narrowed by 50% (). However, a study of over 2,000 students given a values affirmation and/or a utility value intervention by the same research group in the same class found that the values affirmation intervention had no effect on student outcomes (). The utility value intervention was found to have a small positive effect on GPA for all students, but the strongest effect was for students who were both first generation and underrepresented minority (URM; ). The authors suggest that the greater achievement gap observed for students in the first study made it easier to detect the positive effects of the values affirmation intervention as compared to the second study. All of the studies cited above used a randomized controlled design, which has been found in a meta-analysis to give an average effect size for social-psychological interventions of 0.43 (based on 64 studies) compared to an effect size for quasi-experimental studies of 0.64 (based on 28 studies; ).
Almost all of the studies of social-psychological interventions have been done in traditional courses in which instruction is provided face-to-face. However, online education has become a major part of the overall higher education marketplace, especially for non-traditional students who value the flexibility that asynchronous courses afford those with work and family obligations (). Statistics from the US in Fall 2021 show that about 61% of all post-secondary students took at least one course online, and 28% took exclusively online courses (). A “mega-analysis” summarizing 16 meta-analyses on the effectiveness of distance education (i.e., looking at student learning outcomes) concluded that online learning overall seems to be as or more effective than traditional educational approaches (), but this conclusion is not universally accepted and may not be true for all types of students in all contexts (; Xu and Jaggars, 2013). Some evidence suggests, for example, that students with various types of disadvantages at community colleges may do worse in online classes than in traditional classes (Xu and Jaggars, 2013), and students at a large for-profit university who had the option of taking classes either in-person or online did worse when they took the classes online (). Many studies have been carried out of student behavior in online classes. Among the findings are that student self-regulated learning strategies of time management, metacognition, effort regulation, and critical thinking are effective in online classes (). Students with greater intrinsic motivation tend to use more productive learning strategies, whereas students with high extrinsic motivation (i.e., performance focused) tend to use more superficial and less productive study strategies (e.g., memorization, re-reading; ). Furthermore, students' utility value for a course and their self-efficacy have been found to be associated in online courses with productive study strategies () and academic success (). The connections between students' motivations, use of productive study strategies, and academic outcomes online suggest that social-psychological interventions might be useful in promoting student success in online courses.
The decisions by many universities to move their classes online in response to the COVID-19 outbreak dramatically increased the number of students in online courses. This presented an opportunity to test if social-psychological interventions, and more specifically utility value interventions, might help improve student success in such classes. Therefore, the overall aim of this study was to investigate the impact of a utility value intervention on the academic outcomes of undergraduate students in online courses during the COVID-19 pandemic, compared to students in a control condition. Specifically, the purpose of this study was fourfold: (1) to assess differences in course performance in general and among URM students between groups (utility value vs. control; primary); (2) to assess differences in student engagement with course material via page views and participation between groups (secondary); (3) to determine whether socio-demographic variables (i.e., gender, URM status) predicted course grades among utility value and control participants (tertiary); and (4) to explore differences in essay content between conditions (exploratory).
Methods
Study design
This was a cross-sectional randomized controlled trial. The intervention occurred during the 10-week Spring quarter of 2020 at a 4-year public university in the United States. Ethics approval was obtained from the Institutional Review Board (#170886).
Participants
A total of 1,243 students participated in the study (n = 589 utility values; n = 654 control). Not all participants were included in the analyses due to the way by which data were collected and anonymized, as well as missing data. Most participants identified as women (n = 836; 68.40%) and did not identify as a URM (n = 948, 77.30%). It is worth noting that students who identified as Asian or White were considered non-URM in this study. For context, in Fall 2019, 41.3% of undergraduate students enrolled in Life Sciences–including Biological Sciences–at the host institution identified as Asian, 22.9% identified as Hispanic/Latino(a), 21.9% White, 8.3% International, 2.9% African American, 2.2% Domestic Unknown, 0.4% American Indian, and 0.1% Native Hawaiian/Pacific Islander (). Additionally, 38.5% of undergraduate students enrolled in Life Sciences identified as first generation (). Full demographic details by group can be found in Table 1.
Table 1
| Participant characteristics (n = 1,226)a | Utility value (n = 583) | Control (n = 643) | ||
|---|---|---|---|---|
| n | % | n | % | |
| Gender | ||||
| Woman | 402 | 68.95 | 437 | 67.96 |
| Man | 181 | 31.05 | 205 | 31.88 |
| Not specified | – | – | 1 | 0.15 |
| Minority status | ||||
| URM | 138 | 23.67 | 140 | 21.77 |
| Non-URM | 445 | 76.33 | 503 | 78.23 |
Demographic information.
aThe total sample size was 1,243 participants; not all categories summed to equal the total sample due to missing data.
URM, underrepresented minority.
Study procedures
A total of five instructors, representing four upper-year courses in biological sciences, agreed to participate in the study. The courses were in the areas of genetics (two lecture sections), genomics, evolution, and human reproduction, and primarily enrolled third and fourth year students. All of the courses had required prerequisites of one semester of introductory cell biology and one semester of introductory organismic and evolutionary biology. The human reproduction course also required a semester of introductory human physiology as a prerequisite. The courses generally used a mix of lecture and active learning instructional strategies. Assessments and grading schemes varied by course and can be found in Appendix. Students in these courses were given the opportunity to opt-out of participating in the study. Instructors informed their students through the learning management system (LMS) that: (1) their data could be used in educational studies; (2) their data would be anonymized before analysis; and (3) there were no risks to them participating. Students were provided with a link to an opt-out form for those who did not wish to participate. Students who did not complete the opt-out form were assumed to have given consent for their data to be used in the study. Each of the courses had multiple recitation sections (from 2 to 11 per lecture section). The recitation sections were randomly assigned to receive either the utility value intervention or a control “course material summary” intervention (details of both are given below), so all students in a recitation section received the same intervention. An independent-samples t-test was run to determine if there were differences in prior preparation using cumulative GPA between control and intervention conditions. There was no significant difference in cumulative GPA between conditions, t(1, 224) = 0.85, p = 0.39. Two assignments (for both control and interventions groups) were administered in each course and delivered online through the LMS. The dates of the two assignments were selected in consultation with the instructors to minimize conflicts with scheduled exams and therefore differed for each course. Students had 5 days to complete each assignment and submit it online. Because the institution has a quarter system with 10-week quarters, two writing assignments were given rather than the three recommended for a 15–16 week semester (). Instructors were blind to whether students received the intervention or the control treatment.
The intervention
The utility value intervention writing assignments were both self- and other-focused (i.e., adapted from ). Students were asked to select a concept covered in the course, formulate a question, answer the question, and explain how the topic applied to their life and to helping others. The prosocial utility value intervention was selected as the students participating in the intervention were in STEM disciplines, which offer a powerful pathway for advancing prosocial goals, with meaningful impact on public health, environmental sustainability, and overall human wellbeing (). Demonstrating the connection between STEM coursework and students' prosocial aspirations can enhance engagement, boost motivation, and improve academic performance (). Students in the control condition were similarly asked to select a concept covered in the course and formulate a question; however, instead of explaining how the topic applied to their life and others, control participants were asked to answer the question using information from class notes and readings. The same writing assignments were given for writing assignment 1 and 2 across all courses. The assignments were graded for scientific content, writing quality, and following directions. Instructors were asked to award a small amount of credit for correct completion of each assignment to increase compliance with the intervention. Full intervention and control writing assignments can be found in the Supplementary material.
Data collection
The following data were collected on participants from the LMS at the end of the quarter: overall course grade, page views, and participation. Page views were counted by the number of times pages in the LMS were loaded. Student participation was measured as the number of certain actions students took within a course, specifically posting in the discussion forum and submitting an assignment or quiz. Page views and participation are two of many standard measures of general student engagement in studies using LMS log data (). Additionally, students' grades and socio-demographic characteristics (i.e., cumulative GPA, gender, and URM status) were obtained from the university's Registrar's Office. Lastly, students submitted their writing assignments via the LMS, which were analyzed using Linguistic Inquiry and Word Count (LIWC) software. LIWC software “connects important psychosocial constructs and theories with words, phrases, and other linguistic constructions” (, p. 2). LIWC consists of target words and dictionary words (). Target words refer to the words in essays that are read and analyzed by LIWC, while dictionary words are those contained in the LIWC dictionary file (). The LIWC software assesses text within an essay and compares it against the LIWC dictionary “by counting all of the words in a target text, then calculating the percentage of total words that are represented in each of the LIWC sub-dictionaries” (, p. 3). For the purpose of this study, five dictionaries/sub-dictionaries were selected for analysis: (1) word count (i.e., total word count of the essay); (2) personal pronouns–1st person singular words (e.g., I, me, my, myself); (3) personal pronouns–2nd person words (e.g., you, your, u, yourself); (4) cognitive processes (e.g., but, not, if, or, know); and (5) social processes (e.g., you, we, he, she). The dictionaries selected were based on LIWC dictionaries frequently used in utility value studies (; ).
Data analysis
To assess differences in course performance between groups (control vs. intervention) an independent samples t-tests were conducted and to examine the effects of URM status and condition on course grades and GPA, a series of two-way ANOVAs were conducted (primary purpose). To assess differences in student engagement with course material via course website page views and course participation between groups, independent samples t-tests were conducted (secondary purpose). Where there were unequal variances between groups, Welch's t-tests were used. To determine whether socio-demographic variables (i.e., gender, URM status) predicted course grades among utility value and control participants, multiple linear regression was run (tertiary purpose). Lastly, to explore differences in essay content between conditions, Linguistic Inquiry and Word Count (LIWC) text analysis was used (exploratory purpose). The LIWC scores were averaged across the two writing assignments, yielding a total score for each dictionary/sub-dictionary and t-tests were conducted to compare average scores between conditions (utility value vs. control).
It is worth noting that in order to obtain students' grades and socio-demographic characteristics for analysis (i.e., cumulative GPA, gender, and URM status) data were sent to a central unit at the university and were returned anonymized per procedures detailed in the Institutional Review Board protocol. In doing this, the Registrar's Office removed data pertaining to page views and course participation. As such, two datasets were used for analysis: one consisting of the data collected prior to it being sent to the Registrar's Office (inclusive of condition, numerical course grade, page views, participation, and LIWC variables); and a second, anonymized, dataset that was returned from the Registrar's Office (inclusive of condition, gender, URM status, letter course grade, and cumulative GPA).
Results
To determine if there were significant differences in writing assignment completion between utility value and control conditions, chi square tests of association were conducted. A total of 1,003 participants completed writing assignment 1 (n = 473 utility values; n = 530 control) and 992 completed writing assignment 2 (n = 478 utility values; n = 514 control). Overall, there were no significant differences between conditions in writing assignment completion for assignment 1 [χ2(1) = 0.11, p = 0.74] or assignment 2 [χ2(1) = 1.15, p = 0.28].
Primary purpose
An independent samples t-test was run to determine if there were differences in course grade between utility value and control group participants in general. There were 557 utility value participants, and 620 control group participants included in the analysis. Utility value participants had slightly higher course grades (88.98 ± 7.41)1 than control group participants (88.60 ± 7.72); however, there was not a statistically significant difference between groups, Mdifference = 0.38 (95% CI, −0.48 to 1.25), t(1, 175) = 0.87, p = 0.39, d = 0.05.
A two-way ANOVA was conducted to examine the effects of URM status and condition on course grades. As explained in Methods, only course letter grades, not numerical grades, were available for students with demographic information attached. Therefore, in order to assess differences, the letter grades were converted to numerical grade points using the university's 4-point scale ranging from 0 (fail) to 4 (A, A+). The grading system did not have a numerical equivalent for “no pass” (NP) and “pass” (P) and a relatively small number of students (n = 118) received grades of NP or P. As such, students with these letter grades were not included in the analysis. There was no statistically significant interaction between URM status and condition for course grades, F(1, 1, 104) = 0.057, p = 0.81, partial η2= 0.00. Therefore, an analysis of the main effects for URM status and condition were performed. There was a statistically significant main effect of URM status on course grades, F(1, 1, 104) = 45.00, p = < 0.001, partial η2= 0.039. Specifically, non-URM students had significantly higher course grades than URM students (Mdifference = 0.23; 95% CI, 0.16–0.30). There was no statistically significant main effect of condition on course grades, F(1, 1, 104) = 1.45, p = 0.23, partial η2= 0.001.
Similar to the above, a two-way ANOVA was conducted to examine the effects of URM status and condition on cumulative GPA. There was no statistically significant interaction between URM status and condition for cumulative GPA, F(1, 1, 222) = 0.36, p = 0.55, partial η2= 0.00. There was a statistically significant main effect of URM status on cumulative GPA, F(1, 1, 222) = 157.77, p = < 0.001, partial η2= 0.11. Similar to course grades, non-URM students had significantly higher GPAs than URM students in both conditions (Mdifference = 0.35; 95% CI, 0.29–0.40). There was no statistically significant main effect of condition on cumulative GPA, F(1, 1, 222) = 0.45, p = 0.50, partial η2= 0.00.
Secondary purpose
Page views
A Welch t-test was run to determine if there were differences in student engagement with course material via course website page views between utility value and control groups. There were 563 utility value participants, and 630 control group participants included in the analysis. The utility value group engaged with the course website course more (858.17 ± 405.29) than the control group (797.88 ± 347.69). A statistically significant difference of 60.29 (95% CI, 17.14–103.44), t(1, 113.86) = 2.741, p = 0.006, d = 0.16 was found between groups.
Participation
An independent samples t-test was run to determine if there were differences in student engagement with course material via course participation (posting to LMS discussions and submitting assignments and assessments on the LMS) between utility value and control groups. There were 566 utility value group participants, and 629 control group participants included in the analysis. The utility value group participated in the course more (40.49 ± 9.09) than the control group (38.86 ± 8.81). A statistically significant difference of 1.63 (95% CI, 0.61–2.65), t(1, 193) = 3.15, p = 0.002, d = 0.18 was found between groups.
Tertiary purpose
Multiple linear regression was run to understand the effect of gender and URM status on course grades. The overall regression, including both predictors, was statistically significant (R = 0.20, R2 = 0.039, adjusted R2 = 0.037, F = 22.50, p = < 0.001; see Table 2 for full regression results). Gender and URM status accounted for 3.9% of the variation in course grade. One significant predictor was identified, URM status (b = 0.23, p = < 0.001).
Table 2
| Predictors | b | β | CI 95% (lower, upper) | Zero-order r | sr |
|---|---|---|---|---|---|
| Intercept* | 3.44 | 3.28, 3.60 | |||
| Gender | −0.014 | −0.013 | −0.073, 0.046 | −0.025 | −0.014 |
| URM Status* | 0.23 | 0.20 | 0.16, 0.30 | 0.20 | 0.20 |
Multiple linear regression of gender and underrepresented minority status predicting course grade.
*p < 0.001.
b, unstandardized regression coefficient; β, standardized regression coefficient; CI, confidence interval; r, correlation; sr, semi-partial correlations; URM, underrepresented minority.
Exploratory purpose
Participants who completed both writing assignments were included in the exploratory analyses, as data from essays 1 and 2 were averaged for each dictionary/sub-dictionary.
An independent samples t-test was run to determine if there were differences in essay length (i.e., total word count) between utility value and control group participants. There were 434 utility value participants, and 482 control participants included in the analysis. The utility value group wrote longer essays (503.01 ± 117.44) than the control group (447.04 ± 109.67). A statistically significant difference of 55.96 (95% CI, 41.23–70.69), t(914) = 7.46, p < 0.001, d = 0.49 was found between groups.
A Welch t-test was run to determine if there were differences in the use of personal pronouns–1st person singular words between utility value and control group participants. There were 434 utility value participants, and 482 control participants included in the analysis. The utility value group used 1st person words (1.13 ± 0.71) more than the control group (0.25 ± 0.52). A statistically significant difference of 0.88 (95% CI, 0.80–0.97), t(787.69) = 21.25, p < 0.001, d = 1.52 was observed between groups.
A Welch t-test was run to determine if there were differences in the use of personal pronouns–2nd person words between utility value and control group participants. There were 434 utility value participants, and 482 control participants included in the analysis. The utility value group used 2nd person words (0.76 ± 0.66) more than the control group (0.15 ± 0.42). A statistically significant difference of 0.61 (95% CI, 0.54–0.68), t(721.21) = 16.60, p < 0.001, d = 1.24 was found between groups.
An independent samples t-test was run to determine if there were differences in use of social process words between utility value and control group participants. There were 434 utility value participants, and 482 control participants included in the analysis. The utility value group used words such as “you, we, he, and she” (6.90 ± 2.11) more than the control group (4.53 ± 2.02). A statistically significant difference of 2.38 (95% CI, 2.11–2.64), t(914) = 0.18, p < 0.001, d = 0.01 was observed between groups.
A Welch t-test was run to determine if there were differences in use of cognitive process words between utility value and control group participants. There were 434 utility value participants, and 482 control participants included in the analysis. The utility value group used words such as “but, not, if, or, and know” (13.36 ± 2.13) more than the control group (11.78 ± 2.53). A statistically significant difference of 1.59 (95% CI, 1.28 to 1.89), t(910.01) = 10.29, p < 0.001, d = 0.68 was found between groups.
Discussion
The aim of this study was to investigate the impact of a utility value intervention on the academic outcomes of undergraduate students in online courses during the COVID-19 pandemic, compared to students in a control condition. Results revealed that there were no significant differences in course performance between conditions; however, URM status was a significant predictor of course grade. Specifically, non-URM students had significantly higher course grades and cumulative GPAs than URM students. The utility value intervention did have significant effects in this study, as students in the utility value condition engaged with and participated in their courses significantly more than students in the control condition according to LMS data. Further, LIWC analysis revealed that utility value participants wrote significantly longer essays, and used more personal pronouns, second person pronouns, social processes words, and cognitive processes words than students in the control condition. To reduce the burden on students in this first quarter of COVID-19-related online instruction, we did not administer questionnaires to directly assess changes in psychological factors such as interest, motivation or self-efficacy.
It is not surprising that students in the utility value group engaged with and participated in their courses significantly more than students in the control group, as utility value interventions have been found to improve both students' interest in a subject and their academic outcomes (; ; ). While educators have little influence on initial student interest in course content, they play a key role in course design and implementation, which can influence whether students perceive the course to be valuable (). Notably, encouraged students to consider the relevance of their course material via a utility value intervention. Students were asked to reflect on how the course material could be applied to their lives through a writing exercise (). The authors concluded that the intervention increased perceptions of utility value, which predicted interest in course material and performance (). Similarly, tested the efficacy of a utility value intervention on improving community college students' perceived math relevance and achievement in introductory math courses. The authors concluded that students in the utility value condition perceived course material to be more relevant, and thus benefitted from this perception, compared to participants in the control condition (). Additionally, the intervention's positive effect on subject relevance was more prominent among first generation students than other demographic groups (). The evidence to support utility value interventions to improve the academic pipeline for students from diverse backgrounds is growing. Specifically, implemented a utility value intervention in an introductory chemistry course with the goal of promoting persistence and diversity in STEM. Not only did participation in the intervention improve STEM persistence in general, but the effects were largest for students from marginalized and underrepresented racial/ethnic backgrounds, who were 14% more likely to persist in STEM fields following the intervention (). It is evident that participating in utility value interventions can positively impact the academic trajectory of students, and it is therefore not surprising that course engagement and participation was significantly higher among participants in the utility value group compared to the control condition in the current study. It is plausible that by reflecting on the relevance of course material, utility value students were more invested in their learning and success compared to students in the control group, thus explaining students' increased engagement and participation.
It is interesting, however, that there were no significant differences in course grades between groups in our study, unlike in most previous studies. This might be explained, in part, by the courses from which students were recruited for the study. Notably, in previous utility value studies students were enrolled in introductory courses (; ; ); however, students in the current study were enrolled in upper-division courses. The only comparable study of which we are aware tested the utility value intervention in an online upper-level inorganic chemistry course taken primarily by third- and fourth-year students (). In that study, students who completed three utility value assignments over the semester showed no statistically significant difference in grades on exams given after these assignments (including the final exam) compared to students who completed “content summary” control assignments. These results and ours suggest the possibility that utility value interventions have less of an effect on upper-year students, perhaps because students enrolled in upper-division courses are already committed to their majors and motivated to perform well (). Additionally, the topics of these courses might have inherent utility value for students based on their trajectory (e.g., pre-healthcare profession students). Moreover, it is worth noting that the current study took place at the beginning of the COVID-19 pandemic, when grading policies were more lenient (). Therefore, it is plausible that students' grades in both conditions were higher than usual, thus explaining the lack of significance between conditions.
There were no significant differences in course grade between conditions and there was no statistically significant interaction between URM status and condition for course grade. Interestingly, these findings do not align with the literature, as utility value interventions have been found to narrow the achievement gap for URM students (). Given that the interaction between URM status and condition was not significant, results suggest that the utility value intervention did not positively impact URM students' academic performance. However, it is worth noting that URM status was found to be a significant predictor of course grade. Specifically, non-URM students had significantly higher course grades and cumulative GPAs than URM students. This is not surprising as it is well-established that URM students perform poorly academically compared to non-URM students (; ; Whitcomb and Singh, 2021). The current study's findings highlight this disparity. In the context of online learning and the COVID-19 pandemic, found that the transition to online learning significantly widened the achievement gap between URM students and students in the majority group. The widened achievement gap coupled with lenient grading policies as a result of the COVID-19 pandemic (; ) might explain why the intervention did not have a significant impact on URM students' academic performance.
The LIWC analysis revealed that the essay content of those in the utility values condition was significantly more meaningful than the essay content of students in the control condition. These findings align with the literature, as conducted a utility value intervention and concluded that the utility values writing assignments had a more personal focus (i.e., used more personal pronouns) and used more social and cognitive process words than control writing assignments. Similarly, found that students in the utility values condition used more personal pronouns and that their essays contained more social and cognitive mechanism words than students in the control condition. The results from these studies align with the current study, as utility value participants used more personal pronouns, second person pronouns, social process words, and cognitive process words than students in the control condition. Pronouns and verb tense, such as use of first and second person pronouns, can help identify focus and, in turn, highlight individual priorities, intentions, and processing (). In contrast, pronouns within social processes can demonstrate the quality of a close relationship (). For example, the social process word “we” has been found to be related to higher relationship quality compared to second person words such as “you” (). Additionally, cognitive mechanisms such as “cause, know, and ought” can indicate more complex language (). Thus, findings from the current study underscore that utility value participants were more cognitively and socially engaged in course content compared to control participants.
Recommendations for future research
The finding from the current study that a utility value intervention in online upper-level biology classes did not improve student outcomes (grades) over a control treatment, which is similar to the result obtained in an online upper-level chemistry class () and differs from the results of many previous studies carried out in introductory classes taught in person (), suggests several avenues for future research. First, are utility value interventions generally ineffective in improving student outcomes in upper-level classes, and if so, why? Second, are utility value interventions ineffective in improving student outcomes in online classes, and if so, why? Studies using utility value interventions in upper-level in-person classes and in online introductory classes that incorporate measures of various psychological factors could help to answer these two questions.
Limitations
Limitations of this study include the following: first, no psychosocial measures were administered in the current study. Though utility value interventions have demonstrated positive effects on the psychological outcomes of students in the past, in an effort to reduce participant burden, this data were not collected in the current study. Given the abrupt transition to online learning and the evolving nature of the COVID-19 pandemic, the research team purposefully limited data collection to demographics and data that could be collected from the university, to avoid students having to self-report. Future studies might consider assessing psychological and social outcomes of students participating in an online utility value intervention. Second, the sample size of URM students was relatively small. This is likely due to the demographics of the study institution's student body. Institutional data was received post-intervention on student demographics, including URM status. The research team was limited to the data that could be obtained and thus, few demographic details are reported in the current study. Given that in-person utility value interventions have been found to promote retention and persistence in STEM (), it is recommended that future research focus on implementing in-person utility value interventions for students of disadvantaged backgrounds in an effort to dismantle systemic inequities and narrow the achievement gap between URM students and their peers that was further exacerbated by the COVID-19 pandemic. Third, the current study was limited to quantitative data. Future research is encouraged to include qualitative methods to gain deeper insights into students' experiences with the intervention in online environments. Lastly, because the intervention was carried out at a single institution during one quarter only, the study findings might not be generalizable to other types of institutions and student populations. It is also worth highlighting that the current study was conducted during the COVID-19 pandemic, which may impact the generalizability of study findings. Interestingly, findings from the current study mirrored those of , who implemented a utility value intervention with upper-division students during the pandemic and similarly observed no significant effects. While this may suggest that such interventions are less impactful for upper-year students, it is also plausible that byproducts of the pandemic–such as heightened stress, disrupted learning environments, and diminished motivation–interfered with the intervention's effectiveness. It is recommended that researchers replicate the study to assess whether the findings could be generalizable to upper-division and/or online courses today.
Conclusion
The aim of this study was to investigate the impact of a utility value intervention on the academic outcomes of undergraduate students in online courses during the COVID-19 pandemic, compared to students in a control condition. Overall, the intervention had a positive effect as utility value participants engaged with and participated in their courses significantly more than students in the control group. There were no differences observed in course performance between conditions; however, non-URM students had significantly higher course grades and cumulative GPAs than URM students. Despite the fact that the intervention did not significantly impact URM students, overall, the intervention was successful as findings from the LIWC analysis underscored that utility value participants were more actively engaged with course content than control participants. Study findings demonstrate that a utility value intervention administered to upper-division classes in an online learning environment can yield positive effects, though the effects might not be as strong as in-class utility value interventions. Future research should aim to narrow the achievement gap between URM students and their non-URM peers by offering a utility value intervention specifically for students of diverse backgrounds. University personnel might consider incorporating utility values tasks into large first-year introductory online courses in an effort to intervene at a time that is crucial and positively alter the academic trajectory of URM students.
Statements
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics statement
The studies involving humans were approved by University of California, San Diego Institutional Review Board (#170886). The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
Author contributions
KS: Writing – review & editing, Writing – original draft, Formal analysis. SL: Conceptualization, Writing – review & editing, Supervision. LH: Conceptualization, Writing – review & editing, Investigation, Data curation, Supervision.
Funding
The author(s) declare that no financial support was received for the research and/or publication of this article.
Acknowledgments
This project was developed and initiated during LH's sabbatical at the School of Biological Sciences, University of California, San Diego. LH thanks Adelphi University for the sabbatical support, Drs. Stanley Lo and Mark Estelle for sabbatical hosting, and Dr. Thomas Bussey and graduate students in the Joint Doctoral Program in Mathematics and Science Education for discussions and guidance. The authors would also like to thank the instructors who assisted with recruitment, Drs. Christopher Day, Katie Petrie, Jim Cooke, Keefe Reuther, and Sarah Stockwell, as well as the participants of the study.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declare that no Gen AI was used in the creation of this manuscript.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2025.1636446/full#supplementary-material
Footnotes
1.^Data are mean ± standard deviation, unless otherwise stated.
References
1
AceeT. W.WeinsteinC. E. (2010). Effects of a value-reappraisal intervention on statistics students' motivation and performance. J. Exp. Educ.78, 487–512. 10.1080/00220970903352753
2
AhmadiG.MohammadiA.AsadzandiS.ShahM.MojtahedzadehR. (2023). What are the indicators of student engagement in learning management systems? A systematized review of the literature. Int. Rev. Res. Open Distribut. Learn.24, 117–136. 10.19173/irrod1.v241.6453
3
AlonL.SungS.ChoJ.KizilcecR. F. (2023). From emergency to sustainable online learning: Changes and disparities in undergraduate course grades and experiences in the context of COVID-19. Comput. Educ.203:104870. 10.1016/j.compedu.2023.104870
4
AsherM. W.HarackiewiczJ. M.BeymerP. N.HechtC. A.LamontL. B.Else-QuestN. M.et al. (2023). Utility-value intervention promotes persistence and diversity in STEM. Proc. Nat. Acad. Sci.120:e2300463120. 10.1073/pnas.2300463120
5
BettingerE. P.FoxL.LoebS.TaylorE. S. (2017). Virtual classrooms: how online college courses affect student success. Am. Econ. Rev.107, 2855–2875. 10.1257/aer.20151193
6
BoydR. L.AshokkumarA.SerajS.PennebakerJ. W. (2022). The Development and Psychometric Properties of LIWC-22. Austin, TX: University of Texas at Austin, 10.
7
BroadbentJ.PoonW. L. (2015). Self-regulated learning strategies and academic achievement in online higher education learning environments: a systematic review. Internet High. Educ.27, 1–13. 10.1016/j.iheduc.2015.04.007
8
BrodaM.YunJ.SchneiderB.YeagerD. S.WaltonG. M.DiemerM.et al. (2018). Reducing inequality in academic success for incoming college students: a randomized trial of growth mindset and belonging interventions. J. Res. Educ. Eff.11, 317–338. 10.1080/19345747.2018.1429037
9
BrownE. R.SmithJ. L.ThomanD. B.AllenJ. J.MuragishiG. (2015). From bench to bedside: a communal utility value intervention to enhance students' biomedical science motivation. J. Educ. Psychol.107, 1116–113510.1037/edu0000033
10
CanningE. A.HarackiewiczJ. M. (2015). Teach it, don't preach it: the differential effects of directly-communicated and self-generated utility-value information. Motiv. Sci.1, 47–71. 10.1037/mot0000015
11
CanningE. A.HarackiewiczJ. M.PriniskiS. J.HechtC. A.TibbettsY.HydeJ. S.et al. (2018). Improving performance and retention in introductory biology with a utility-value intervention. J. Educ. Psychol.110, 834–849. 10.1037/edu0000244
12
ChanC. K. Y. (2023). A review of the changes in higher education assessment and grading policy during COVID-19. Assess. Eval. High. Educ.48, 874–887. 10.1080/02602938.2022.2140780
13
ChoM.HeronM. L. (2015). Self-regulated learning: the role of motivation, emotion and use of learning strategies in students' learning experiences in a self-paced online mathematics course. Dist. Educ.36, 80–99. 10.1080/01587919.2015.1019963
14
DoleJ. A.SinatraG. (1998). Reconceptualizing change in the cognitive construction of knowledge. Educ. Psychol.33, 109–128. 10.1080/00461520.1998.9653294
15
EcclesJ. S.WigfieldA. (2020). From expectancy-value theory to situated expectancy-value theory: a developmental, social cognitive, and sociocultural perspective on motivation. Contemp. Educ. Psychol.61:101859. 10.1016/j.cedpsych.2020.101859
16
FreemanS.EddyS. L.McDonoughM.SmithM. K.OkoroaforN.JordtH.et al. (2014). Active learning increases student performance in science, engineering, and mathematics. Proc. Natl. Acad. Sci.111, 8410–8415. 10.1073/pnas.1319030111
17
GaspardH.DickeA. L.FlungerB.BrissonB. M.HafnerI.NagengastB.et al. (2015). Fostering adolescents' value beliefs for mathematics with a relevancy intervention in the classroom. Dev. Psychol.51, 1226–1240. 10.1037/dev0000028
18
GregoireM. (2003). Is it a challenge or a threat? A dual-process model of teachers' cognition and appraisal processes during conceptual change. Educ. Psychol. Rev.15, 147–179. 10.1023/A:1023477131081
19
HarackiewiczJ. M.CanningE. A.TIbbbettsY.PriniskiS. J.HydeJ. S. (2016). Closing achievement gaps with a utility-value intervention: disentangling race and social class. J. Pers. Soc. Psychol.111, 745–765. 10.1037/pspp0000075
20
HarackiewiczJ. M.CanningE. A. TIbbbetts, Y.GiffenC. J.BlairS. S.RouseD. I.et al. (2014). Closing the social class achievement gap for first-generation students in undergraduate biology. J. Educ. Psychol.106, 375–389. 10.1037/a0034679
21
HarackiewiczJ. M.HechtC. A.AsherM. W.BeymerP. N.LamontL. B.WheelerN. S.et al. (2023). A prosocial value intervention in gateway STEM courses. J. Pers. Soc. Psychol.125, 1265–1307. 10.1037/pspa0000356
22
HarackiewiczJ. M.PriniskiS. J. (2018). Improving student outcomes in higher education: the science of targeted intervention. Annu. Rev. Psychol.69, 409–435. 10.1146/annurev-psych-122216-011725
23
HechtC. A.HarackiewiczJ. M.PriniskiS. J.CanningE. A.TibbettsY.HydeJ. S.et al. (2019). Promoting persistence in the biological and medical sciences: an expectancy-value approach to intervention. J. Educ. Psychol.111:1462. 10.1037/edu0000356
24
HullemanC. S.GodesO.HendricksB. L.HarackiewiczJ. M. (2010). Enhancing interest and performance with a utility value intervention. J. Educ. Psychol.102, 880–895. 10.1037/a0019506
25
HullemanC. S.HarackiewiczJ. M. (2009). Promoting interest and performance in high school science classes. Science326, 1410–1412. 10.1126/science.1177067
26
HullemanC. S.KosovichJ. J.BarronK. E.DanielD. B. (2017). Making connections: replicating and extending the utility value intervention in the classroom. J. Educ. Psychol.109, 387–404. 10.1037/edu0000146
27
KosovichJ. J.HullemanC. S.PhelpsJ.LeeM. (2019). Improving algebra success with a utility-value intervention. J. Dev. Educ.42, 2–10. Available online at: https://www.jstor.org/stable/45221757
28
LazoswkiR. A.HullemanC. S. (2016). Motivation interventions in education: a meta-analytic review. Rev. Educ. Res.86, 602–640. 10.3102/0034654315617832
29
MarchandG. C.GutierrezA. P. (2012). The role of emotion in the learning process: comparisons between online and face-to-face settings. Internet High. Educ.15, 150–160. 10.1016/j.iheduc.2011.10.001
30
NadelsonL. J.HeddyB. C.JonesS.TaasoobshiraziG.JohnsonM. (2018). Conceptual change in science teaching and learning: introducing the dynamic model of conceptual change. Int. J. Educ. Psychol.7, 151–195. 10.17583/ijep.2018.3349
31
National Center for Education Statistics (2023). Undergraduate Enrollment. Condition of Education. U.S. Department of Education, Institute of Education Sciences. Available online at: https://nces.ed.gov/programs/coe/indicator/cha (Accessed July 20, 2023).
32
PosnerG.StrikeK.HewsonP.GertzogW. (1982). Accommodation of a scientific conception: toward a theory of conceptual change. Sci. Educ.66, 211–227. 10.1002/sce.3730660207
33
PriniskiS. J.RosenzweigE. Q.CanningE. A.HechtC. A.TibbettsY.HydeJ. S.et al. (2019). The benefits of combining value for the self and others in utility-value interventions. J. Educ. Psychol.111, 1478–1497. 10.1037/edu0000343
34
RosenzweigE. Q.HarackiewiczJ. M.PriniskiS. J.HechtC. A.CanningE. A.TibbettsY.et al. (2018). Choose your own intervention: using choice to enhance the value of a utility-value intervention. Motiv. Sci.5, 269–276. 10.1037/mot0000113
35
RosenzweigE. Q.WigfieldA.HullemanC. S. (2020). More useful or not so bad? Examining the effects of utility value and cost reduction interventions in college physics. J. Educ. Psychol.112, 166–182. 10.1037/edu0000370
36
SimardY.GauthierC.RichardM. (2019). L'efficacité de la formation à distance au niveau postsecondaire: une méga-analyse. Can. J. High. Educ.49, 17–36. 10.47678/cjhe.v49i2.188292
37
SimmonsA. B.HecklerA. F. (2020). Grades, grade component weighting, and demographic disparities in introductory physics. Phys. Rev. Phys. Educ. Res.16:020125. 10.1103/PhysRevPhysEducRes.16.020125
38
SinatraG. M. (2005). The “warming trend” in conceptual change research: the legacy of Paul R. Pintrich. Educ. Psychol.40, 107–115. 10.1207/s15326985ep4002_5
39
StarkE. (2019). Examining the role of motivation and learning strategies in student success in online versus face-to-face courses. Online Learn. J.23, 234–251. 10.24059/olj.v23i3.1556
40
TausczikY. R.PennebakerJ. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. J. Lang. Soc. Psychol.29, 24–54. 10.1177/0261927X09351676
41
TheobaldE. J.HillM. J.TranE.AgrawalS.ArroyoE. N.BehlingS.et al. (2020). Active learning narrows achievement gaps for underrepresented students in undergraduate science, technology, engineering, and math. Proc. Natl. Acad. Sci.10.1073/pnas.1916903117
42
TibbettsY.HarackiewiczJ. M.PriniskiS. J.CanningE. A. (2016). Broadening participation in the life sciences with social-psychological interventions. CBE-Life Sci. Educ. 15:es4. 10.1187/cbe.16-01-0001
43
TotonchiD. A.FrancisM. K.TibbettsY.HuelskoetterE.DavisJ.SmithA.et al. (2025). Improving community college students' success in math: findings from two utility-value studies. J. Exp. Educ.93, 1−24. 10.1080/00220973.2023.2243610
44
UC InfoCenter (2025). Undergraduate Students. Available online at: https://diversity.ucsd.edu/accountability/undergrad.html (Accessed July 3, 2025).
45
VanslambrouckS.ZhuC.LombaertsK.PhilipsenB.TondeurJ. (2018). Students' motivation and subjective task value of participating in online and blended learning environments. Internet High. Educ.36, 33–40. 10.1016/j.iheduc.2017.09.002
46
WaltonG. M.MurphyM. C.LogelC.YeagerD. S.GoyerJ. P.BradyS. T.et al. (2023). Where and with whom does a brief social-belonging intervention promote progress in college?Science380, 499–505. 10.1126/science.ade4420
47
WaltonG. M.WilsonT. D. (2018). Wise interventions: psychological remedies for social and personal problems. Psychol. Rev.125, 617–655. 10.1037/rev0000115
48
WangY.LewisS. E. (2022). Efficacy and insights gained from a utility value intervention with inorganic chemistry students. J. Chem. Educ.99, 2798–2807. 10.1021/acs.jchemed.2c00045
49
WhitcombK. M.SinghC. (2021). Underrepresented minority students receive lower grades and have higher rates of attrition across STEM disciplines: a sign of inequity?Int. J. Sci. Educ.43, 1054–1089. 10.1080/09500693.2021.1900623
50
WigfieldA.EcclesJ. S. (2000). Expectancy-value theory of achievement motivation. Contemp. Educ. Psychol.25, 68–81. 10.1006/ceps.1999.1015
51
XuD.JaggarsS. S. (2013). The impact of online learning on students' course outcomes: evidence from a large community and technical college system. Econ. Educ. Rev.37, 46–57. 10.1016/j.econedurev.2013.08.001
52
YeagerD. S.WaltonG. M.BradyS. T.AkcinarE. N.PauneskuD.KeaneL.et al. (2016). Teaching a lay theory before college narrows achievement gaps at scale. Proc. Natl. Acad. Sci.113, E3341–E3348. 10.1073/pnas.1524360113
Appendix
Table A1
| Course | Quizzes | Midterm + final exams | Homework | Writing assignments (including 2 UVI or control) |
|---|---|---|---|---|
| Genetics | 8 | 1 + 1 | 4 | 5 |
| Genomics | 7 | 1 | 9 | 2 |
| Evolution | 1 | 18 | 6 | |
| Human Reproduction | 9 | 8 | 2 |
Summary table of course assessments and grading schemes.
Summary
Keywords
university, underrepresented minority students, online courses, COVID-19, utility value
Citation
Shillington KJ, Lo SM and Hobbie L (2025) A utility value intervention to support student engagement in online upper-level undergraduate courses. Front. Educ. 10:1636446. doi: 10.3389/feduc.2025.1636446
Received
27 May 2025
Accepted
11 August 2025
Published
04 September 2025
Volume
10 - 2025
Edited by
Melissa C. Srougi, North Carolina State University, United States
Reviewed by
Ying Wang, University of Virginia, United States
Stacy Priniski, Temple University, United States
Updates
Copyright
© 2025 Shillington, Lo and Hobbie.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Katie J. Shillington kshillington@wlu.edu
†ORCID: Katie J. Shillington orcid.org/0000-0003-1363-8778
Stanley M. Lo orcid.org/0000-0003-3574-2197
Lawrence Hobbie orcid.org/0000-0002-8953-1435
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.