Salient Experiences in Student Development: Impact of an Undergraduate STEM Teacher Preparation Program

Introduction

There is an acute need for content discipline-trained science teachers in the United States (Vest, 2010; García & Weiss, 2019; See and Gorard, 2019). The lack of fully trained science educators is, in part, due to low undergraduate retention across science, technology, engineering, and mathematics (STEM) disciplines (Graham et al., 2013; NSTC, 2018). Attrition from STEM career pipelines is particularly pronounced in PEERs (persons excluded due to ethnicity or race) and other traditionally underrepresented groups in STEM fields, including STEM teachers (Metcalf, 2016; National Academies of Sciences, 2016; Mourad et al., 2018; NSTC, 2018; National Student Clearinghouse Research Center, 2019; Theobald et al., 2020). STEM-based learning communities can enhance academic success, increase persistence, and retain students for graduation at higher rates (Carrino and Gerace, 2016; Dagley et al., 2016; Fulton and Britton, 2011; Kuh, 2008:; Pasque and Murphy, 2005; Heaney and Fisher, 2011; Soldner et al., 2012). STEM-based learning communities rooted in best-practices in undergraduate STEM education are shown to improve learning outcomes and retention in PEERs (Theobald et al., 2020). STEM-based learning communities also provide informal learning ecosystems for students that include the ability to interact and collaborate with one another outside of the pragmatic, traditional classroom experiences (Kotys-Schartz et al., 2011). Furthermore, providing students with exposure to those with divergent identities from their own can impact their identity development during college because these interactions “…developmentally challenge students’ conceptions of themselves and that requires adaption and commitment to certain attitudes, values, beliefs, and actions” (Pascarella and Terenzini, 1991, p. 190). Consequently, STEM-based learning communities can lead to improved academic success as measured via degree completion (Baker and Pomerantz, 2001; Hill and Woodward, 2013; Dagley et al.2016; Russell, 2017), as well as provide positive influences on self-confidence (MacPhee et al., 2013). However, the replicability of programmatic descriptions is often limited, due to variations in scope, application, and availability of information regarding programmatic experiences (Jacobi, 1991; Crisp and Cruz, 2009).

Thus, we investigated the impact of specific, replicable experiences in a pre-service teacher STEM program. This teacher preparation program is called NoyceSCIENCE and is offered at the University of Nebraska at Omaha (UNO). The NoyceSCIENCE program is designed to recruit science majors to become high school science teachers in high-needs schools, and differs from more traditional teacher preparation programs because the undergraduates must either major within a science discipline while also obtaining certification (rather than majoring in teacher education) or be a dual-major (i.e., be both a science major and a teacher education major, fulfilling both academic obligations). It is this emphasis on content first, pedagogy second that makes this preparation unique. Specifically, through the NoyceSCIENCE program, students participate in structured 1) training in culturally responsive pedagogy, 2) STEM research experiences with apprenticeship style mentorship, and a variety of 3) professional development opportunities, such as serving as a learning assistant in course-based undergraduate research experiences. These interventions are in addition to the skills, knowledge, and networks obtained via their degree programs.

Herein, we investigate the individual and collective impact of these experiences on NoyceSCIENCE participants within the theoretical framework of Chickering and Reisser (Chickering, 1969; Chickering and Reisser, 1993). Of note, this framework is well-aligned to this project because it bridges cognitive development and psychosocial theory (i.e., student development theory). Subsequent iterations of Chickering and Reisser, (1993) model of vectors illustrates the fluid nature of students’ psychosocial development along seven core areas; specifically 1) developing competence, 2) managing emotions, 3) moving through autonomy toward interdependence, 4) developing mature interpersonal relationships, 5) establishing identity, 6) developing purpose, and 7) developing integrity (Evans et al., 2009). This framework posits that the learning experience of college students do not solely involve learning that occurs in classrooms; rather, it is a holistic process of educating the whole self. Further, this process does not happen linearly over time, but typically in developmental stages occurring in vectors, which Leggette and Jarvis (2015) liken to spokes on a wagon wheel to illustrate the interconnected nature of each vector. We recognize there are other more recent iterations of Chickering’s Student Development Theory, including Rahm’s amended vision to the Chickering (1969) framework, calling attention to relational learning, student identity, and affective factors (Rahm, 2014). However, the Chickering (1969) framework is most well-aligned with our research focus, thus we have chosen to focus our research through this lens of student development.

In this study, we sought to assess how the experiences provided to participants in this pre-service teacher preparation program (NoyceSCIENCE), which is integrated with best practices in discipline-based education research, contributed to undergraduate student development (NRC, 2012). We focused our research efforts towards understanding the impact of the experiences within this program, particularly experiences more unique to this program (i.e., Scholars serving as Learning Assistants in Course-based undergraduate research experiences and as mentors in the NE STEM 4U program). Further, these experiences are more readily integrated in other programmatic settings and could provide insight into particularly meaningful components of undergraduate student learning communities. We ultimately aim to understand the following questions:

1) Which integrated experiences from NoyceSCIENCE are the most salient to student development as described from the perspective of the student?

2) Is there an intersection for student development for different levels of NoyceSCIENCE engagement (i.e., Interns vs Scholars), or is one at a greater advantage? and

3) Does the development of specific Vectors change throughout participation in the NoyceSCIENCE program?

We hypothesized that the interventions that are unique to our program (i.e., LA in CUREs and NE STEM 4U) would stand out as being most impactful. To answer these questions, we collected qualitative data via journal entries of participants across 3 years of the program.

Methods

Characteristics of Participants

Any student at UNO that is interested in joining the NoyceSCIENCE program may apply. Students must formally apply and include a personal statement in their application. There are no formal GPA requirements, and students at all levels (freshman through seniors) can, and have, participated. The only requirement is that participants are pursuing a science disciplinary major (e.g., biology, chemistry, geology, physics, etc.). The pool of student applicants has consistently been above the financial number of awards available to provide.

The program supports students at two levels: “Noyce Interns” and “Noyce Scholars”. At the “Noyce Intern” level, there are no restrictions for participation. The “Noyce Intern” is designed for undecided science majors who were previously uninterested in a teaching profession, allowing them to experiment within the science teaching career field to learn more about STEM education and to reflect upon and to consider their interest level in teaching. Noyce Interns work for a set period of time (i.e., one summer or one semester) and are directly mentored by a faculty member in their area of study. Ideally, Noyce Interns will ultimately decide to pursue teaching as a profession and be supported with a greater ecosystem of resources and fellowship at the “Noyce Scholar” level. However, if Interns decide that teaching is not in their career plan, they are able to separate from the program without any repercussions. The “Noyce Scholar” is designed for science majors who intend to become secondary teachers, and they receive a 2-years scholarship and internship from the NoyceSCIENCE program. Noyce Scholar restrictions include having either 1) a disciplinary major, or a dual-major (i.e., majoring in a science discipline as well as teacher education), and 2) the intent to teach in a high-needs school after graduation (AAAS, 2020). Noyce Scholars have GPA requirements, not from the Noyce program, but from the teacher preparation program at UNO, that they must maintain to reach graduation, and ultimately certification. As Scholars receive financial support and an internship from the program, they participate in more NoyceSCIENCE experiences than Interns over a longer period. The characteristics of our Noyce Interns and Noyce Scholars have been highly variable. Our participants have had a range of GPAs (1.5—4.0), and our students are from diverse backgrounds, including first-generation college students and non-traditional students. It is important to emphasize that we are not solely selecting students who have strong academic backgrounds.

NoyceSCIENCE Experiences

While other pre-service teacher preparation programs have reported on the support pathways offered, we will focus more specifically on two interventions that are embedded in our program that make them most unique: 1) NoyceSCIENCE students (both Scholars and Interns) serving as learning assistants in Course-based undergraduate research experiences, and 2) Scholars and Interns serving as undergraduate mentors in our NE STEM 4U program. Consequently, our focus on modeling effective pedagogy with our pre-service teacher participants is done so in two ways: Active learning via the CUREs (with undergraduates as the learner community) and NE STEM 4U with youth (i.e., students closer to the age target for our pre-service teachers).

The undergraduate learning assistant (LAs) model is well-researched in STEM education, particularly that of the positive impact they can have on student learning outcomes in STEM courses (Talbot et al., 2015; Alzen et al., 2018). Here, we have implemented an undergraduate learning assistant model in course-based undergraduate research experiences (CUREs), which are effective and inclusive active-learning focused alternatives to traditional laboratory courses (Bangera and Brownell, 2014; Corwin et al., 2015). Scholars and Interns are paired with an instructor teaching a semester-long CURE, where they serve in a learning assistant teaching model for the duration of the course (Talbot et al., 2015; Alzen et al., 2018). The CUREs that Scholars and Interns have served as an LA in include general chemistry courses, general physics courses, and a virology course.

For the undergraduate mentoring component (the NE STEM 4U program), participants work with K-8 youth in socioeconomically disadvantaged areas to provide engaging, hands-on, minds-on STEM activities. Our NE STEM 4U program builds a collective, long-term ecosystem among Noyce participants within their home institution, and with those from around the United States. For a full description of the NE STEM 4U program, please see Cutucache et al. (2016); Cutucache et al. (2018); Leas et al. (2017); Nelson et al. (2018); and Stevenson et al. (2021).

Finally, even though participation in the NE STEM 4U program and serving as a CURE LA is a requirement of all participants in the NoyceSCIENCE program, all NoyceSCIENCE participants have engaged in activities above and beyond this single method of professional development, including undergraduate research experiences, faculty mentoring, workshops led by experts, national conferences, peer group meetings, and volunteer experiences, among others (Table 1). Scholar or Intern participation in these experiences (i.e., those outside of the compulsory CURE LA and NE STEM 4U mentor) differed based on individual interest and motivation to be involved. Activities like volunteer experiences and national conferences were frequently chosen by students based on the recommendation of faculty mentors or peers, and based on personal values and experiences offered in the local community. Students logged their thoughts on participating in these experiences (among other degree-related and personal thoughts) in NoyceSCIENCE journals. As we intended to collect these journals for qualitative investigation, we did not prompt students regarding the content that should be present in the journals, except that they should log their experiences, thoughts, feelings, and activities regarding all NoyceSCIENCE related participation at least once weekly for each semester they are in the program. All methods described here are approved under IRB #015-17-EX.

TABLE 1

TABLE 1. In their journals, undergraduate pre-service STEM teachers describe the experiences they participate in during their time in the Noyce NebraskaSCIENCE program. Here, we identify these experiences and provide a brief explanation based on students’ journal entries.

Data Collection

To assess the impact of the experiences within this program on students, we collected journals (n = 29 total journals, n = 14 intern journals, n = 15 scholar journals) from Interns (n = 7 students) and Scholars (n = 9 students) for each semester (Fall, Spring, and Summer) of participation in NoyceSCIENCE from Fall 2017-Fall 2019. Each student logged multiple journals, depending on the number of semesters they were involved in the program. Students were given limited instruction for journal entries; only that they should categorize journals by experiences and date. Students were not directed to limit their entries to NoyceSCIENCE related experiences, nor were they given page limits, thus there is great variation in the amount and content of each journal. For this study, all identifying information was removed from journal entries prior to importing into NVivo 12 for Mac (©QSR International, Victoria, Australia) for qualitative analysis.

Data Analysis

We used descriptive coding (Saldaña, 2016) for each journal entry to develop the sub-codes for each of these parent codes, as our sub-codes focus on the “topic” not the “content” of the passages (Miles et al., 2014). We utilized the Chickering’s Vectors framework, with a codebook, per best practices of qualitative data analysis (Chickering, 1969; Stake, 2005; Miles et al., 2014; Creswell and Poth, 2017; Yin 2018).

To understand how participants communicate their stage of development through the intervention experiences requires collecting rich, descriptive data (Yin, 2018) and deciphering unanticipated concepts of development (Charmaz, 2008). Applied research environments, such as the project described herein, are ideal for qualitative research where site-specific conditions are used to inform (and, potentially, to make recommendations on) application to the field (Merriam & Tisdell, 2015). As we coded, we cataloged emerging patterns in the data to help to “interpret how the individual components of the study weave together” (Saldaña, 2016, p. 48). Utilizing these best practices, our step-by-step process is outlined below.

First, we quickly read through each journal entry to familiarize with the participant’s experiences. Following, we read each journal entry a second time, thoroughly categorizing each experience, feeling, discussion point, etc., as one of Chickering’s 7 vectors (Developing Competence, Developing Integrity, Developing Mature Interpersonal Relationships, Developing Purpose, Establishing Identity, Managing Emotions, and Moving Through Autonomy Towards Independence). If a sentence did not fit a vector, the entry was not coded, thus not every journal is 100% covered. The third review of each journal was to code for the 11 experiences (Table 1) that students participate in during the NoyceSCIENCE program. Each NoyceSCIENCE participant partook in multiple experiences. Finally, each journal entry was categorized by level of participation (i.e., Intern vs Scholar), and the number of years of participation. These took place as follows: Coding Cycle 1 (Chickering’s Vectors, 7 parent codes)→Coding Cycle 2 (Professional Development Experiences; 11 parent codes)→ Coding Cycle 3 (Demographics; Percent coverage and count data).

Our data are presented in two forms: 1) percent coverage, and 2) count data (NVivo™ 12 for Mac). Percent coverage represents the percentage that a ‘code’ (either one of Chickering’s 7 Vectors or one of the 11 NoyceSCIENCE experiences) covers the journals by character count. Count data represents the number of occurrences of a code in the journals and is useful for conducting quantitative analyses on journal entries, as we have included here. Importantly, raw count data do not account for individual differences in each participant’s writing (e.g., some students are more verbose than others). To account for this, we use a ratio of the count data for analysis (per individual student) that represents the total word length of a participant’s journal entry, divided by the number of semesters in a program, as compared to the overall average word count of all participants per journal entry.

All statistical analyses were conducted in R version 3.5.2 (© The R Foundation for Statistical Computing). We tested for the effects of participation in an experience, participation level, and duration within the program (Intern, New Scholar <1 year, Veteran Scholar >1 year) on presence of each of Chickering’s 7 Vectors, to measure the impact of these experiences on student development within NoyceSCIENCE (Table 1). To analyze the ratio count data, we first removed all occurrences of 0 counts where students did not participate in an experience. Next, we conducted an outlier analysis and removed outliers <30. Still, preliminary visual data exploration indicated the high occurrence of 0 counts within the data, so we first conducted a Vuong test before each analysis to estimate whether a standard Poisson Regression model or a zero-inflated Poisson Regression model was more appropriate (Long, 1997; Desmarais and Harden, 2013). For multivariate logistic regression, the z-value is the regression coefficient divided by the standard error to yield a regression coefficient. If initial models were significant, we then conducted an interaction post-hoc analysis (Long, 1997; Desmarais and Harden, 2013).

Results

Our aim was to provide an investigation into the pathway of student development through the meshwork of Chickering’s Vectors. The compilations of these data provide an overarching view of student development within the NoyceSCIENCE program, as well as the specific impact of experiences toward Vectors. The data described are based on self-report data (journal entries) and confirmed with experiential evidence and include descriptive data in the form of percent coverage of journal entries, and count (using a ratio to account for journal length and duration in the program) data of a Vector or experience within a journal entry. Supporting quotes as examples for each Vector are located in Table 2. First, we describe the top four Vectors present in the program across 3 years, including all participants (n = 16 students, 29 journal entries) by percent coverage: 1) Developing Competence (55.65%), 2) Moving Through Autonomy Towards Independence (39.52%), 3) Establishing Identity (18.60%), and 4) Managing Emotions (10.49%).

TABLE 2

TABLE 2. Quotes from students that represent each of Chickering’s Vectors.

Which integrated Experiences From NoyceSCIENCE are the Most Salient to Student Development?

Here, we identified which experience reports the highest percent coverage of each of Chickering’s 7 Vectors, indicating which activity/intervention most contributes to student development of that particular Vector. Undergraduate Mentoring was the highest contributing experience to Developing Integrity (14.61%), Developing Mature Interpersonal Relationships (11.44%), Developing Purpose (18.48%), and Managing Emotions (33.92%). Learning from Experts was the experience that contributed the highest to Developing Competence (88.08%), Volunteer Experience contributed the highest toward Establishing Identity (62.25%), and Undergraduate Research Experience was the highest contributing experience to Moving Through Autonomy Toward Independence (69.53%; Figure 1; Table 1, 2). Finally, we identified the top four experiences that contributed to student development in the NoyceSCIENCE program by percent coverage (the sum of each Vector’s percent coverage within each experience divided by the number of Vectors). The top contributing experience was Undergraduate Mentoring (32.42%), followed by Volunteer Experiences (30.76%), Faculty Mentorship (26.14%), and Learning from Experts (24.74%; Figure 1; Table 1, 2).

FIGURE 1

FIGURE 1. Here we present the activities and experiences integrated in Noyce NebraskaSCIENCE that contribute to student development via Chickering’s Vectors (outer circle), including: The top four most contributing activities to student development (inner circle), and other contributing activities (middle circle). The activity or experience that most contributes to each Vector is indicated via color-coding and the associated icon.

Next, we measured the effect of experience participation on student development via Chickering’s Vectors through the ratio count data. The experiences Undergraduate Mentor, Undergraduate Research Experiences, and Traditional Lab Learning Assistant represented the greatest contribution to Developing Competence (Standard Poisson Regression—Undergraduate Mentor: Z = 3.87, Pr>|z| < 0.001; Undergraduate Research Experiences: Z = 3.81, Pr>|z| < 0.001; Traditional Lab Learning Assistant: Z = 3.31, Pr>|z| < 0.001), Managing Emotions (Standard Poisson Regression—Undergraduate Mentor: Z = 3.71, Pr>|z| < 0.001; Undergraduate Research Experiences: Z = 2.56, Pr>|z| = 0.01; Traditional Lab Learning Assistant: Z = 3.31, Pr>|z| = 0.003), and Moving Through Autonomy Towards Independence (Standard Poisson Regression—Undergraduate Mentor: Z = 2.79, Pr>|z| = 0.005; Undergraduate Research Experiences: Z = 4.32, Pr>|z| < 0.001; Traditional Lab Learning Assistant: Z = 3.68, Pr>|z| < 0.001; Developing Competence: all post-hoc p < 0.03; Managing Emotions: all post-hoc p < 0.03; Moving Through Autonomy Towards Independence: All post-hoc p < 0.01; Figure 2; Supplementary Table S1). The experience Undergraduate Mentor contributed more than Faculty Mentorship to Establishing Identity, but otherwise all experiences contributed similarly (Standard Poisson Regression— Undergraduate Research Experiences: Z = 2.56, Pr>|z| = 0.01; post hoc p < 0.001; Figure 2; Supplementary Table S1). For the remaining Vectors: Developing Integrity (Standard Poisson Regression—all Pr>|z| > 0.98), Developing Mature Interpersonal Relationships (Standard Poisson Regression—all Pr>|z| > 0.98), and Developing Purpose (Standard Poisson Regression—all Pr>|z| > 0.98), all experiences contributed similarly to student development.

FIGURE 2

FIGURE 2. Traditional Lab Learning Assistant, Undergraduate Mentor, and Undergraduate Research Experiences contributed more to Developing Competence (A), Managing Emotions (B), and Moving Through Autonomy Towards Independence (C) than other experiences, while Undergraduate Mentor contributed more than Faculty Mentorship to Establishing Identity (D). Experiences that share a letter are not significantly different. N values (count ratio of a Vector per Experience) are indicated below significant groupings.

Is there an Intersection for Student Development for Both Interns and Scholars, or Is one at a Greater Advantage?

The top four Vectors based on percent coverage of journal entries were consistent for students who participated as a Scholar and as an Intern: 1) Developing Competence (Scholars: 63.69%; Interns: 57.75%), 2) Moving Through Autonomy Towards Independence (Scholars: 43.17%; Interns: 46.32%), 3) Establishing Identity (Scholars: 29.14%; Interns: 18.02%), and 4) Managing Emotions (Scholars: 14.29%; Interns: 17.13%). However, when considering the count ratio of all 7 Vectors combined, Interns (1.33 ± 0.38) report more occurrences of student development than Scholars (0.98 ± 0.14; Zero-inflated Poisson Regression—Estimate: 0.40, Std. Error: 0.1, z-value: 4.08, Pr>|z| = 4.43^e−05). Interestingly, the ratio counts of Interns (4.33 ± 1.74) and New Scholars (5.31 ± 2.15) were greater than Veteran Scholars (2.53 ± 0.56) for Developing Competence (Standard Poisson Regression—Veteran Scholars: Z = −2.75, Pr>|z| = 0.006), indicating that a majority of the Vectors reported for Interns come from Developing Competence, and not the other six Vectors (all post-hoc p < 0.01; Figure 3; Supplementary Table S2).

FIGURE 3

FIGURE 3. Intern and New Scholar (Scholars who have participated in NoyceSCIENCE for <1 year) had greater ratio counts of Developing Competence than Veteran Scholars (Scholars who have participated in NoyceSCIENCE for >1 year). Experiences that share a letter are not significantly different. N values (count ratio of a Vector per participation level) are indicated below significant groupings and mean ± standard error is indicated above significant groupings.

Does the Development of Specific Vectors Change Throughout Participation in the NoyceSCIENCE Program?

Finally, we have compared the ratio count of Chickering’s Vectors present in Interns, New Scholars those who have participated in NoyceSCIENCE <1 year, and Veteran Scholars (those who have participated >1 year), and found that there are more occurrences of Vectors within Interns (1.33 ± 0.38) and New Scholars (1.33 ± 0.44) than Veteran Scholars (0.85 ± 0.11; Zero-inflated Poisson Regression—Estimate: 0.38, Std. Error: 0.1, z-value: 3.74, Pr>|z| = 0.002; all post-hoc p < 0.002; Figure 3). However, percent coverage results provide more context to these results and highlight that Intern and New Scholar participation is heavily dominated by the Vectors Developing Competence (88.04%) and Moving Through Autonomy Towards Independence (81.85%), where Vectors such as Establishing Identity (36.98%), Developing Purpose (12.12%), Developing Integrity (2.06%), and Developing Mature Interpersonal Relationships (0.69%) begin to emerge in later years of participation (Veteran Scholars), which we have illustrated further in Figure 4.

FIGURE 4

FIGURE 4. Developing Competence and Moving Through Autonomy Towards Independence are the main Vectors present in Interns (B) and New Scholars [Year 1 of (A)], represented here by percent coverage (those who have participated in NoyceSCIENCE for <1 year). However, Vectors like Establishing Identity, Developing Purpose, Managing Emotions, Developing Integrity, and Developing Mature interpersonal Relationships emerge in Veteran Scholars [Year 2 of (A); those who have participated in NoyceSCIENCE for >1 year].

Discussion

In this manuscript, we describe the progression of NoyceSCIENCE participants (i.e., both Interns and Scholars) through the lens of Chickering’s Vector framework (Chickering, 1969; Chickering and Reisser, 1993). We aim to demonstrate the impact of the NoyceSCIENCE program on student development as a result of participation in various activities and experiences (specifically, the contributions of the CURE-LA and NE STEM 4U interventions). Importantly, these data are the result of multiple years of implementation of the program, as well as multiple levels of undergraduates (from freshmen through seniors). Finally, a major strength of this work is that any student can participate, regardless of grade level or GPA. When institutions of Higher Education are often looking for ways to bolster ties with the community, expand the undergraduate student pool, and retain students to graduation, we wished to determine if there are particular experiences within our program that are the most salient to student development, as described from the perspective of the student.

Which Integrated Experiences from NoyceSCIENCE are the Most Salient to Student Development as Described from the Perspective of the Student?

In this study we observed that the singular top-contributing experience for undergraduates was that of Undergraduate Mentoring (32.42%) within the context of our NE STEM 4U program. This finding alone was unsurprising, given our previous work studying the impact of this program both via self-report, and via mixed-methods work that included observational and quantitative data (Cutucache et al., 2016; Nelson and Cutucache, 2017; Nelson et al., 2018). However, what was surprising was how influential this single intervention was across nearly every Vector. Thus, serving as an undergraduate mentor through the NE STEM 4U program provided the greatest boon for student progression through their developmental journey along Vectors. Secondly, volunteer experiences (30.76%) showed substantial influence for students. Together, these findings suggest that a strong tie to the community via outreach and volunteerism are core ways for students to develop during their college experience. Whether this is a direct result of near-peer learning, a sense of reflection and personal satisfaction upon completion of community-based work, broadening their conceptual and interpersonal communication skills in STEM, or greater empathy developed from working with a variety of stakeholders remains to be determined. Faculty mentorship was identified as the third greatest influencer of student development, followed by Learning from Experts (of which some of the facilitators of the workshops covered under ‘learning from experts’ included faculty). These data demonstrate the profound impact that individual faculty can have on students during their college experience, and support the growing body of literature indicating that the traditional model of a single faculty member with a large classroom resulting in a less-personalized experience should be re-examined (Freeman et al., 2014; Theobald et al., 2020). The overall impact of participation in the NoyceSCIENCE interventions indicates that the program is facilitating opportunity, involvement, and belonging broadly within the STEM community. Notably, we hypothesized that undergraduate mentoring (through the NE STEM 4U program), and serving as an LA for a CURE would be the most impactful experiences in the NoyceSCIENCE program. However, students did not communicate that serving as an LA for a CURE was more impactful than other experiences in terms of student development. We posit that there may be unique aspects of the NE STEM 4U program that lend to its greater impact on student development from a students’ perspective, providing interesting avenues for future research.

Is There an Intersection for Student Development for Different Levels of NoyceSCIENCE Engagement (i.e., Interns vs Scholars), or is One at a Greater Advantage?

We also sought also to determine if the duration within the program (e.g., less than 1-year, or more than 1-year), or participation level in the program (Scholar vs Intern), influenced a student’s development. And, which experiences are more critical (as self-reported by the student) during different stages of the collegiate experience (i.e., year in school and sheer time in on NoyceSCIENCE experiences). We observed that there were more occurrences of Vectors within Interns and New Scholars as compared with Veteran or post-1-year Scholars, indicating Interns and Scholars new to the program identify more occurrences of student development than those who have spent time in the program.

Does the Development of Specific Vectors Change Throughout Participation in the NoyceSCIENCE Program?

Our findings that Interns (who typically spend only a single semester or summer in the program) and new Scholars (those in the program <1 year) describe more occurrences of student development than veteran Scholars (those in the program >1 year) led us to further investigate whether duration in the program was correlated with student development. Interestingly, the data described above related to occurrence of vectors are confounded by the emphasis on just two Vectors, Developing Competence and Moving Through Autonomy Towards Independence—these occur most frequently in Interns and new Scholars. Other Vectors, including Establishing Identity, Developing Purpose, Developing Integrity, and Developing Mature Interpersonal Relationships begin to emerge in later years of participation, particularly in veteran Scholars (Figure 4). Thus, NoyceSCIENCE students identify student development in Competence and Independence concentrated early in the program, with other less frequently observed aspects of student development emerging as they progress.

Translation of Findings to the Wider Science, Technology, Engineering, and Mathematics Community

These findings are of broad interest and application when we align them with the experience that most contributed to each vector. We see that undergraduate research experiences and learning from experts (via participation in workshops) are the greatest contributors to Developing Competence and Moving Through Autonomy Towards Independence, the Vectors that are most concentrated in the early stages of participation in the program (Figure 4). In the case of Vectors that emerged later in program participation in veteran Scholars (Establishing Identity, Developing Purpose, Developing Integrity, and Developing Mature Interpersonal Relationships), we note that undergraduate mentoring through participation in NE STEM 4U, and volunteer experiences are the top contributors to these Vectors. Thus, we posit that undergraduate science learning communities, particularly those that focus on teacher preparation, would benefit from integrating experiences that mimic that of the mentoring opportunities offered through NE STEM 4U within their program, along with the other salient experiences we observed, including faculty mentorship, learning from experts, and volunteer experiences.

We note that the Vector Establishing Identity occurred at a lower frequency, and was arguably aligned with longer time in the program (as it was mostly concentrated in veteran Scholars). These data are helpful in understanding retention pathways for students more broadly, particularly in the high attrition rate of PEERs in STEM (Metcalf, 2016; National Academies of Sciences, 2016; Mourad et al., 2018; NSTC, 2018; National Student Clearinghouse Research Center, 2019; Theobald et al., 2020). Here we provide data that may show when a student fosters self-identity in a STEM career, and what particular experiences are impactful in this key component of STEM retention. This information can help ensure effective alignment of mentoring students as to what experiences they may most benefit from, targeting critical “pivot points” in undergraduate retention in STEM degrees, hopefully contributing to the great need for STEM teachers in the United States.

Limitations

It is important to note that the journals (qualitative data) from both Interns and Scholars are not limited to academic experiences. Students were provided minimal prompting on what must be included in the journals as a reporting requirement for the program, so they are largely free-form. The journals included personal experiences, life events, and areas of growth outside of their daily academic endeavors. These additions demonstrated meta-cognition, though this is not the focus of this manuscript. Thus, our findings could include influence from broader life experiences, including multiple factors that impact student development, such as growth in meta-cognition, which is consistent with our previous work (Nelson and Cutucache, 2017). Additionally, related to a students’ progression toward independence, one co-variate that we are unable to subtract is the sheer process of writing their journals, which may have also contributed to their progress particularly related to autonomy and establishing identity, as others have described. Students who participate in reflective writing are more engaged and self-directed due to the intrinsic meta-cognitive process of self-assessment (Levett-Jones, 2007; Lengelle et al., 2014).

Another key limitation of the work described herein is the heavily qualitative nature via student self-report data. The limitations of self-report data are well-documented; although we aimed to overcome the reticence for students to write authentic thoughts and viewpoints by offering very little structure as well as data de-identification (Austin et al., 1998; Balakrishnan, 1999; Wilcox, 2005; Fan et al., 2006; Rosenman et al., 2011). Another way that we have worked to minimize the bias within the self-report data is to allow students to use the journal as a free-form method of collecting their thoughts. Therefore, it’s not prescriptive, nor leading in nature. While this allows a more realistic representation of students’ experiences and development during their time in NoyceSCIENCE and helps us to measure the development that we would not otherwise observe, we emphasize that the top contributing experiences within the program work together with other internal and external experiences that aid in student development through the lens of Chickering’s Vectors.

Future Directions

A major goal of these interdisciplinary projects is to ultimately define an “empirical formula” or “systematic method” of sorts for university faculty, leaders, and administrators to best inform ways to recruit, retain, and prepare students for the 21st century workforce. Certainly these data are limited only to that of a single institution, and degree pathways only across biology, chemistry, geology/geography, physics, and teacher education, but the findings identified herein are substantiated by the overarching retention rates within undergraduate students that participate in the NE STEM 4U program (Nelson et al., 2018). In general, Scholars who participated more than a year in the program described different Vectors of student development than other participants, however, these findings warrant further investigation to address this question. Thus, the future expansion of the NoyceSCIENCE intervention will include specific questions regarding dosing—for example, does the amount of time in an experience, frequency of experience, or neither influence a student’s development? This study showed that there was no emphatic difference between length of time in the program (Figure 3), but does that change based on the number or types of experiences? Furthermore, is their development influenced by their cohort of near-peers? For example, is there a tangible benefit to having cohorts of students with similar career aspirations or goals in their learning community? Our future work will seek to address to help to operationalize and to investigate these questions.

Finally, these perspectives on learning communities will aid Institutions of Higher Education in resourcing and supporting core programs that facilitate science teacher preparation. Every program has limited time, access to faculty/staff, or funding for such programs. Therefore, we aim to identify core attributes on which to invest to support an expanding STEM pipeline of pre-service teachers.

Conclusion

Universities continuously seek improved understanding on how to best prepare undergraduates as future lifelong learners. We utilized the NoyceSCIENCE program, and specifically the experiences associated with participation in the program (namely, the CURE-LA and NE STEM 4U salient interventions), as an example model to better understand student development from the learner’s eyes from the beginning through end of program participation. We suggest that these data are applicable to other programs and student learning communities, especially given that our findings indicate where the greatest value and outcome arises from a student’s perspective. We observed four items to be most impactful from the perspective of the student: Learning from experts, volunteer experiences, faculty mentorship, and undergraduate mentorship. Undergraduate mentoring contributed the most to development across every vector, particularly those that emerge in Scholars who have participated in the program for more than a year, thus we denote this experience as critical for student development, as modeled in the NE STEM 4U program.

Ultimately, our goal is to understand how students in STEM disciplines develop in order to best prepare them for their post-college career. Similarly, we aim to identify core experiences (pedagogically) for each student that add to their retention within a STEM discipline, to graduation, and to retain them in a STEM career post-graduation. However, as higher education experiences have received the greatest scrutiny to-date, we also aim to highlight how the marriage of theory (i.e., student development theory, Chickering, 1969) with practice (e.g., experiential learning) that best prepares our students for their futures.

The White House STEM America report (NSTC, 2018) highlights the need to 1) Build strong foundations for STEM literacy, 2) increase diversity, equity, and inclusion in STEM, and 3) to prepare the STEM workforce for the future. This report aims to suggest salient additions for programs to consider, based on student perceptions, to expand future efforts to STEM learning.

Data Availability Statement

The datasets presented in this article are not readily available because Per our IRB guidelines we do not have permission to publicly share our raw, de-identified data. Requests to access the datasets should be directed to ccutucache@unomaha.edu.

Ethics Statement

The studies involving human participants were reviewed and approved by the Institutional Review Board All methods described here are approved under IRB #015-17-EX. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

CC, NG, and WT designed the project and research questions. AS, KJ, and PJ contributed to the methodology and data analysis related to this intervention. AS contributed most substantially to data analysis. All authors contributed to the data interpretation, analysis, and writing of the manuscript, including editing the manuscript.

Funding

This work was supported by the National Science Foundation DUE-1659058. The contributions described herein are solely that of the authors and not the funding agency.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors thank the participants within this study.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2021.575188/full#supplementary-material

References

[AAAS] American Association for the Advancement of Science (2020). The Robert Noyce Teacher Scholarship Program. AAAS Advancing Res. Innovation STEM Edu. Preservice Teach. High-Need Sch. Districts (Arise).https://www.nsfnoyce.org/about/(Accessed April 2, 2020).

Google Scholar

Alzen, JaL., Langdon, L. S., and Otero, V. K. (2018). A Logistic Regression Investigation of the Relationship between the Learning Assistant Model and Failure Rates in Introductory STEM Courses. Int. J. STEM Educ. 5 (1), 56–12. doi:10.1186/s40594-018-0152-1

CrossRef Full Text | Google Scholar

Austin, E. J., Deary, I. J., Gibson, G. J., McGregor, M. J., and Dent, J. B. (1998). Individual Response Spread in Self-Report Scales: Personality Correlations and Consequences. Personal. Individual Differences 24, 421–438. doi:10.1016/s0191-8869(97)00175-x

CrossRef Full Text | Google Scholar

Baker, S., and Pomerantz, N. (2001). Impact of Learning Communities on Retention at a Metropolitan university. Coll. Student Retention Res. Theor. Pract. 2 (2), 115–140.

Google Scholar

Balakrishnan, J. D. (1999). Decision Processes in Discrimination: Fundamental Misrepresentations of Signal Detection Theory. J. Exp. Psychol. Hum. Percept Perform. 25, 1189–1206. doi:10.1037/0096-1523.25.5.1189

CrossRef Full Text | Google Scholar

Bangera, G., and Brownell, S. E. (2014). Course-based Undergraduate Research Experiences Can Make Scientific Research More Inclusive. CBE Life Sci. Educ. 13 (4), 602–606. doi:10.1187/cbe.14-06-0099

PubMed Abstract | CrossRef Full Text | Google Scholar

Carrino, S. S., and Gerace, W. J. (2016). Why STEM Learning Communities Work: The Development of Psychosocial Learning Factors through Social Interaction. Learn. Communities Res. Pract. 4 (1), Article 3.

Google Scholar

Charmaz, K. (2008). “Constructionism and the Grounded Theory,” in Handbook of Constructionist Research. Editors J. A. Holstein, and J. F. Gubrium (New York: The Guilford Press), 397–412.

Google Scholar

Chickering, A. W. (1969). Education and Identity. San Francisco: Jossey-Bass.

Google Scholar

Chickering, A. W., and Reisser, L. (1993). Education and Identity. San Francisco, CA 94104: The Jossey-Bass Higher and Adult Education SeriesJossey-Bass Inc., Publishers, 350 Sansome St.

Google Scholar

Corwin, L. A., Graham, M. J., and Dolan, E. L. (2015). Modeling Course-Based Undergraduate Research Experiences: An Agenda for Future Research and Evaluation. CBE Life Sci. Educ. 14 (1), es1. doi:10.1187/cbe.14-10-0167

PubMed Abstract | CrossRef Full Text | Google Scholar

Creswell, J. W., and Poth, C. N. (2017). Qualitative Inquiry and Research Design: Choosing Among Five Approaches. Thousand Oaks, CA: SAGE Edge. 9781506330204.

Google Scholar

Crisp, G., and Cruz, I. (2009). Mentoring College Students: A Critical Review of the Literature between 1990 and 2007. Res. High Educ. 50, 525–545. doi:10.1007/s11162-009-9130-2

CrossRef Full Text | Google Scholar

Cutucache, C., Boham, T., Luhr, J., Sommers, A., Stevenson, N., Sointu, E., et al. (2018). Ne Stem 4U Afterschool Intervention Leads to Gains in STEM Content Knowledge for Middle School Youth. Cogent Educ. 5, 1558915. doi:10.1080/2331186X.2018.1558915

CrossRef Full Text | Google Scholar

Cutucache, C. E., Luhr, J. L., Nelson, K. L., Grandgenett, N. F., and Tapprich, W. E. (2016). NE STEM 4U: An Out-Of-School Time Academic Program to Improve Achievement of Socioeconomically Disadvantaged Youth in STEM Areas. Int. J. STEM Edu. 3 (1), 1–7. doi:10.1186/s40594-016-0037-0

CrossRef Full Text | Google Scholar

Dagley, M., Georgiopoulos, M., Reece, A., and Young, C. (2016). Increasing Retention and Graduation Rates through a STEM Learning Community. J. Coll. Student Retention: Res. Theor. Pract. 18 (2), 167–182. doi:10.1177/1521025115584746

CrossRef Full Text | Google Scholar

Desmarais, B. A., and Harden, J. J. (2013). Testing for Zero Inflation in Count Models: Bias Correction for the Vuong Test. Stata J. 13 (4), 810–835. doi:10.1177/1536867x1301300408

CrossRef Full Text | Google Scholar

Evans, N. J., Forney, D. S., Guido, F. M., Patton, L. D., and Renn, K. A. (2009). Student Development in College: Theory, Research, and Practice. John Wiley & Sons.

Google Scholar

Fan, X., Miller, B. C., Park, K.-E., Winward, B. W., Christensen, M., Grotevant, H. D., et al. (2006). An Exploratory Study about Inaccuracy and Invalidity in Adolescent Self-Report Surveys. Field Methods 18 (3), 223–244. doi:10.1177/152822x06289161

CrossRef Full Text | Google Scholar

Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., et al. (2014). Active Learning Increases Student Performance in Science, Engineering, and Mathematics. Proc. Natl. Acad. Sci. U S A. 111 (23), 8410–8415. doi:10.1073/pnas.1319030111

PubMed Abstract | CrossRef Full Text | Google Scholar

Fulton, K., and Britton, T. (2011). National Commission on Teaching and America’s Future. Available at: https://files.eric.ed.gov/fulltext/ED521328.pdf.STEM Teachers in Professional Learning Communities: From Good Teachers to Great Teaching

Google Scholar

García, E., and Weiss, E. (2019). The Perfect Storm in the Teacher Labor Market" Series. Washington, D.C.: Economic Policy Institute. The Teacher Shortage Is Real, Large and Growing, and Worse Than We Thought. The First Report

Google Scholar

Graham, M. J., Frederick, J., Byars-Winston, A., Hunter, A. B., and Handelsman, J. (2013). Science Education. Increasing Persistence of College Students in STEM. Science 341 (6153), 1455–1456. doi:10.1126/science.1240487

PubMed Abstract | CrossRef Full Text | Google Scholar

Heaney, A., and Fisher, R. (2011). Supporting Conditionally-Admitted Students: A Case Study of Assessing Persistence in a Learning Community. J. Scholarship Teach. Learn. 11 (1), 62–78.

Google Scholar

Hill, W., and Woodward, L. S. (2013). Examining the Impact Learning Communities Have on College of Education Students on an Urban Campus. J. Coll. Student Dev. 54 (6), 643–648. doi:10.1353/csd.2013.0085

CrossRef Full Text | Google Scholar

Jacobi, M. (1991). Mentoring and Undergraduate Academic success: A Literature Review. Rev. Educ. Res. 61, 505–532. doi:10.3102/00346543061004505

CrossRef Full Text | Google Scholar

Kotys-Schwartz, D., Besterfield-Sacre, M., and Shuman, L. (2011). “Informal Learning in Engineering Education: Where We Are—Where We Need to Go,” in Frontiers in Education Conference (FIE), T4J1-T4J7. Available at: http://fie-conference.org/fie2011/papers/1235.pdf.

Google Scholar

Kuh, G. D. (2008). Excerpt from High-Impact Educational Practices: What They Are, Who Has Access to Them, and Why They Matter. Assoc. Am. Colleges Universities 14 (3), 28–29.

Google Scholar

Leas, H. D., Nelson, K. L., Grandgenett, N., Tapprich, W. E., and Cutucache, C. E. (2017). Fostering Curiosity, Inquiry, and Scientific Thinking in Elementary School Students: Impact of the NE STEM 4U Intervention. Jyd 12 (2), 103–120. doi:10.5195/jyd.2017.474

CrossRef Full Text | Google Scholar

Leggette, H. R., and Jarvis, H. (2015). How Students Develop Skill an Identity in an Agricultural Communications Course. J. Appl. Commun. 99 (1), 38–51. doi:10.4148/1051-0834.1039

CrossRef Full Text | Google Scholar

Lengelle, R., Meijers, F., Poell, R., and Post, M. (2014). Career Writing: Creative, Expressive and Reflective Approaches to Narrative Identity Formation in Students in Higher Education. J. Vocational Behav. 85, 75–84. doi:10.1016/j.jvb.2014.05.001

CrossRef Full Text | Google Scholar

Levett-Jones, T. L. (2007). Facilitating Reflective Practice and Self-Assessment of Competence through the Use of Narratives. Nurse Educ. Pract. 7 (2), 112–119. doi:10.1016/j.nepr.2006.10.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Long, J. S. (1997). “Regression Models for Categorical and Limited Dependent Variables,” in A Handbook of Statistical Analyses Using R. Editors B. S. Everitt, and T. Hothorn (Thousand Oaks, CA: Sage Publications).

Google Scholar

MacPhee, D., Farro, S., and Canetto, S. S. (2013). Academic Self-Efficacy and Performance of Underrepresented STEM Majors: Gender, Ethnic, and Social Class Patterns. Analyses Soc. Issues Public Pol. 13 (1), 347–369. doi:10.1111/asap.12033

CrossRef Full Text | Google Scholar

Merriam, S. B., and Tisdell, E. J. (2015). Qualitative Research: A Guide to Design and Implementation. 4th Edition. Hoboken, NJ: Jossey-Bass. 978-1-119-00361-8.

Google Scholar

Metcalf, H. (2016). Broadening the Study of Participation in the Life Sciences: How Critical Theoretical and Mixed-Methodological Approaches Can Enhance Efforts to Broaden Participation. CBE Life Sci. Educ. 15 (3), rm3. doi:10.1187/cbe.16-01-0064

PubMed Abstract | CrossRef Full Text | Google Scholar

Miles, M. B., Huberman, A. M., and Saldaña, J. (2014). Qualitative Data Analysis: A Methods Sourcebook. 3rd ed. Thousand Oaks, CA: Sage Publications.

Google Scholar

Mourad, T. M., McNulty, A. F., Liwosz, D., Tice, K., Abbott, F., Williams, G. C., et al. (2018). The Role of a Professional Society in Broadening Participation in Science: A National Model for Increasing Persistence. BioScience 68 (9), 715–721. doi:10.1093/biosci/biy066

CrossRef Full Text | Google Scholar

National Academies of Sciences (2016). Engineering, and Medicine, Barriers and Opportunities for 2-year and 4-year STEM Degrees: Systemic Change to Support Students' Diverse Pathways.

Google Scholar

National Student Clearinghouse Research Center (2019). High School Benchmarks 2019: National College Progression Rates. Herndon, VA: National Student Clearninghouse.

Google Scholar

Nelson, K. L., Rauter, C. M., and Cutucache, C. E. (2018). Life Science Undergraduate Mentors in NE STEM 4U Significantly Outperform Their Peers in Critical Thinking Skills. CBE Life Sci. Educ. 17 (4), ar54. doi:10.1187/cbe.18-03-0038

PubMed Abstract | CrossRef Full Text | Google Scholar

Nelson, K. L., and Cutucache, C. E. (2017). How Do Former Undergraduate Mentors Evaluate Their Mentoring Experience 3-years post-mentoring: A Phenomenological Study. Qual. Rep. 22 (7), 2033. doi:10.46743/2160-3715/2017.2991

CrossRef Full Text | Google Scholar

[NRC] National Research Council (2012). Discipline-based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering. Washington, DC: The National Academies Press. doi:10.17226/13362

CrossRef Full Text | Google Scholar

[NSTC] National Science and Technology Council’s Committee on STEM Education (2018). Charting a Course for Success: America’s Strategy for STEM Education. Available at: www.whitehouse.gov/STEM-Education-Strategic-Plan-2018 (Accessed April 2, 2020).

Google Scholar

Pascarella, E. T., and Terenzini, P. T. (1991). How College Affects Students: Findings and Insights from Twenty Years of Research. San Francisco, CA: Jossey-Bass.

Google Scholar

Pasque, P. A., and Murphy, R. (2005). The Intersections of Living-Learning Programs and Social Identity as Factors of Academic Achievement and Intellectual Engagement. J. Coll. Student Dev. 46 (4), 429–441. doi:10.1353/csd.2005.0041

CrossRef Full Text | Google Scholar

Rahm, J. (2014). Reframing Research on Informal Teaching and Learning in Science: Comments and Commentary at the Heart of a New Vision for the Field. J. Res. Sci. Teach. 51 (3), 395–406. doi:10.1002/tea.21141

CrossRef Full Text | Google Scholar

Rosenman, R., Tennekoon, V., and Hill, L. G. (2011). Measuring Bias in Self-Reported Data. Int. J. Behav. Healthc. Res. 2 (4), 320–332. doi:10.1504/IJBHR.2011.043414

CrossRef Full Text | Google Scholar

Russell, L. (2017). Can Learning Communities Boost success of Women and Minorities in STEM? Evidence from the Massachusetts Institute of Technology. Econ. Edu. Rev. 61, 98–111. doi:10.1016/j.econedurev.2017.10.008

CrossRef Full Text | Google Scholar

Saldaña, J. (2016). The Coding Manual for Qualitative Researchers. 3rd Ed. Thousand Oaks, CA: Sage.

Google Scholar

See, B. H., and Gorard, S. (2019). Why Don't We Have Enough Teachers?: A Reconsideration of the Available Evidence. Res. Pap. Edu. 35, 416–442. doi:10.1080/02671522.2019.1568535

CrossRef Full Text | Google Scholar

Soldner, M., Rowan-Kenyon, H., Inkelas, K. K., Garvey, J., and Robbins, C. (2012). Supporting Students' Intentions to Persist in STEM Disciplines: The Role of Living-Learning Programs Among Other Social-Cognitive Factors. J. Higher Edu. 83 (3), 311–336. doi:10.1353/jhe.2012.0017

CrossRef Full Text | Google Scholar

Stake, R. (2005). “Qualitative Case Studies,” in The Sage Handbook of Qualitative Research. Editors N. K. Denzin, and Y. S. Lincoln. 3rd ed. (Thousand Oaks, CA: Sage), 443–466.

Google Scholar

Stevenson, N., Sommers, A., Grandgenett, N., Tapprich, W., and Cutucache, C. (2021). The Informal Learning Review No, 166, 24–29. Available at: https://informallearning.com/storage/issues/166%20-%20March-April%202021.pdf.NE STEM 4U: An 8-year Reflection on Building the Next Generation of the STEM Workforce via Professional Development Experiences

Google Scholar

Talbot, R. M., Hartley, L. M., Marzetta, K., and Wee, B. S. (2015). Transforming Undergraduate Science Education with Learning Assistants: Student Satisfaction in Large-Enrollment Courses. J. Coll. Sci. Teach. 44 (5), 24–30.

Google Scholar

Theobald, E. J., Hill, M. J., Tran, E., Agrawal, S., Arroyo, E. N., Behling, S., et al. (2020). Active Learning narrows Achievement Gaps for Underrepresented Students in Undergraduate Science, Technology, Engineering, and Math. Proc. Natl. Acad. Sci. U S A. 117 (12), 6476–6483. doi:10.1073/pnas.1916903117

PubMed Abstract | CrossRef Full Text | Google Scholar

Vest, C. M. (2010). Rising above the Gathering Storm, Revisited: Rapidly Approaching Category 5. Washington, DC: National Research Council.

Google Scholar

Wilcox, R. R. (2005). Introduction to Robust Estimation and Hypothesis Testing. Academic Press. 0127515429.

Google Scholar

Yin, R. (2018). Case Study Research and Applications: Design and Methods. Thousand Oaks, CA: Sage.

Google Scholar