Programs focusing on undergraduates (UGs) providing mentoring are widespread within and outside of science, technology, engineering, and mathematics (STEM) disciplines. The effects of these programs are not beyond empirical analysis, with much of the existing research on mentoring focusing only on the impact of mentoring on mentees, objective data (e.g., exam scores, course grades, grade point average, etc.), or quantitative data (Crisp and Cruz, 2009; Gershenfeld, 2014), which ultimately limits the scope of understanding and application. Our present study is a systematic review to determine the methodological rigor of research measuring outcomes for UG mentors (i.e., the individuals doing the mentoring, as opposed to those benefiting from the mentoring, as is commonly reported in the literature). We reviewed studies between 2013 and 2020, since 2014 (Gershenfeld) was the last publication on this topic and would not have included articles in press (i.e., during 2012 and published in 2013) at the time of its writing. In all, we identified 80 studies containing quantitative and/or qualitative insights from UG mentors.

Jacobi (1991) review of a decade (1980–1990) of mentoring research on mentor and mentee perspectives proposed a need for improved methodology and reasoned for the importance of situating mentoring programs and research within a theoretical base. Consequently, Jacobi (1991) put forward four major theoretical frameworks of mentoring programs: 1) involvement with learning, 2) academic and social integration, 3) social support, and 4) developmental support. Hannafin et al. (1997) indirectly extended and expounded upon this reasoning for use of the grounded theory design namely alignment of methods, theoretical or conceptual framework, and research are essential in understanding learning environments.

Nora and Crisp’s (2007) report on a survey of UG mentor perspectives and a corresponding literature review detailed the functional roles of mentors and prompted their assertion that mentoring programs and research continued to lack theoretical/conceptual bases. Nora and Crisp (2007) identified four major components that mentoring programs can utilize to provide a strong conceptual base namely 1) education/career goal establishment and evaluation, 2) emotional and psychological support, 3) academic content knowledge support, and 4) presence of a role model. Two years later, Crisp and Cruz, (2009) updated the review by Jacobi (1991), outlining a continued lack of methodological rigor in a wider body of mentoring research between 1990 and 2007.

The last major review prior to this was conducted by Gershenfeld (2014) with the intention of extending the analysis of mentoring research to include published works between 2008 and 2012. Gershenfeld (2014) ultimately reported some improvement in the application of theoretical or conceptual frameworks but similarly outlined persistent methodological shortcomings. Of particular note, Gershenfeld (2014) identified some of what is termed “key mentoring program components” (Supplementary Tables S1, 2) and innovatively applied the Levels of Evidence-Based Intervention Effectiveness (LEBIE; shown in Table 1; Jackson, 2009) scale to evaluate methodological rigor.

However, Gershenfeld (2014) identified a skew in article rankings by the LEBIE scale, assigning only 3’s, 4’s, and 5’s (which are inferior scores, as 5 = concerning). They attributed this skew to the scale’s rankings tending toward typical quantitative studies, in which the presence of equivalent controls and randomization is more common. In isolation, this issue would be significant, but Gershenfeld (2014) employed other forms of evaluation to ensure appropriate analysis of qualitative and mixed-method study designs, a strategy in which the present study adopts as well.

The aim of this study was to extend the analysis of research on the effect of mentoring on mentors, from the last review of such literature (i.e., the period covering 2013–2020). We aimed to address two key research questions:

1) Does the application of the LEBIE scale (Jackson, 2009) to evaluate mentoring research that contains mentor perspectives published between 2013 and 2020 mirror that shared by Gershenfeld (2014)? Or, did the field respond with more expansive mentoring evaluation practices after that publication?

2) Identify “key mentoring program components” (Gershenfeld, 2014), theoretical or conceptual frameworks (if provided), methods, and general findings of the mentoring literature. We sought to determine what these components are, based upon the frameworks of Jacobi (1991) and Hannafin et al. (1997), Nora and Crisp (2007), Crisp and Cruz (2009), and Gershenfeld (2014).

Ultimately, these results will allow for recommendations for future researchers to improve upon methodological rigor in research that studies the impact of mentoring on mentors.

The methods employed for this systematic review are consistent with the practices within the literature, namely of Cronbach and Shapiro (1982) and Moher et al. (2009), using the following Cronbach’s units, treatments, outcomes, and study designs (UTOS) framework. Our population of interest (Units) is UG mentors within STEM and peripheral fields. We focused on the provision of mentoring by UGs (Treatments) as an intervention, including but not limited to mentoring within peer-mentoring, service-learning, course-related, internship, and research programs. The Outcomes we are interested in for eligibility are those reported openly by or requiring insights from UG mentors on what effect the experience had on them. Due to the exploratory nature of this study and the widely variable outcomes measured, we do not further constrict this parameter. However, we did also identify and report on other subjective components (e.g., demographics, compensation, support, frequency, etc.). As one of our major goals is to identify methods employed, all Study Designs are eligible for review, so long as outcomes are reported and are in line with the aforementioned parameter.

We completed a literature search within the Education Resources Information Center database (ERIC) and multiple databases within ESBCO (namely Academic Search Complete, Education Source, E-Journals, PsycARTICLES, PsycINFO, Psychology and Behavioral Sciences Collection, and Teacher Reference Center) using the respective search terms “mentor and undergraduate” in ERIC and “mentor and UG” in EBSCO. One set of search terms could not be used exclusively within both databases due to an issue with ERIC producing only two search results with the latter and EBSCO producing thousands of unrelated results with the former. Our other search criteria included scholarly articles, written in English, peer reviewed, and published between 2013 and 2020 (see Figure 1 for stepwise exclusion). We used a date range that included any articles published, while the Gershenfeld paper (i.e., the last most recent review) would have been under review (i.e., 2013) and through the final full year prior to submission (i.e., 2020). Therefore, this systematic review includes studies from 2013 to 2020, covering the entire ERIC database and multiple databases within EBSCO, and yielding 1,231 positive hits.

After the removal of duplicates, the article titles and abstracts were screened for any indication of findings related to UG mentors (e.g., title and/or abstract explicitly contain the words undergraduate/UG mentors and suggest or explicitly state something about mentor perspectives/insight), which would fulfill our Units parameter. Those included through this initial screening were reviewed in full for eligibility if the focus was on the provision of mentoring by UGs, findings were reported, and insights from the mentors’ perspective were provided (i.e., explicit statements and data were provided to demonstrate each), therefore fulfilling our Outcomes and Treatment parameters. Articles or programs pertaining to service-learning were included only if the service-learning involved provision of mentoring by UGs, and any articles or programs concerning traditional pre-service teaching internship programs (e.g., co-teaching within a classroom setting under the supervision of a certified teacher) were excluded, as such positions do not revolve around the adoption of a mentor role. While mentors may certainly serve as teachers and teachers may certainly serve as mentors, they are generally observed and/or measured as separate roles albeit closely related (Crisp and Cruz, 2009; Gershenfeld, 2014; Jacobi, 1991; Nelson et al., 2017; 2017; Nora and Crisp, 2007), prompting our decision to exclude pre-service programs in order to maintain focus on mentoring in alignment with our Treatment parameter.

Throughout each step of the review process, two authors (ASL and KLN) independently read and evaluated relevant articles/sections (e.g., abstract vs. methods vs. whole document), addressing any discrepancies prior to moving on. We routinely compared independent running documents containing all positive hits and subsequent inclusions of articles/extracted data (i.e., independent versions of Figure 1, Table 1, and Supplementary Tables) in a stepwise manner, while the third author (CEC) addressed any discrepancies not clearly resolved by the other two (e.g., whether certain language indicated a program to be a pre-service program). Nearly all inclusion criteria and data that we collected were concerned with the presence or absence of some attribute or the reporting of what is explicitly stated or not stated by article authors and were based on priorly established frameworks as discussed in our research questions. For this reason, many inconsistencies between reviewers could be attributed to one of the authors missing a qualifying article or not identifying data. However, some discrepancies did arise from unclear language or subjective interpretation (e.g., analytic logic, sequencing, and data priority in studies utilizing mixed methods design). Inconsistencies of the former type were resolved by comparing data sets and identifying where the criteria or data were located within articles/sections, while discrepancies of the latter type were resolved through discussion with the third author.

In total, there were 1,231 positive hits through the database query after duplicate removal. Of these, n = 80 met all of our inclusion criteria and were analyzed by the following evaluative tools. We used the LEBIE scale (Jackson, 2009) to examine methodological rigor (Table 1) in terms of study design (e.g., presence of equivalent vs. non-equivalent vs. no control group) and evidence of effectiveness (e.g., evidence that intervention results in some positive change over time or is better than or comparable to a control/placebo). To examine program and research functionality and qualities, we used (Nora and Crisp, 2007) conceptualization of core functional roles (e.g., assist with a course, provide peer-mentoring, service-learning, etc.) and (Gershenfeld, 2014) key-mentoring program and research components (namely mentor and mentee demographics, compensation, frequency of mentoring, support, N = number of mentors, quantitative vs. qualitative vs. mixed methods, how data are collected, and major findings). In line with prior researchers from Jacobi (1991) and Hannafin et al. (1997) to Nora and Crisp (2007) and Gershenfeld, (2014), we also identified theoretical/conceptual frameworks (if stated by authors).

Finally, for relevant studies, we examined characteristics deemed essential within the literature to mixed methods designs (Supplementary Tables S3), including an explicit statement that mixed methods research is being utilized, rationale for using mixed methods research, integration of quantitative and qualitative data (merging, connecting, or building), analytic logic (independent or dependent), sequencing/timing (concurrent or sequential), and/or priority (quantitative, qualitative, or both; Creswell, 2013; Creswell and Plano Clark, 2017; Harrison et al., 2020; O’Cathain et al., 2008; plano Clark and Ivankova, 2016). We took the former three from eligible studies (i.e., stated or not and what was stated) but interpreted the latter three for all but one. Ultimately, our results will consist of LEBIE scale ratings, compiled qualitative data on program and research qualities, and reporting of relative proportions of qualities where possible. Of note, where we discuss proportions/percentages, the sample size (n) may not equal the total number of eligible studies (n = 80) due to some qualities not being reported or present in certain studies (e.g., mixed methods design), and percentages may add up to be greater than 100% due to certain studies reporting multiple elements within a given quality (e.g., different types of compensation given to different participants).

Consistent with prior research, we have included many components of the articles we reviewed and the mentoring programs they analyzed (contained within the following table and supplemental materials). It is and always was our intention to compile this large amount of data in order to provide easy access to overview these studies for other mentoring researchers (we have grouped similar data together for this reason). However, our primary goal is to identify trends within mentoring programs and research approaches in addition to analyzing methodological rigor in studies on the subject in order to provide suggestions for improvement of future research. To this end, our results and discussion will be focused on our research questions to determine rigor (i.e., Table 1, Supplementary Tables S1, 2) and methodology (i.e., Supplementary Tables S2, 3).