Measuring what matters: implementing the personal and professional development pulse to capture holistic educational gain in higher education

Abstract

Introduction:

Universities play a central role in preparing students with the essential skills they need to thrive in their future careers and broader life experiences. As society and the labour market continues to evolve, the expectations placed on graduates also change. Consequently, higher education institutions must ensure students develop broad and diverse capabilities required for long-term employability and sustained professional success.

Method:

This study reports the development and psychometric validation of the Personal and Professional Development Pulse, an institutionally aligned instrument designed to capture a broad, holistic measure of students’ educational gain throughout their university degree. The Pulse comprises 22 self-reflective statements, grouped into one of 5 domains of educational development, with statements answered on a rating scale of strongly disagree (1), to strongly agree (7). The Pulse was completed by 5180 new entrants. Responses were Rasch analyzed to determine the psychometric properties of the questionnaire.

Results:

Results indicated ordered category functioning (i.e., each response category becomes most likely at a progressively higher level of ability/confidence), a strong item hierarchy (item separation = 14.95; reliability = 1.00—Items are consistently ordered from easiest to hardest, and that ordering is reliable and meaningful), and good discrimination between respondents (person separation = 3.00; reliability = 0.90—distinguishing between responders with different levels of confidence/ability), alongside acceptable fit statistics (statements are valid and coherent in what they intent to measure). Principal component analysis supported acceptable unidimensionality, with all items aligning to a single underlying construct.

Discussion:

The validated Pulse provides a robust, holistic measure of students’ educational gain across their degree, capturing key graduate capabilities. Findings identified relative strengths in collaboration, application and empathy. Employability and networking items required higher ability/confidence, indicating that these items take longer to develop. Institutional recommendations include implementing employability and networking related opportunities consistently within curricular and co-curricular delivery. Maximising the impact of the Pulse should be considered as a complementary component of a broader, holistic approach to student development, incorporating reflective dialogue, co-curricular engagement, and opportunities to evidence capability development over time.

Introduction

Graduate outcomes and employability have become central priorities within higher education (HE), reflecting increasing expectations that universities cultivate the broad transferable skills required for students to succeed in a rapidly changing labour market. Research highlights the multidimensional nature of employability and the need for institutions to embed competence development within curriculum and co-curricular structures (Abelha et al., 2020). Systematic reviews demonstrate persistent mismatches between graduates’ skills and employers’ needs, underscoring the importance of integrating both technical and transversal competencies—such as communication, problem solving, teamwork and adaptability—into programme design (Abelha et al., 2020). Broader theoretical perspectives also emphasise that employability extends beyond immediate employment outcomes, instead representing a lifelong process of developing and mobilising different forms of capital through curricular, extra-curricular and work-integrated learning experiences (Nghia et al., 2022). Complementing this, narrative reviews of employability models argue that universities must move beyond narrow, skills-based approaches to consider relational, contextual and pedagogical dimensions of educational development that prepare graduates for both current and future challenges (Cook, 2022).

The regulatory body for English HE, the Office for Students (OfS), has previously made efforts to capture the holistic education development (or gains) students make during their education and subsequent readiness for employment. However, previous works commissioned by the OfS were unsuccessful in finding a single measure able to capture holistic education gain which could be used across the sector. Notable examples include Higher Education Learning Gain Analysis (HELGA; OfS, 2019) which investigated the feasibility of using existing administrative data, drawn from sources including Universities and Colleges Admissions Service (UCAS) and Higher Education Statistics Agency (HESA), and The National Mixed Methods Learning Gain Project (NMMLGP; Jones-Devitt et al., 2019) which explored the feasibility of using a national-level, centrally administered survey. The absence of agreement in defining holistic educational gain across the sector led to several suggesting this as “the ‘dark matter’ of student outcomes—Like dark matter, we know it's out there, we just can't pin it down” (Kandiko Howson, 2022).

The OfS originally positioned holistic educational gain as an opportunity for “universities and colleges to understand the effect of different teaching practices and therefore improve the support available to students” (OfS, 2022). Recently, however, the current regulatory narrative from the OfS has altered, shifting from ‘enhancement to assurance’ and from ‘curiosity to compliance’ (Wonkhe, 2025). A lack of consensus in measuring holistic educational gain has resulted in variations of understanding (McGrath et al., 2015), shaped by the initial motivation for employing such a measure. The broader political and regulatory context in terms of quality, performance, accountability of HE and associated funding linked to these aforementioned criteria (Kandiko Howson, 2022) provides a stark contrast to a student-centred approach which recognises students’ own agency in becoming successful, where defining ’success’ is constantly negotiated and achievable through their personal reflective journey, ultimately leading to self-transformation (Moon, 2000).

Measuring holistic educational gain permeates beyond the English HE system. Within the United States of America (USA), holistic educational gain is more embedded in the culture of HE (Randles and Cotgrave, 2017) with grades (grade point average-GPA) and standardised tests (the Collegiate Learning Assessment-CLA, and the Measure of Academic Proficiency and Progress, MAPP) used in a widespread manner (McGrath et al., 2015). However, the USA has similarly seen challenges with using a single, standardised measure of educational gain, due to the freedom afforded to the individual institution to determine the evaluation approach taken; as is the case within the United Kingdom (UK, Andrade, 2018). Furthermore, institutions oppose the use of a standardised national test due to its failure to capture the diversity, breadth and depth of discipline-specific knowledge and learning (Douglass et al., 2012), and the outcomes reflecting a small proportion of gains resulting from completing a HE degree (Banta and Pike, 2012). Within Australasia, utilised alongside GPA, the Australasian Survey of Student Engagement (AUSSE) captures student self-reported learning and development across a range of dimensions and provides an additional lens (beyond student grades) through which to view student educational gain (Hagel et al., 2012).

The number of competing ‘voices’ within the holistic educational gain landscape has partly influenced the range of methods used to measure educational gain. McGrath et al. (2015) provided a summary of methods that could be used to measure educational gain; they can be grouped into the following areas: Grades (degree classifications, GPAs or exam scores); Surveys (nationally administered surveys such as student experience and student satisfaction surveys and locally developed such as skills audit and career readiness); Standardised tests (discipline-specific or general skills relating to elements of personal development or content knowledge); and Mixed methods (combining several tools).

A criticism of applying quantitative metrics to define holistic educational gain is the inability of the measure to sufficiently capture a student's broad development during their degree (Evans et al., 2018). Boud (2018) argues that student outcomes, as captured through formal assessments, are not appropriate proxies for educational gain, as the aggregation of diverse learning outcomes into a single final mark obscures the processes of development that occur over time. This position is supported by McGrath et al. (2015), who distinguishes between attainment and learning gain, noting that whilst final grades reflect the level of performance achieved, learning gain is concerned with the extent of change in a student's knowledge, skills, and understanding across their educational journey. Furthermore, these metrics were not originally designed to capture gains in learning as we have now come to understand, with such approaches not necessarily suitable for institutions with a widening participation mission, who welcome students with a variety of entry profiles, beyond traditional formal qualifications, including those with industry experience. In response to this criticism, more recently, frameworks have been developed and translated into questionnaires for students to self-reflect on their broader educational gains (e.g., Speight et al., 2018; Vermunt et al., 2018).

Specifically designed questionnaires have merit in terms of providing opportunity to capture students’ broader educational gains, however, to ensure the results obtained from the questionnaire are valid, it is necessary to ensure the questionnaire is adequate for the intended use in the intended target population and performance of the questionnaire provides acceptable levels of validity and reliability (Rosellini and Brown, 2021; Pesudovs et al., 2007; Stefana et al., 2025).

Utilising best practice frameworks for the development and psychometric evaluation of measurement instruments (e.g., Rosellini and Brown, 2021; Pesudovs et al., 2007; Stefana et al., 2025), the current paper develops and evaluates the performance of a questionnaire to understand university students’ holistic educational gain, which aligns to the widening participation mission and values of the host institution, presents opportunity for institutional reflection regarding its student body and their development towards graduate employability. The current paper presents, through a system of quality assessment, the step-by-step development of the questionnaire (articulated in the methods section of this paper) and the performance (presented in the results). Through taking this systematic approach, the paper aims to provide, for others across the sector, a valid and reliable questionnaire which is an appropriate vehicle for capturing their students’ holistic educational development which aligns to the essential skills students need to thrive in their future careers and broader life experiences.

Materials and method

Development of the instrument

Pre-study hypothesis and intended population

When developing any instrument, best practice requires clearly stating the specific construct the instrument is intended to measure and the intended population of study (e.g., Rosellini and Brown, 2021; Pesudovs et al., 2007; Stefana et al., 2025). The host institution, Anglia Ruskin University (ARU), has a passion for widening access to, and participation in, HE, which is embodied in its mission, “Transforming lives through innovative, inclusive and entrepreneurial education and research.” ARU's student body is best characterised by its diversity, attracting students from groups that are underrepresented in HE. 30.2% of students fall into quintile 1 of at least one of the Indices of Multiple Deprivation (IMD), Tracking Underrepresentation by Area (TUNDRA) and Income Deprivation Affecting Children Index (IDACI) measures. ARU attracts more mature (57.1% aged 21+), minority ethnic (36.0%), female (62.9%), and local (36.5%) students than the respective sector averages (29.9% aged 21+, 29.0% minority ethnic, 56.1% female, 21.8% local, OfS, 2025). 34.7% of ARU's students have an Access/Foundation/‘other Level 3’ course as their entry qualification (sector average, 17.0%), and 16.7% have ‘other’ entry qualifications—typically mature learners admitted based on their prior and experiential learning (sector average, 8.1%).

ARU aspires to make a tangible positive difference to the lives of students it serves and the lives of those around them. ARU's values—ambition, innovation, courage, community, integrity, and responsibility—inform the leading learning and innovation ecosystem that delivers an inclusive educational experience for students.

ARU is committed to supporting student development as lifelong learners. ARU's holistic, whole curriculum approach draws on Tomlinson (2017) Graduate Capitals framework and is reflected in the ARU Graduate Capitals—Knowledge, Social, Cultural, Identity, Adaptability (Psychological) and The Whole Person.

Actual content area

The development of a questionnaire should clearly state the specific construct the questionnaire is intended to measure (pre-study hypothesis) and the intended population of study (Rosellini and Brown, 2021; Pesudovs et al., 2007; Stefana et al., 2025). The extent to which the questionnaire (content) meets the pre-study hypothesis specifications is somewhat subjective and is recognised through the face validity and content validity (Pesudovs et al., 2007). Face validity resides in the appearance of validity in the absence of empirical testing (Cook and Beckman, 2006). Content validity reflects the extent to which the items in the questionnaire reflect the entirety of the concept being measured (Pesudovs et al., 2007). With the challenge of proving that the items chosen were representative of all possible items (Seiler, 1973), content validity is often presented as a detailed description of steps taken to ensure that the items represent the construct (Cook and Beckman, 2006). In the subsequent section, we demonstrate how actual content area has been adhered to, through detailing the item identification, item selection and pilot questionnaire (Rosellini and Brown, 2021; Pesudovs et al., 2007; Stefana et al., 2025).

Item identification & selection

In keeping with ARU's student-centred approach, students formed a core part of the working group established for developing a question set. Six students joined the working group, providing experiences from 2 mature, 4 female, 3 international, 4 undergraduate and 2 postgraduate (taught degrees), across all four faculties from a range of courses.

Expert knowledge is a valuable resource for developing a question set (Pesudovs et al., 2007) and should include a diverse knowledge base (Stefana et al., 2025). Staff were recruited who provided representation across the diverse subjects ARU offers, and from professional services across different service areas. Representation included the Deputy Director of Student and Library Services, the Head of Employability and Careers, a Careers Information and Systems Manager, a Senior Employability & Careers Adviser, two Senior Education Leaders responsible for student outcomes, three academic subject leads, the Head of Student Success and a learning technologist.

Due to the high proportion of ARU's students who enrol through non-traditional entry routes and therefore do not have relevant recent grade outcomes from their previous education (reported in Pre-study Hypothesis and Intended Population), existing approaches for quantifying educational gain using ‘grades’ or assessment-related metrics were not suitable for ARU's student body (e.g., Evans et al., 2018), instead—using ARU's education strategy as the blueprint for articulating educational gain—the working group reviewed current literature and distilled educational gain into broad domains. Through further refinement against ARU's Values and Graduate Capitals, five broad domains (see Table 1) were proposed to capture educational gain.

From utilising existing questionnaires from across the institution that considered career and graduate readiness, sector-wide questionnaires (National Student Survey, Graduate Outcomes Survey, Student Experience Survey) and relevant literature, questions were initially proposed that aligned to each domain (Table 1). The working group was able to discount duplicate/similar questions and those that did not align with the five broad domains. The final number of questions was determined through the average time taken to complete the institutions’ previous Career Readiness survey (previously 25 questions), user acceptance testing (considering survey fatigue) and a MOSCOW rating system used to frame working group discussions regarding the inclusion (or exclusion) of ‘borderline’ questions.

22 self-reflective statements were initially proposed, each of which aligned to one of the 5 domains (Supplementary Table S1).

Pilot questionnaire

Students and staff were invited to complete the questionnaire. Feedback was sought on: the naming of each domain; the statements (questions within the questionnaire); hints (text to contextualise each statement); and the name of the questionnaire. Free text responses were captured through a user acceptance questionnaire (Table 2). Following 159 responses, content analysis (Neuendorf, 2017) was applied in an inductive manner to the free-text comments to establish areas for further development/refinement.

Findings from the content analysis were presented to the working group and through a consensus-seeking approach resulted in slight rewording of two domain names, as they did not translate into ‘everyday student language’, and greater clarity relating to some of the hints used for contextualising the statements (questions). Utilising the wider student voice through administering the pilot questionnaire, asking students to define the domain names resulted in these being articulated more appropriately. The finalised domain names, along with their definitions—which have been aligned to the institution's values and graduate capitals—are presented in Table 1. A final version of the questionnaire and hints were presented for consultation across various fora with colleagues involved with key Education and Employability strategic responsibility and in formal settings, seeking approval for use (Supplementary Table S1). The name of the educational gain tool was confirmed as the Personal and Professional Development Pulse.

Delivery of the questionnaire

An online platform was developed to deliver the Personal and Professional Development Pulse, utilising a 7-point response scale: Strongly Disagree (1), Disagree (2), Slightly Disagree (3), Neutral (4), Slightly Agree (5), Agree (6), Strongly Agree (7).

Ethical permission for the study was provided by the Institution Research Ethics Panel. Students were invited to participate between September 2024 and May 2025 either by university-wide email or via their online learning platform. Participants provided consent electronically during their registration.

Results

A number of statistical methodologies can be used to understand the performance of the instrument e.g., Conventional descriptive statistics, Cronbach's alpha, Factor analysis and Rasch analysis (Rosellini and Brown, 2021; Pesudovs et al., 2007); Rasch analysis was chosen for the current analysis due to the additional depth afforded within the analysis.

Rasch analysis is a probabilistic model that converts raw ordinal scores into an interval scale (Bond and Fox, 2001; Bond et al., 2021), scored in log odds units, or logits. Person and item measure estimates are determined for the underlying construct, which are amenable to parametric statistical analysis. Rasch analysis also allows examination of the properties of a questionnaire.

Winsteps version 5.2.1 (Linacre, 2008) was used to Rasch analyse the Personal and Professional Development Pulse statements with a single Andrich rating scale. A higher person measure score represents greater perceived ability (i.e., in this context, ‘higher confidence’) in the parameter in question. A higher item difficulty score indicates that perceived ability/confidence must be better to be able to agree with the item (i.e., a ‘harder’ question).

At the start of the academic year 2023/24, all new entrants (including all levels of undergraduate study and post-graduate taught) were invited to take the Personal and Professional Development Pulse. 5,180 new entrants responded, providing representation from each Faculty and all levels of study, representing a 37% completion rate. The profile of participants who completed the Personal and Professional Development Pulse reflected the broader student population (n = 29,074); mature (i.e., aged 21+) 58.3% (institution 57.1% aged 21+), minority ethnic 33.1% (institution 36.0%), female 59.8% (institution 62.9%), and local 37.7% (institution 36.5%); institutional population represents ‘core’ students, excluding partnerships and franchises. Data availability: https://doi.org/10.25411/aru.30860066.

Initial observation of the data highlighted that within the 5180 responses, approximately 1.5% of the sample responded with a maximum self-confidence score to all statements (i.e., self-rating ‘7’ to all statements). Approximately 2.8% of the sample responded with a minimum self-confidence score to all statements (i.e., self-rating ‘1’ to all statements).

Preliminary Rasch analysis

Initial rating scale analysis examines how well the response categories (e.g., strongly disagree to strongly agree) function, including whether each higher response category represents a higher level of the underlying trait, occur with sufficient frequency, and progress logically along the underlying construct (Linacre, 1999). Results indicated that category function measures were ordered [(−2.20), −1.26, −0.75, −0.29, +0.32, +1.41, (+3.21) logits], highlighting that each response category becomes most likely at a progressively higher level of ability/confidence. Options 2 (‘disagree’) and 3 (’slightly disagree’) were the least frequently selected options and were not the most commonly used option at any point on the functional scale. One approach could be to collapse responses to these two categories together, so that the combined category became the most frequently selected at the relevant point on the functional scale. However, when this was done, no manifest difference to the results arose in terms of item difficulties or fits. It was therefore decided to present the uncollapsed data, both for ease of interpretation and to avoid having fewer disagreement than agreement categories.

Fit statistics are used to establish the function of individual items in the Rasch model, showing whether they reliably measure the intended construct. Infit statistics are concerned with responses closer to the mean ability, and outfit statistics are influenced by off-target responses away from the mean ability. Mean-square fit statistics range from zero to infinity where an infit and outfit of 1 represent expected fit of an item to the Rasch model. Fit values <1 indicate overfit suggesting item redundancy, and values >1 indicate misfit suggesting that the items are measuring something other than the proposed construct. The ideal range of fit values to indicate that items are contributing to the scale is 0.6–1.4 (Latham et al., 2015), but items up to 2.0 are not considered to be damaging to the scale (Bond et al., 2021; Latham et al., 2017; Wright and Linacre, 1994). Table 3 indicates only 1 item (statement 14) with slightly sub-optimal outfit value, and all items have been retained in the analysis.

Instrument performance

Performance of the instrument as a whole is now considered. Person separation statistics provide an indication of the ability of the instrument to discriminate between respondents. Good discrimination between respondents indicates that the instrument can clearly distinguish between people with different levels of the trait (e.g., confidence/ability), with a person separation index of 2.0 (reliability >0.80: a similar measure to the traditional Cronbach alpha coefficient; Bond et al., 2021; Bond and Fox, 2001Gothwal et al., 2009); suggested as a minimum acceptable level (Bond et al., 2021; Bond and Fox, 2001; Gothwal et al., 2009). The derived person separation of 3.00, and person reliability of 0.90 are greater than these values, indicating acceptable ability of the instrument to order people by functional level. Item separation statistics provide an indication of the reliability of the ordering of item difficulty, allowing meaningful interpretation of progression from easier to more demanding aspects of the construct. An item separation index of 3.0 (reliability >0.90) is suggested as a minimum acceptable level (Bond et al., 2021; Bond and Fox, 2001; Gothwal et al., 2009). The derived item separation of 14.95, and person reliability of 1.00 are greater than these values, indicating the ordering of item difficulty is robust. The mean person measure is 1.03 (SD 1.56) logits, indicating that this sample has, on average, perceived ability that is better than the average difficulty of the items. Targeting differences of >1.0 logit could be taken to indicate that items of ‘greater difficulty’ could be included to better discriminate between the perceived ability of those students scoring highly on the current instrument.

Principal component analysis (PCA) of residuals is used to assess whether the instrument measures a single underlying construct/latent trait (unidimensionality), or whether there are additional dimensions, which would suggest other constructs are being measured. The variance in the data that is accounted for by the Rasch dimension is first considered, with at least 60% of variance explained by the primary measure considered to demonstrate reasonable overall unidimensionality in the instrument (Gothwal et al., 2009). 46.6% of variance in the data was explained by the primary Rasch measure, slightly lower than ideal. For there to be considered a significant second dimension however, the first contrast found within the residuals after the primary model has been extracted should have a strength of at least two to three items, i.e., an eigenvalue of 2.0–3.0 (Marella et al., 2009; Tabrett and Latham, 2011). After extraction of the primary model, an additional contrast was seen in the residuals with an eigenvalue of 2.5 (6.0% variance explained), although the level of variance explained by the contrast was only 13% of that explained by the primary measure. Four items loaded positively (correlation >0.40) on the contrast: statements 9 (0.65), 7 (0.63), 10 (0.59), and 6 (0.55). Given that all items in the contrast came from the ‘Employability’ statements, treating these questions as a separate subscale may have future value, but in the present analysis the level of unidimensionality demonstrated was considered acceptable.

The perceived ability of the respondents and difficulty of the items are summarised in the person-item map shown in Figure 1, and item difficulties in Table 3.

Figure 1

Discussion

Universities are instrumental in equipping students with the wide-ranging skills needed to succeed both in their future careers and in life more broadly. As societal demands and the labour market continue to shift, so too do the expectations placed upon graduates. In response, HE institutions must continually evolve their practices to ensure students develop the broad, adaptable capabilities that underpin long-term employability and sustained professional growth. This study reports the development and psychometric validation of the Personal and Professional Development Pulse, an institutionally aligned instrument designed to capture a broad, holistic measure of students’ educational gain throughout their university degree. The Personal and Professional Development Pulse reflects a student-centred approach which recognises students’ own agency in becoming successful, understanding ’success’ as something which requires constant negotiation and is achievable through a personal reflective journey, ultimately leading to self-transformation (Moon, 2000).

Performance of the personal and professional development pulse

Through analysing responses of the Pulse, psychometric properties obtained through Rasch analysis demonstrate the questionnaire behaved acceptably as a robust outcome measure, it was possible reliably to differentiate easier to harder items (statements—item ordering) and clearly distinguish between people with different levels of the trait (e.g., confidence/ability—good person ordering). 1 item (statement 14, Table 3) reported a slightly sub-optimal outfit value. The statement encourages responders to consider whether their “academic performance so far is in line with my [their] career aspirations”. Considering that the Personal and Professional Development Pulse is delivered at the start of a student's degree, a slightly sub-optimal outfit value may be a consequence of differing interpretations of the question, with some reflecting on the relative immediacy since starting in HE, and others reflecting on their broader educational journey, which includes previous education environments; a clearer hint associated with this statement could provide greater clarity in future iterations (Supplementary Table S1, question nr 1).

The mean person measure (+1.03, SD 1.56) suggests that the sample is generally more able than the items (items are ‘easy’). Within the sample collected, where students were ‘new’ to the institution (irrespective of their level of study), they could have over-reported their ability, either intentionally (wary of the results from the tool being used as an initial assessment of competence) or a consequence of the Dunning-Krueger effect—a cognitive bias where people with low ability or knowledge in a domain tend to overestimate their competence (Kruger and Dunning, 1999). Mindful of either outcome, during the development and implementation of the Personal and Professional Development Pulse, presentation of the Pulse was coupled with clear communication of the personal nature of the exercise, with results not being used by the institution as a measure of ability. Furthermore, the Pulse was also presented within a package of digital reflective resources, (drawing on the work of Borton, 1970) to encourage students to reflect on their abilities and skills prior to engaging with the Pulse.

Whist the implementation of the Pulse was presented within a package of digital reflective resources (which included videos introducing reflection, vignettes from students highlighting the value of reflection in their practice and opportunity to engage with a structured reflection linked to their own practice), the ability for students to engage in meaningful reflection (and by implication engage with the self-reflective nature of the Pulse) warrants additional discussion. As students enter HE they will have varying understanding of reflection/reflective practice, having experienced it differently during their previous education environment or place of work. Reflective practice is becoming increasingly prominent within HE, in both formal (summative) and informal settings and is already used widely in many subjects/disciplines (e.g., Education and Health based studies, Mohamed et al., 2022). However, at the start of the student journey, when initially taking the Pulse, to more effectively support students in developing their reflective skills, additional support is likely required, beyond the digital resources previously highlighted. Supporting individuals achieving depth and critical insight within their reflection can be achieved with others in groups (Gibbs, 1988; Grant et al., 2017) or through one-on-one sessions (Karnieli-Miller, 2020). Of note, since implementation of the Pulse, across the institution, within the first six weeks of students starting their course, students meet with their personal tutor (individually or in small groups) to reflect on their Pulse scores, consider whether they accurately reflect their ability/confidence and develop an action plan to support their continued development.

What the results tell us about our students

Figure 1 indicates the relative difficulty of each item in the Personal and Professional Development Pulse. Easier items (i.e., participants perceived themselves as having higher ability, or confidence in their ability) included teamwork (Q3), empathy (Q1), working independently (Q4) and application of knowledge (Q20). The relative confidence with which students responded to questions related to collaboration, application, and empathy, reflects the institution's strong portfolio of vocational/health and social care courses offered. Allen’s (2002) research into student decision making regarding the institution at which to study identified that students spontaneously ’sensed’ that an institution ‘felt right’ for them to enrol and study, establishing a feeling of ‘fit’ as the prospective student could ‘place themselves’ in the institution as a result of being able to reasonably see themselves attending (Diamond et al., 2012). Establishing a sense of ‘belonging’ has also been shown to impact student decision making regarding the institution at which to study. In this context, belonging refers to a deeper connection to, and familiarity with, a community and environment (Tovar and Simon, 2010) and considers how students perceive themselves in relation to the characteristics of others within institutional environments (Reay et al., 2001; Thomas and Quinn, 2007). Students viewed institutions where there were others ‘like them’ (e.g., ethnic diversity, mature students or similar social class backgrounds) as a ‘low risk’ decision to study (Reay et al., 2001). Coupled with an institutional curriculum design whereby these courses maximise opportunities for students to apply their strengths, through team working, applying knowledge, and demonstrating patient/professional skills, it is therefore reasonable to suggest that the nature of the courses offered within the institution resonates with applicants, and those with these skills are attracted to enrol as they perceive existing strengths in this area and experience emotions including ‘fitting in’ and ‘belonging’ when choosing the institution at which to study.

The ‘hardest’ items (i.e., perceive themselves having lower ability, or confidence in their ability) reflected in the Personal and Professional Development Pulse were questions linked to employability (Q10, Q9, Q7, Q6), networking (Q2) and academic performance (Q14; Figure 1). These difficulties are reflected in the literature, with students from less advantaged backgrounds encountering barriers when seeking graduate-level employment (Lowe, 2023). These challenges stem not only from systemic bias and discrimination but also from limited access to social networks, cultural capital, and parental guidance, consequently often lacking support for essential activities such as preparing job applications, developing CVs, and exploring graduate career pathways (Lowe, 2023). To prepare students and graduates to compete effectively in a globalized knowledge-based economy, the core curriculum must contain opportunities to establish central employment skills and employability development activities (Hirudayaraj, 2011). Whilst it is recognised that universities provide well-supported co-curricular careers and employability services, there remains a disparity across individual disciplines with respect to employability-related content in the taught and assessed curriculum (Lowe, 2023). With employability opportunities and professional experiences highlighted as a significant factor in predicting career readiness and employment outcomes (Lin-Stephens et al., 2024), curriculum enhancements should consider the inclusion of live briefs (also termed industry brief or applied project), whereby students are provided with an authentic experience solving problems or delivering projects for an external organisation or client, and broader assessments that serve to develop and evaluate skills that employers test and value, such as CV and cover letter writing, mock interviews, group projects simulating workplace collaboration, critical thinking/evaluative judgement and reflective portfolios demonstrating skill development.

Conclusion

While the present study demonstrates that the Personal and Professional Development Pulse provides a valid and reliable mechanism for capturing holistic educational gain within the specific context of Anglia Ruskin University (ARU), it is important to acknowledge both the scope and intended function of the instrument. Rather than seeking to establish a universal or fully generalisable measure of holistic educational gain, the Pulse is positioned as a contextually grounded tool, aligned to institutional values, graduate attributes, and the lived realities of a widening participation student body. This approach reflects wider methodological guidance, which emphasises that measurement instruments should be explicitly defined in relation to their intended construct and population, rather than assuming cross-contextual equivalence (e.g., Rosellini and Brown, 2021; Stefana et al., 2025). More broadly, the challenge of creating unified or standardised measures of educational gain across diverse institutional and national contexts is well documented, particularly where constructs such as employability, identity development, and personal growth are shaped by local curriculum design, student demographics, and socio-cultural factors. Recent methodological reviews highlight that the development and validation of such tools necessarily involves iterative, context-sensitive processes, with adaptation required when transferring instruments across settings (Stefana et al., 2025).

With regard to transferability, the underlying structure of the Pulse, particularly the identification of broad domains relating to personal, academic, and professional development, may have relevance to institutions with comparable widening participation profiles and strategic emphasis on graduate outcomes. However, further work would be required to establish its applicability across different institutional contexts, student populations, and disciplinary settings. This would include additional psychometric validation, testing for invariance across subgroups, and potential recalibration of items to reflect local priorities and student experience. As such, the present study provides a foundation for future research and application which includes a clear methodology for other to utilise, rather than a definitive solution, highlighting both the potential and the limitations of attempting to measure holistic educational gain in higher education.

The value of the Personal and Professional Development Pulse lies not in standardisation, but in its ability to operationalise holistic educational gain in a way that is meaningful, interpretable, and actionable within a defined institutional environment. Importantly, the findings indicate that the instrument should not be considered a standalone measure, but rather as a complementary component of a broader, holistic approach to student development, incorporating reflective dialogue (e.g., through personal tutoring or professional development tutoring), co-curricular engagement, and opportunities to evidence capability development over time. Such approaches align with contemporary perspectives on educational gain, which emphasise the integration of quantitative self-report measures with reflective and experiential learning processes to capture complex, multi-dimensional development (e.g., Rook et al., 2026; Rutherford et al., 2025).

The success of the Personal and Professional Development Pulse is therefore contingent on its longitudinal and embedded application across the student lifecycle, with implementation at multiple points throughout the degree to enable the capture of educational ‘distance travelled’. In keeping with the student-centred ethos underpinning the Pulse and maximising its impact within this student body, frequent curricular and co-curricular integration is required, allowing students and staff to interpret results in situ and respond through targeted support and intervention, as part of an increasingly flexible and relational approaches to education provision (Su and Wood, 2023). In practice, and although not the primary focus of this study, implementation at the host institution has been accompanied by a range of supporting resources to ensure that where students identify lower levels of confidence or ability, they are provided with appropriate opportunities to develop and achieve success over the course of their degree and provided with opportunity to reflect on their holistic educational journey.

References

• 1

  AbelhaM.FernandesS.MesquitaD.SeabraF.Ferreira-OliveiraA. T. (2020). Graduate employability and competence development in higher education—a systematic literature review using PRISMA. Sustainability12 (15), 5900. 10.3390/su12155900

  • CrossRef
  • Google Scholar
  • View reference in article

• 2

  AllenD. E. (2002). Toward a theory of consumer choice as sociohistorically shaped practical experience: the fits-like-A-glove (FLAG) framework. J. Consum. Res.28 (4), 515–532. 10.1086/338202

  • CrossRef
  • Google Scholar
  • View reference in article

• 3

  AndradeM. S. (2018). Learning gain–A US perspective. Higher Educ. Pedagog.3 (1), 46–48. 10.1080/23752696.2018.1508361

  • CrossRef
  • Google Scholar
  • View reference in article

• 4

  BantaTWPikeG. (2012). “Making the Case Against—one More Time. the Seven red Herrings About Standardized Assessments in Higher Education” in Handbook on measurement, assessment, and evaluation in higher education, eds. C. Secolsky and D. B. Denison (New York: Routledge), 24–30.

  • Google Scholar
  • View reference in article

• 5

  BondT. G.FoxC. M. (2001). Applying the Rasch Model: Fundamental Measurement in the Human Sciences. Mahwah, NJ: Lawrence Erlbaum Associates.

  • Google Scholar
  • View reference in article

• 6

  BondT. G.YanZ.HeeneM. (2021). Applying the Rasch Model: Fundamental Measurement in the Human Sciences. 4th ed. New York: Routledge.

  • Google Scholar
  • View reference in article

• 7

  BortonT. (1970). Reach, Touch and Teach. London: Hutchinson.

  • Google Scholar
  • View reference in article

• 8

  BoudD. (2018). Assessment could demonstrate learning gains, but what is required for it to do so?Higher Educ. Pedagog.3 (1), 54–56. 10.1080/23752696.2017.1413671

  • CrossRef
  • Google Scholar
  • View reference in article

• 9

  CookD. A.BeckmanT. J. (2006). Current concepts in validity and reliability for psychometric instruments: theory and application. Am. J. Med.119 (2), 166.e7. 10.1016/j.amjmed.2005.10.036

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 10

  CookE. J. (2022). A narrative review of graduate employability models: their paradigms, and relationships to teaching and curricula. J. Teach. Learn. Grad. Employab.13 (1), 37–64. 10.21153/jtlge2022vol13no1art1483

  • CrossRef
  • Google Scholar
  • View reference in article

• 11

  DiamondA.VorleyT.RobertsJ.JonesS. (2012). Behavioural Approaches to Understanding Student Choice. York: Higher Education Academy.

  • Google Scholar
  • View reference in article

• 12

  DouglassJ.ThomsonG.ZhaoC. M. (2012). The learning outcomes race: the value of self-reported gains in large research universities. High. Educ.63 (3), 317–335. 10.1007/s10734-011-9496-x

  • CrossRef
  • Google Scholar
  • View reference in article

• 13

  EvansC.Kandiko HowsonC.ForsytheA. (2018). Making sense of learning gain in higher education. High Educ Pedagog3 (1), 1–45. 10.1080/23752696.2018.1508360

  • CrossRef
  • Google Scholar
  • View reference in article

• 14

  GibbsG. (1988). Learning by Doing: A Guide to Teaching and Learning Methods. London: Further Education Unit.

  • Google Scholar
  • View reference in article

• 15

  GothwalV. K.WrightT. A.LamoureuxE. L.PesudovsK. (2009). Rasch analysis of visual function and quality of life questionnaires. Optom. Vis. Sci.86 (10), 1160–1168. 10.1097/OPX.0b013e3181bab10c

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 16

  GrantA.McKimmJ.MurphyF. (2017). Developing Reflective Practice: A Guide for Medical Students, Doctors and Teachers. Hoboken: John Wiley & Sons.

  • Google Scholar
  • View reference in article

• 17

  HagelP.CarrR.DevlinM. (2012). Conceptualising and measuring student engagement through the australasian survey of student engagement (AUSSE): a critique. Assess. Eval. High. Educ.37 (4), 475–486. 10.1080/02602938.2010.545870

  • CrossRef
  • Google Scholar
  • View reference in article

• 18

  HirudayarajM. (2011). First-generation students in higher education: issues of employability in a knowledge-based economy. Online J. Workforce Educ. Dev.5 (3), 2.

  • Google Scholar

• 19

  Jones-DevittS.PickeringN.AustenL.DonnellyA.AdesolaJ.WestonA. (2019). Evaluation of the National Mixed Methods Learning Gain Project (NMMLGP) and Student Perceptions of Learning Gain. Office for Students. Available online at:https://www.officeforstudents.org.uk/media/6a83a1c6-3933-4d9b-8b95-43f7bfe142e4/final-evaluation-of-nmmlgp.pdf(Accessed January 2024)

  • Google Scholar
  • View reference in article

• 20

  Kandiko HowsonC (2022). Is Educational Gain the “dark Matter” of Student Outcomes?Wonkhe. Available online at:https://wonkhe.com/blogs/is-educational-gain-the-dark-matter-of-student-outcomes/(Accessed September 2023)

  • Google Scholar
  • View reference in article

• 21

  Karnieli-MillerO. (2020). Reflective practice in the teaching of communication skills. Patient. Educ. Couns.103 (10), 2166–2172. 10.1016/j.pec.2020.06.021

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 22

  KrugerJ.DunningD. (1999). Unskilled and unaware of it: how difficulties in recognising one’s own incompetence lead to inflated self-assessments. J. Pers. Soc. Psychol.77 (6), 1121. 10.1037/0022-3514.77.6.1121

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23

  LathamK.BaranianM.TimmisM. A.FisherA.PardhanS. (2017). Relative difficulties of daily living tasks with retinitis pigmentosa. Optom. Vis. Sci.94 (3), 317–328. 10.1097/OPX.0000000000001046

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 24

  LathamK.BaranianM.TimmisM.PardhanS. (2015). Emotional health of people with visual impairment caused by retinitis pigmentosa. PLoS One10 (12), e0145866. 10.1371/journal.pone.0145866

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 25

  Lin-StephensS.LeonL. C.ManuguerraM.BarkatsasT.RussellL. (2024). Exploring factors contributing to graduate outcomes: using career registration methodology (CRM) to track students’ employability activities, career readiness, and graduate outcomes. Educ. Sci.14 (12), 1294. 10.3390/educsci14121294

  • CrossRef
  • Google Scholar
  • View reference in article

• 26

  LinacreJ. M. (1999). Investigating rating scale category utility. J. Outcome. Meas.3, 103–122.

  • Pubmed Abstract
  • Google Scholar
  • View reference in article

• 27

  LinacreJM. (2008). WINSTEPS Rasch Measurement Computer Program. Winsteps.com.

  • Google Scholar
  • View reference in article

• 28

  LoweT. (2023). Embedding employability into the curriculum: five recommendations to improve widening participation students’ graduate employability. J. Learn. Dev. High Educ.26, 1–11. 10.47408/jldhe.vi26.925

  • CrossRef
  • Google Scholar
  • View reference in article

• 29

  MarellaM.GothwalV. K.PesudovsK. (2009). Validation of the visual disability questionnaire (VDQ) in India. Optom. Vis. Sci.86 (7), E826–E835. 10.1097/OPX.0B013E3181AE1B3F

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 30

  McGrathC. H.GuerinB.HarteE.FrearsonM.ManvilleC. (2015). Learning Gain in Higher Education. Santa Monica. CA: RAND Corporation.

  • Google Scholar
  • View reference in article

• 31

  MohamedM.RashidR. A.AlqaryoutiM. H. (2022). Conceptualizing the complexity of reflective practice in education. Front. Psychol.13, 1008234. 10.3389/fpsyg.2022.1008234

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 32

  MoonJ. (2000). Reflection in Learning and Professional Development: Theory and Practice. London: Kogan Page.

  • Google Scholar
  • View reference in article

• 33

  NeuendorfK. A. (2017). The Content Analysis Guidebook. 2nd ed. Thousand Oaks, CA: Sage.

  • Google Scholar
  • View reference in article

• 34

  NghiaT. L. H.BuiB. C.SinghJ. K. N.LuV. N. (2022). “From employability to employment outcomes and career advancement: a literature review,” Graduate Employability Across Contexts, eds. T. L. H. Nghia, B. C. Bui, J. K. N. Singh, and V. N. Lu (Singapore: Springer). 10.1007/978-981-19-3959-4_2

  • CrossRef
  • Google Scholar
  • View reference in article

• 35

  OfS (2022). Office for Students. Learning gain. Available online at:https://www.officeforstudents.org.uk/advice-and-guidance/the-tef/learning-gain/(Accessed May 2024)

  • Google Scholar

• 36

  OfS (2025). Office for Students. Student characteristics data. Available online at:https://www.officeforstudents.org.uk/data-and-analysis/student-characteristics-data/(Accessed February 2025)

  • Google Scholar
  • View reference in article

• 37

  OfS (2019). Office for Students. Higher Education Learning Gain Analysis (HELGA): Can Administrative Data be Used to Measure Learning Gain?Available online at:https://www.officeforstudents.org.uk/media/614623ac-b603-4922-ae2d422afa354590/helga-report.pdf(Accessed January 2024)

  • Google Scholar
  • View reference in article

• 38

  PesudovsK.BurrJ. M.HarleyC.ElliottD. B. (2007). The development, assessment, and selection of questionnaires. Optom. Vis. Sci.84 (8), 663–674. 10.1097/OPX.0b013e318141fe75

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 39

  RandlesR. J.CotgraveA. (2017). Measuring student learning gain: a review of transatlantic measurements of assessments in higher education. Innov. Pract.11 (1), 50–59. 10.24377/LJMU.iip.vol11iss1article155

  • CrossRef
  • Google Scholar
  • View reference in article

• 40

  ReayD.DaviesJ.DavidM.BallS. J. (2001). Choices of degree or degrees of choice? Class, ‘race’ and the higher education choice process. Sociology35 (4), 855–874. 10.1177/0038038501035004004

  • CrossRef
  • Google Scholar
  • View reference in article

• 41

  RookL.DeanB. A.TofaM.EllmersG.Villeneuve-SmithS.KennedyM.et al (2026). Student feedback for integrating reflection into the higher education curriculum; a cross disciplinary study. High. Educ. Res. Dev.45 (3), 766–780. 10.1080/07294360.2025.2548274

  • CrossRef
  • Google Scholar
  • View reference in article

• 42

  RoselliniA. J.BrownT. A. (2021). Developing and validating clinical questionnaires. Annu. Rev. Clin. Psychol.17 (1), 55–81. 10.1146/annurev-clinpsy-081219-115343

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 43

  RutherfordS.PritchardC.FrancisN. (2025). Assessment IS learning: developing a student-centred approach for assessment in higher education. FEBS. Open. Bio.15 (1), 21–34. 10.1002/2211-5463.13921

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 44

  SeilerL. H. (1973). The 22-item scale used in field studies of mental illness: a question of method, a question of substance, and a question of theory. J. Health. Soc. Behav., 14 (3), 252–264. 10.2307/2137117

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 45

  SpeightL.CrawfordK.HaddelseyS. (2018). Towards measures of longitudinal learning gain in UK higher education: the challenge of meaningful engagement. High Educ Pedagog3 (1), 196–218. 10.1080/23752696.2018.1476827

  • CrossRef
  • Google Scholar
  • View reference in article

• 46

  StefanaA.DamianiS.GranziolU.ProvenzaniU.SolmiM.YoungstromE. A.et al (2025). Psychological, psychiatric, and behavioral sciences measurement scales: best practice guidelines for their development and validation. Front. Psychol.15, 1494261. 10.3389/fpsyg.2024.1494261

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 47

  SuF.WoodM. (2023). Relational pedagogy in higher education: what might it look like in practice and how do we develop it?Int J Acad Dev28, 230–233. 10.1080/1360144X.2023.2164859

  • CrossRef
  • Google Scholar

• 48

  TabrettD. R.LathamK. (2011). Factors influencing self-reported vision-related activity limitation in the visually impaired. Invest. Ophthalmol. Vis. Sci.52 (8), 5293–5302. 10.1167/iovs.10-7055

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 49

  ThomasL.QuinnJ. (2007). First Generation Entry into Higher Education: An International Study. Berkshire: Open University Press.

  • Google Scholar
  • View reference in article

• 50

  TomlinsonM. (2017). Forms of graduate capital and their relationship to graduate employability. Educ. Train.59 (4), 338–352. 10.1108/ET-05-2016-0090

  • CrossRef
  • Google Scholar
  • View reference in article

• 51

  TovarE.SimonM. A. (2010). Factorial structure and invariance analysis of the sense of belonging scales. Meas. Eval. Couns. Dev.43 (3), 199–217. 10.1177/0748175610384811

  • CrossRef
  • Google Scholar
  • View reference in article

• 52

  VermuntJ. D.IlieS.VignolesA. (2018). Building the foundations for measuring learning gain in higher education: a conceptual framework and measurement instrument. High Educ. Pedagog.3 (1), 266–301. 10.1080/23752696.2018.1484672

  • CrossRef
  • Google Scholar
  • View reference in article

• 53

  Wonkhe (2025). Reclaiming the Narrative of Educational Excellence Despite the Decline of Educational Gain. Wonkhe. Available online at:https://wonkhe.com/blogs/reclaiming-the-narrative-of-educational-excellence-despite-the-decline-of-educational-gain/(Accessed December 2025)

  • Google Scholar
  • View reference in article

• 54

  WrightB. D.LinacreJ. M. (1994). Reasonable mean-square fit values. Rasch Meas Trans8, 37.

  • Google Scholar
  • View reference in article