Edited by: Anastasiya A. Lipnevich, The City University of New York, United States
Reviewed by: Wei Shin Leong, Nanyang Technological University, Singapore; Jana Groß Ophoff, University of Tübingen, Germany
This article was submitted to Assessment, Testing and Applied Measurement, a section of the journal Frontiers in Education
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Research attention has shifted from feedback delivery mechanisms to supporting learners to receive feedback well (Winstone et al.,
Providing feedback that assesses learner performance relative to goals or objectives is proposed as a necessary process in optimizing performance. Universities have spent significant resources in their attempts to improve student satisfaction in relation to assessment and feedback. For example, increasing learner assessment literacy through the use of rubrics is thought to make available the tacit knowledge that academics often carry around in their heads. However, most of these interventions have had very little impact on student satisfaction with assessment and feedback and National Student Survey scores on this area remain relatively stable (and low) across the HE sector (Evans et al.,
Providing feedback may not be sufficient. Developing learner's skills in integrating feedback by evaluating and making judgments about the courses of action necessary for progression appears to be a necessary additional step (Nicol and Macfarlane-Dick,
In its broadest sense, feedback is reported to hold two fundamental roles, it acts as a mediator of “in-flight” performance or as a moderator of subsequent performance, by upregulating or downregulating subsequent goals (Ashford and De Stobbeleir,
Given the importance of feedback, the mechanisms for delivering effective feedback to HE learners has been the focus of research for some time (Nicol and Macfarlane-Dick,
Despite the research focus on delivery mechanisms in HE, and learners understanding which of these mechanisms serve them best, fostering greater awareness and receptivity to feedback remains problematic (Winstone et al.,
Fostering an environment that encourages positive dialogue is a pillar of good feedback practice (Nicol and Macfarlane-Dick,
Within the HE context, several barriers are reported. Barriers relate to lack of awareness of the feedback process; poor knowledge of associated strategies and opportunities for development; lacking agency and associated self-regulatory strategies; and low engagement and volition with addressing the issues raised in feedback. Managing these barriers is suggested to be an important step in a move toward encouraging learner receptivity to feedback (Winstone et al.,
A receiving focus in relation to feedback is supported in Winstone et al. (
However derived, engaging students in the development of adaptive knowledge, skills and attitudes that underpins hard won gains in learning is crucial, in particular if learners are to develop the ability to manage themselves during the courses of their studies and into employment (Forsythe and Jellicoe,
Within the occupational domain, Boudrias et al. (
Despite the noted limitations in Boudrias et al. (
Drawing on these suggestions, the first aim of the current study is to explore the factor structure of a modified feedback integration measure that has been utilized in occupational domains (Boudrias et al.,
Addressing the issues above, the first research aim was to determine a data driven approach to understanding feedback integration in tertiary academic audience based on a modification of a measure provided by Boudrias et al. (
Two pools of participants were recruited to examine cognitive and behavioral factors associated with integration of feedback in learning. A first convenience sample of 353 first and second-year psychology undergraduate students were recruited to participate in the current study. Two sources of opportunity recruitment were used. In the first of these a convenience sample of 163 second year undergraduate participants were recruited. The first recruitment opportunity was time-limited and did not generate a large enough sample for exploratory analysis. As a result, a further sample of first year students (
A second convenience sample of 402 second year students registered on a half year psychology module were requested to participate as part of a wider data collection process. Forty-six responses were excluded in the second sample where participants failed to respond to one or more of the items. Following exclusions, participants were
The current study employed a cross-sectional design and structural equation modeling (SEM) to explore two pools of responses to determine the factor structure of the FLS. A data driven approach using EFA was used to explore the first sample. The second sample was used in a LVSEM approach to simultaneously confirm the psychometric properties of the scale and to test a linear hypothetical path through the factors derived in EFA.
A 34-item measure examined perspectives supporting integration of feedback in learning. Items were derived from an existing measure of feedback integration typically used in occupational research (Boudrias et al.,
Participants completed the survey online via a hyperlink directing participants to the Qualtrics (
All analyses were conducted in an R environment (R Core Team,
FLS items were assessed for normality. In relation to the first sample, employed in EFA, sampling adequacy was assessed using the Kaiser-Myer-Olkin (KMO) statistic. Bartlett's test of sphericity was used to assess whether inter-item correlations were sufficiently large to continue with EFA. For the second sample, data were examined for multivariate normality. As Mardia's Kurtosis test was violated (Crede and Harms,
Model fit was assessed using the Normed
The
Internal consistency of the FLS was assessed using the Cronbach's alpha. A lower bound estimate of α = 0.70 was considered acceptable (Nunnally and Bernstein,
The
With the exception of one variable across both samples, skewness and kurtosis values were between −2 and 2. Whilst there is lack of clarity in the literature, skewness and kurtosis values were below “rules of thumb” indicated by Kline (
The KMO statistic for the model was above the 0.50 threshold (KMO = 0.90) and Bartlett's test of sphericity was significant (
Characteristics of participants in each group.
Females:Males | 276:45 | 307:49 |
Age (years):Mean (±SD) | 19.54 (±2.98) | 20.31 (±3.64) |
Three iterations were sufficient to derive simple factor structure. In the first iteration, EFA with an oblique (oblimin) using a maximum likelihood approach, visual inspection of the scree plot and the 0.70 eigenvalue criteria revealed a five-factor solution. However, an unclear factor structure was indicated. Six items reported factor loadings less than the suggested 0.4 criteria. Following the removal of these items, and using the same cut-off criteria, a second iteration revealed a five-factor solution. In this iteration however, simple structure was not achieved, with one further item failing to load on to the five derived factors. Following the removal of this single item, a third and final iteration of the same EFA procedure was undertaken. A five-factor structure converged during the final iteration with 27 individual items retained. Eigenvalues for the respective factors were 7.95, 2.03, 1.12, 0.99, and 0.71. Factor one, made up of eleven items, referencing credible source challenge, for example “
Factors, items, and factor loadings for EFA and LVSEM analyses.
1. Credible Source Challenge (11) | CR7 | 0.772 | 0.732 |
CR2 | 0.765 | 0.697 | |
CR4 | 0.763 | 0.662 | |
CR8 | 0.726 | 0.651 | |
CR5 | 0.652 | 0.509 | |
CR3 | 0.640 | 0.725 | |
CR9 | 0.620 | 0.544 | |
CR6 | 0.584 | 0.369 | |
CI3 | 0.497 | 0.481 | |
CR1 | 0.462 | 0.602 | |
CI2 | 0.440 | 0.521 | |
2. Behavioral Changes and Developmental Actions (5) | DC3 | 0.799 | 0.761 |
DC2 | 0.714 | 0.642 | |
DC1 | 0.669 | 0.761 | |
BC4 | 0.593 | 0.527 | |
BC3 | 0.527 | 0.564 | |
3. Feedback Acceptance (3) | AC2 | 0.916 | 0.927 |
AC1 | 0.889 | 0.807 | |
AC3 | 0.725 | 0.705 | |
4. Motivational Intention (4) | MI3 | 0.861 | 0.646 |
MI2 | 0.859 | 0.724 | |
MI1 | 0.575 | 0.804 | |
BC1 | 0.411 | 0.652 | |
5. Awareness from Feedback (4) | AW3 | 0.730 | 0.664 |
AW2 | 0.644 | 0.549 | |
AW1 | 0.585 | 0.626 | |
AW4 | 0.493 | 0.727 |
Factor model of FLS with standardized factor loadings represented on unidirectional arrows. Factors in this figure are represented by the following key. ML1, Credible Source Challenge; ML2, Feedback Acceptance; ML3, Behavioral Changes and Developmental Actions; ML4, Motivational Intentions; and ML5, Awareness from Feedback. See
Subscale scores for both samples in relation to the FLS are reported in
Descriptive statistics (where values are means and standard deviation ±) and internal consistency (Cronbach's alpha) for the FLS.
1. Credible Source Challenge | 4.2 (±0.67) | 0.90 | 4.5 (±0.57) | 0.86 |
2. Behavioral Changes and Developmental Actions | 3.6 (±0.91) | 0.81 | 3.7 (±0.82) | 0.81 |
3. Feedback Acceptance | 3.9 (±0.87) | 0.89 | 4.0 (±0.82) | 0.85 |
4. Motivational Intention | 4.6 (±0.67) | 0.83 | 4.8 (±0.66) | 0.82 |
5. Awareness from Feedback | 4.6 (±0.65) | 0.75 | 4.5 (±0.73) | 0.78 |
Inter-item correlations for EFA.
Inter-item correlations for LVSEM.
A measurement model was specified that simultaneously confirmed the latent factor structure and examined a unidirectional path through identified factors. In the model specified, from twenty-seven items identified in EFA, eleven were free to load on to the latent factor Credible Source Challenge; five items were free to load on to the latent factor Behavior and Development Change; four items each were free to load on Feedback Acceptance and Motivational Intention, respectively; and the remaining three items were free to load on latent factor Awareness from Feedback. Following consideration of the factor structure, four tentative paths were specified at the latent variable level. These were that feedback acceptance predicts credible source challenge. In turn credible source challenge predicts greater awareness from feedback. Subsequently, it was predicted that awareness from feedback will predict motivational intentions. Our final prediction was that motivational intentions will predict the endorsement of behavioral changes and developmental actions resulting from feedback.
The initial iteration did not achieve acceptable fit to the data without modifications. Following inspection of modification indices, a number of items were allowed to covary due to conceptual congruity, see
Factors and related items identified in data driven analysis were confirmed in the LVSEM model.
Direct paths between latent factors specified in the measurement model.
1. | Feedback Acceptance | → | Credible Source Challenge | 0.45 |
0.21 |
2. | Credible Source Challenge | → | Awareness from Feedback | 0.41 |
0.17 |
3. | Awareness from Feedback | → | Motivational Intention | 0.67 |
0.45 |
4. | Motivational Intention | → | Behavioral Changes and Developmental Actions | 0.47 |
0.22 |
Latent variable structural equation model of FLS with standardized factor loadings (reported on unidirectional arrows), error terms (circled values), and covariances (reported on bidirectional arrows). Factors in this figure are represented by the following key: ACC, Feedback Acceptance; CRD, Credible Source Challenge; AWA, Awareness from Feedback; MI, Motivational Intentions; and BDC, Behavioral Changes and Developmental Actions. See
All the paths specified in the model were significant (
Following modifications, the final model achieved an acceptable to good fit to the data (see
Three alternate models were explored. In the first such model [A1], the first specified path predicted that credible source challenge led to awareness from feedback. In turn, awareness from feedback was allowed to predict acceptance of feedback. A subsequent path was specified from acceptance to motivational intentions. In the second model [A2], credible source challenge was allowed to predict both awareness from feedback and acceptance of feedback. Both factors in turn predicted motivational intentions. A third model [A3] took a similar linear approach to the hypothesized model; however, in this approach credible source challenge was allowed to predict acceptance of feedback, transposing the order in the hypothesized model. Next, acceptance led to awareness, and then in turn to motivational intentions. In each of alternative models, as with the hypothesized model, behavioral change and developmental action predicted by motivational intentions was specified as the final path.
Remaining constant in all models, modification indices suggested the specification of four covariances between item error terms. Two additional modifications were suggested to the first alternate model. The first of these suggested a path between acceptance of feedback to credible source challenge. The second modification suggested a path from awareness from feedback to motivational intentions. These paths reintroduced the suggested directional paths from the hypothesized model. For comparison purposes, fit measures for each model are presented in
Comparison of robust fit measures for hypothesized and alternative models.
Hyp | 1.59 | 22,613.56 | 22,853.80 | 0.041 (0.035–0.046) | 0.934 | 0.927 | 0.066 |
A1 | 1.59 | 22,617.21 | 22,865.21 | 0.041 (0.035–0.046) | 0.934 | 0.926 | 0.066 |
A2 | 1.59 | 22,615.41 | 22,859.53 | 0.041 (0.035–0.047) | 0.934 | 0.927 | 0.066 |
A3 | 1.65 | 22,641.92 | 22,882.17 | 0.049 (0.041–0.056) | 0.927 | 0.919 | 0.091 |
“
The current investigation examined the factor structure of a modified measure of feedback integration. Analyses explored and confirmed five latent factors associated with feedback integration by HE learners. Alongside the confirmatory analysis, a hypothesis driven model was reported at least marginally superior fit to alternative models explored. This indicated a directional path through each of the derived five factors. Acceptance of feedback led to credible source challenge, in turn predicting awareness from feedback. Greater awareness subsequently predicted motivational intentions. Finally, motives predicted action. Both models achieved at least acceptable fit to the data, and the paths between factors represented unique proportions of the variance in the model.
The current study refined and validated a measure, drawn from the occupational domain, examining the nature of feedback integration in undergraduate learners. A first data driven derived a feedback in learning scale with a five factor structure. The first factor, credible source challenge, addresses the credibility of the source providing feedback and the challenge they provide. Behavioral change and developmental actions, the second factor, represents the learner's desire to take action following feedback. Next, acceptance from feedback considers whether the feedback received is acknowledged by the learner. The penultimate factor represents the motivational intentions in response to feedback. The final fifth factor relates to awareness from feedback, specifically whether learners were more aware of their strengths and weaknesses following feedback. Except where noted, findings largely support factors derived by Boudrias et al. (
A latent variable structural equation modeling approach, was used to address a second research aim. First, the latent factor structure identified in the first exploratory investigation was confirmed. Conjointly, four hypothesized paths were proposed between each of the five latent factors following a consideration of theory and the model indicated by Boudrias et al. (
Although one model is reported in the current study, alternate explanatory models were examined following good practice (Crede and Harms,
Findings from the current study speak to five factors associated with feedback integration in tertiary learning. These findings are particularly noteworthy as they highlight the importance of raising learner awareness of strengths and challenges as a central role for intervention. Awareness from feedback is seen to relate directly to learners' motivational intentions, which accounted for the greatest proportion of the variance in the path model. Learner motives led to behavioral changes and developmental actions endorsed by learners following feedback. Further, these findings suggest that learners may seek out additional feedback and action plans from credible sources of information. This understanding may come from a credible source, such as a tutor or a trusted peer. The relationships seen in the current study appear to address the three considerations highlighted by Hattie and Timperley (
Supporting recent theoretical models, factors including the increasing self-awareness, goal setting, and engagement and motivation are also established as central forces in recent models of feedback recipience (Winstone et al.,
Using the measure developed and validated here for diagnosis and intervention will prove useful as a cost effective route to identifying and addressing maladaptive behaviors. For example, the FLS is a tool that facilitates identification of learners with lower levels of acceptance, trust, awareness, motivational intent, and desire to act in response to feedback. Following identification, addressing suboptimal feedback behaviors using appropriate pedagogies appears to be an effective mechanism to assist learners in developing the evaluative judgments that are necessary to optimize learning (Winstone et al.,
Despite providing a parsimonious model of feedback integration, the current study has its limitations. The model of feedback integration reported here represents one model of feedback integration, it is possible that any number of other hypothetical models may account for the data just as well, and possibly better. Although, this approach aligns well with theories of self-regulated learning (Zimmerman,
In summary, the current investigation indicates that the FLS represents a valid and reliable measure of feedback integration behaviors in undergraduate learners. Three aligned practical implications of the FLS are suggested. Firstly, the measure may assist in identifying active components associated with feedback integration in undergraduate learners. Using the FLS for identification of behaviors and change over time, as a meaningful mechanism for capturing gains in learning provides a useful tool to promote further research. In addition, using the FLS as part of interventions and pedagogies to raise learner self-awareness may support learners to take the steps necessary to evaluate and make necessary changes to optimize learning. Future research is necessary to validate the FLS as reliable tool in other tertiary settings to determine if the measure has utility beyond the current setting and domain of learning. However, these ideas are consistent with theory (Zimmerman,
All datasets generated for this study are included in the manuscript and/or the
This study was carried out in accordance with the recommendations of The British Psychological Society, with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Institute of Psychology, Health and Society Research Ethics Committee of The University of Liverpool.
MJ designed the study, analyzed the data, and led the writing of the article. AF supervised, helped in designing the study, and aided in writing the article.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The Supplementary Material for this article can be found online at:
Summary of modified measure and scoring methodology.
MV1 | A large proportion of the feedback I received last year was negative | * |
MV2 | A major part of the feedback I received targeted my weaknesses | * |
MV3 | A large part of the feedback I received was positive | * |
FV1 | I believe the assessments I have taken part in give the opportunity to measure elements clearly related to my education | ° |
The following examples are good tools to assess my potential: | ° | |
FV2 | - Examinations | ° |
FV3 | - Essays | ° |
FV4 | - Blogs | ° |
FV5 | - Discussion/Debates | ° |
FV6 | - Presentations | ° |
I see the connection between tests and assessment methods used and what is required in my future: | ° | |
FV7 | - Examinations | ° |
FV8 | - Essays | ° |
FV9 | - Blogs | ° |
FV10 | - Discussion/Debates | ° |
FV11 | - Presentations | ° |
I believe this assessment process can correctly identify individual characteristics related to my performance on: | ° | |
FV12 | - Examinations | ° |
FV13 | - Essays | ° |
FV14 | - Blogs | ° |
FV15 | - Discussion/Debates | ° |
FV16 | - Presentations | ° |
AC1 | I believe the feedback I received adequately reflects the person I am | Acceptance |
AC2 | I believe the feedback I received depicts me accurately | Acceptance |
AC3 | I recognize myself in the description my assessor has made of me | Acceptance |
The staff who assessed me are outstanding in their: | ||
CR1 | - Ability to assess my competencies | Credible |
CR2 | - Ability to make me feel comfortable | Credible |
CR3 | - Expertise in assessing people's competencies and potential | Credible |
CR4 | - Quality of listening | Credible |
CR5 | - Mastery of assessment tests and tools | Credible |
CR6 | - Understanding of the context for which I am assessed | Credible |
CR7 | - Capacity to gain my confidence | Credible |
CR8 | - Tact and Diplomacy | Credible |
CR9 | - Ability to rapidly size people and their personality | Credible |
The assessor was outstanding in their: | ||
CI1 | - Ability to draw me out of my comfort zone | * |
CI2 | - Ability to destabilize me in a positive manner | Credible |
CI3 | - Capacity to confront my way of perceiving things | Credible |
CI4 | - Capacity to question how I perceive myself | * |
AW1 | I have a better idea of the type of work environment in which I perform well | Awareness |
AW2 | I understand better why some things or people make me react | Awareness |
AW3 | I know myself better | Awareness |
AW4 | I am more aware of the strengths that I can draw on from my studies | Awareness |
MI1 | I am motivated to engage in developmental activities in line with the feedback I received | Motivational |
MI2 | I am determined to work on the development areas identified | Motivational |
MI3 | I am motivated to develop myself in the direction of the feedback I received | Motivational |
Following feedback I: | ||
BC1 | Have changed my less-efficient behaviors described in the feedback I have received | Motivational |
BC2 | Make more use of my strengths identified during the feedback session when I encounter a problem in my studies | * |
BC3 | have sought more feedback from others to develop competencies discussed during the feedback | BD Change |
BC4 | asked others for suggestions on how I could improve competencies described in the feedback | BD Change |
BC5 | have changed my study behavior in a way consistent with the feedback I received | * |
DC1 | have voluntarily participated in developmental activities in line with the feedback I received | BD Change |
DC2 | have asked my tutor for a development plan in line with the feedback I received | BD Change |
DC3 | have searched for developmental activities in line with competencies described during the feedback | BD Change |