Improving competence for practice: a realist evaluation of co-produced master-level digital sexual health promotion courses for health and welfare professionals

Introduction: Sparse education on sexual health promotion for health and welfare professionals can lead to unequal sexual health care provision, with related needs and rights being insufficiently met. To improve knowledge for health and welfare professionals, co-production and shared learning are essential, as students’ learning, and professional development are facilitated by effective partnerships between universities and workplaces.

Methods: Using a theoretical framework based on the concepts co-production, work-integrated learning, digital teaching and digital didactics, an initial programme theory was created. Thereafter, a realist evaluation was made of outcomes from five different digital master-level courses, all covering various aspects of sexual health promotion. Four data sources were used: 1) midway course evaluation from 32 students analysed with reflective thematic analysis, 2) focus group interviews with four teachers, analysed with reflective thematic analysis, 3) Students’ Attitudes to Sexual Health Extended survey pre- and post-test with 17 students, analysed with comparative statistics, and 4) learning analytics based on data from the digital tool FeedbackFruits, where interactions were analysed using a digital whiteboard).

Results: The analysis resulted in a refined programme theory indicating that students found courses relevant for practice, became more comfortable talking about sexual health in practice and applied their learning in practice.

Discussion: The study supports the value of co-production in developing and delivering digital master-level courses in sexual health promotion for professionals, and the added value of work-integrated learning in this setting and highlights the need for digital literacy among both students and teachers. Confirmation or contradiction of the findings should be tested in this and other settings, and preferably with a larger sample.

Introduction

Both global and Scandinavian examinations of educational programs for health and welfare professionals show that important topics related to sexual and reproductive health and rights (SRHR) are often lacking, this can lead to unequal sexual health care, with related needs and rights being insufficiently met (Areskoug-Josefsson et al., 2019; Endler et al., 2022; Solberg and Areskoug Josefsson, 2024). Related to this shortage, Norwegian authorities have acknowledged the need for improved education of health and welfare professionals (Ministry of Health and Care Services, 2016). This paper concerns co-produced continued higher education intended to change professional practice related to SRHR in general, and sexual health promotion in particular. To improve knowledge for professionals, co-production is essential, as students’ learning and professional development are facilitated by effective partnerships between universities and workplaces (Kenny et al., 2023). Co-production in higher education is complex but provides unique learning and development opportunities (Bell et al., 2021; Bilous et al., 2018). Co-production involves learning through continuous dialogue and reflection, guided by sharing of power, building and maintaining relationships, inclusivity, reciprocity and respecting and valuing the knowledge of all those working together (National Institute for Health and Care Research, 2021). As there are many definitions of co-production which can differ depending on context, a shift toward values and principles of co-production is recommended (Masterson et al., 2022, 2024). This paper presents the evaluation of a co-produced higher education initiative to increase professionals’ abilities to promote sexual health through master-level courses aimed at health and welfare professionals.

In higher education, courses are often evaluated through traditional criteria, but there has been a change toward the promotion of a learning culture, including evaluating learning impact (Edström, 2008; Schellekens et al., 2023; Skivington et al., 2021). This paper evaluates potential impact of five digital, co-produced, master-level sexual health promotion courses in Norway and provides knowledge for development, implementation, and evaluation of co-produced digital courses in continued higher education. Outcomes of large-scale interventions such as this can be difficult to evaluate, especially when using singular or traditional evaluation criteria (Edström, 2008). There is a risk of measurement certainty being the only focus when complex interventions are evaluated, instead of focusing what is of most importance to stakeholders and decision-makers (Skivington et al., 2021). This calls for a broader perspective when evaluating co-produced courses in higher education. This paper aims to contribute to understanding how evaluation of higher education digital courses intended to change professional practice related to sexual health promotion can be conducted, using student and teacher experiences and learning analytics derived from digital tools.

The Sexual Health project co-produced seven digital courses on master-level, with different topics related to sexual health during 2020–2022 (Areskoug Josefsson and Lunde, 2024). All courses were aimed at students with a previous bachelor’s degree in health care, social work or pedagogy who were currently working and the courses thus offered interprofessional continued education within sexual health promotion. Development of courses consisted of co-production with different stakeholders: people with lived experience, NGOs, professionals, and university employees. Pilot-testing of the courses was performed, providing information used to improve their quality (Areskoug Josefsson and Lunde, 2024).

As large-scale co-production of digital courses in higher education is complex, novel and resource-demanding (Areskoug Josefsson and Lunde, 2024; Folkman et al., 2023), outcome evaluation is essential. Traditional course evaluations (student feedback and student completion rates) can provide important information for improvement (Flodén, 2017). However, we regarded such methods insufficient to evaluate the intended outcomes of the Sexual Health project, namely: providing increased sexual health promotion competence and ensuring relevant competence for students leading to changes in practice. Digital teaching can itself have limitations in education of healthcare professionals (Frenk et al., 2022) and evaluating courses in sexual health promotion adds yet a layer of complexity, as professionals in various contexts in the Nordic countries consider the topic important but experience barriers in doing so (Arvidsson et al., 2024; Björkas et al., 2024; Lindskog et al., 2024; Svae et al., 2023). Therefore, realist evaluation, RE, was chosen to explore both visible and invisible mechanisms involved in complex and multifaceted outcomes (Pawson, 2013).

RE is a theory-driven, practice-oriented method to evaluate programs and interventions by closely examining how mechanisms and contexts interplay to produce outcomes (Pawson and Tilley, 1997). REs assume that positive program outcomes only occur when the program activates mechanisms in certain contexts and aims to understand the underlying mechanisms (M) which produce change, the contextual factors (C) necessary to activate these mechanisms, and how the combination of context and mechanisms produces outcomes (O) (Pawson and Tilley, 1997). Rather than assuming a binary ‘on–off’ activation which is triggered, it has been suggested that activation of mechanisms operates on a continuum, and a refined Contextual factors, Mechanisms, and Outcomes (CMO) formula has been presented (Dalkin et al., 2015). This understanding was suitable for our exploration. We first attempted to identify what in the courses that might work, for whom, in what circumstances and why (Pawson, 2013); then conducted data collection, followed by an analysis to test and revise the initial hypothesis and program theory, and ended with a refined program theory. The cycle started with developing a suitable theoretical framework. The chosen theoretical framework for understanding potential impact of the developed courses consists of four concepts: co-production, work-integrated learning, digital teaching, and digital didactics, all hypothesized as relevant for learning outcomes.

In addition to the central values and principles provided by National Institute for Health and Care Research (2021), knowledge co-production can be seen as context-based, pluralistic, goal-oriented, and interactive (Norström et al., 2020), which aligns with the development of courses within the Sexual Health project (see a more detailed project description below). Work-integrated learning is a transdisciplinary approach where workplaces and universities collaborate to integrate theoretical knowledge with practice, to support students in the transition of using new knowledge in their working life (Olsson et al., 2021; Sunnemark et al., 2023), as in the Sexual Health project. Work-integrated learning can support interprofessional learning for healthcare professionals, and ensure competent healthcare staff (Frenk et al., 2022). Methods of implementing work-integrated learning differ and include high to low levels of student engagement (Campbell et al., 2021).

All courses developed in the Sexual Health project were provided on-line. Digital teaching increased drastically during the COVID-19 pandemic. However, knowledge gaps remain regarding the use of digital tools and their intended outcomes (Frenk et al., 2022). Digital teaching can increase accessibility to higher education but also lead to fewer social interactions between students, technical problems, and lack of equal opportunities due to low digital literacy (Erlam et al., 2021). By using the opportunities of digitalization, exploring learning spaces (e.g., physical, virtual and digital learning spaces), and different pedagogical practices, students can be empowered, as learners and citizens (Ossian Nilsson, 2019). The courses developed in the Sexual Health project included different digital didactic tools. To improve digital didactics in higher education and ensure equal learning opportunities, digital literacy among students and teachers should be addressed (Røe et al., 2022). A key challenge is to ‘create an interactive context, a learning community, with appropriate levels of social presence, providing higher-order learning’ (Keller and Hrastinski, 2009, p. 104). Digital didactics includes both the use of digital tools and an analysis of how these tools can facilitate learning (Areskoug Josefsson et al., 2022). This learning analytics provides opportunities to assess, predict and evaluate learning outcomes (Areskoug Josefsson et al., 2023; Seufert et al., 2019). Digital tools can provide easy access to learning analytics; however, there is a risk of using this easily accessible output in superficial data presentations that can lead to false conclusions (Seufert et al., 2019).

Materials and methods

Design

RE was used to evaluate outcomes in five courses in the Sexual Health project. All courses aimed to increase professionals’ competence within sexual health promotion and thereby improve practice. This evaluation explores what worked, how and why and examines contextual factors that affected mechanisms related to the implementation of the courses. It also aims to identify benefits and barriers experienced by involved students and teachers. From this, improvement areas will be identified. In line with RE, a ‘ground-up’ approach was adopted to explore the sexual health courses, their intended impacts including both implicit and explicit aims of the Sexual Health project, the tools and expertise available and how these aims were achieved through exploration of mechanisms that generated outcomes. The RE cycle proposed by Pawson and Tilley (1997): theory–hypotheses–observations–program specifications and the RAMESES-II reporting guidelines for RE (Wong et al., 2016) have been followed.

The design allowed for an iterative process, with inductive and deductive reasoning and the development of an initial program theory. Monthly research group discussions between authors were held, and data, analysis, and preliminary findings were discussed, and possible underlying and explicit mechanisms explored. During a two-day workshop all hunches and preliminary results were discussed to identify what outcomes, both planned and unintended, that had been reached. We developed a series of initial program theories based on the results from previous pilot tests and follow-up research (Areskoug Josefsson and Lunde, 2024; Folkman et al., 2023), and the chosen theoretical framework (i.e., co-production, work-integrated learning, digital teaching, and digital didactics). An initial program theory was formulated representing our initial hypotheses, based on the chosen theoretical framework, of what might work, for whom, in what circumstances and why (Figure 1). In the figure, five central contexts are presented (course structure, digital teaching, course content, digital didactics and work-integrated learning). Various mechanisms, divided into reasoning and resources as proposed by Dalkin et al. (2015), are also presented as well as expected outcomes.

Figure 1

Figure 1. Context, Mech (Mechanism) and Outcome configurations for initial program theory.

Evaluation environment–the delivery of digital sexual health promotion courses

The Sexual Health project took place at a large Norwegian University, in a department with previous experience of teaching sexual health promotion courses digitally at master’s level to health and welfare professionals (Areskoug Josefsson and Lunde, 2024). Five of the seven courses (two had not started) were included in this evaluation: (1) Sexual health and intellectual disability, (2) Sexual health as a resource through the lifespan, (3) Sexual health literacy, (4) Sexual anatomy and physiology, and (5) Sexual abuse: prevention, detection and follow-up. The courses were taught by both new and experienced university lecturers, and persons having professional or lived experience were invited as guest lecturers. Four to five two-hour synchronous digital seminars (webinars) were held in each course, where students presented tasks, had opportunities to ask questions, discuss course topics, and listen to short lectures. The digital learning platform Canvas was used to provide learning material (for example: films, quizzes, discussions), course literature, and assignments. The students were encouraged to interact with each other and colleagues at their workplace, about the content of the course. Specific work-integrated learning activities and tasks used in the courses were to test communication models, present new perspectives from research on sexual health promotion to colleagues, and to implement an improvement project. All students had a least a bachelor’s degree, and a majority were currently employed and working. They had various professions (e.g., social workers, midwifes nurses, physiotherapists, police, preschool teachers, teachers, and social educators) and were divided into interprofessional study groups to enhance interprofessional learning. All courses were part-time and provided 10 ECTS credits, except Sexual health and intellectual disabilities which provided 15 ECTS credits.

Sampling and analysis

The qualitative and quantitative data used in this RE consisted of course evaluations with students, a focus group interview with teachers, completion of the Students’ Attitudes toward Addressing Sexual Health Extended (SA-SH-Ext) survey and learning analytics of student interaction from FeedbackFruits (a digital, pedagogical tool for interaction between students and with teachers). Each of the four different data sets had a specific recruitment process and sampling strategy. The data was collected in courses during the first semester of 2023, and there were in total 77 students in the courses.

Sampling and analysis: midway course evaluations

To capture student evaluations of the ongoing course, all students were invited to non-anonymous digital discussions as midway course evaluation in the learning platform Canvas. They could read the discussion thread and provide an unlimited number of comments, and the following seven questions were posed: (i) How do you talk about the subjects in the course with colleagues and friends?, (ii) How do you use the learning resources (digital resources including literature, lectures and interaction with other students) in your own learning in the course?, (iii) How do you use the knowledge gained from this course in your daily work or practice?, (iv) What elements do you find positive in the course?, (v) In what ways has the course been relevant for your daily work or practice?, (vi) In what way were interactive documents (e.g., FeedbackFruits) valuable for interaction with others in the course?, and (vii) What can be improved in the course?

The midway course evaluation was answered by 32 students (response rate 42%, range 17–58% between courses). The data was first analyzed by author AJ and ML, before a collective analysis. The analysis of the discussion threads was inspired by reflexive thematic analysis (Braun and Clarke, 2019), and patterns were sought and extracted from the text. The six phases of the reflexive thematic analysis (familiarization with the data, initial coding, the production of themes, sorting through themes, defining, and refining themes and writing an augmenting presentation) were used, resulting in three main themes with two main themes each (see Table 1 in the Results section).

Table 1

Table 1. Students’ mid-term evaluation.

Sampling and analysis: focus group interview

For the focus group interview, teachers responsible for the courses were invited to a focus group interview. The focus group interview was digital and moderated by author AJ (who had not been involved in the course development), with support from a faculty colleague with interview experience (who had not been involved in the course development). It was conducted at the very end of the semester, in early June 2023, lasted about 1 h, audio-recorded and transcribed verbatim by an external transcriber. A semi-structured interview guide with five main areas was used to address experiences related to digital teaching, digital didactics, and learning: (i) What is it like to lead a digital course? (Pedagogics? Administration? Design?), (ii) What thoughts do you have on digital and interactive resources? (Tools? Prioritizing? Usefulness?), (iii) How do you evaluate your competence and engagement in the course? (Is your competence of value?), (iv) What are your thoughts on pedagogical/didactic challenges and possibilities in a digital course (Relations in digital teaching?), and (v) What is your view on students’ learning from the course? (Examples of learning for practice?). Eight teachers were invited, four agreed to participate, but due to technical challenges, only three completed the focus group interview. The data was first analyzed by author AJ and ML, before a collective analysis. This analysis was also inspired by reflexive thematic analysis (Braun and Clarke, 2019), and patterns and themes were extracted from the text, as described above, resulting in two main themes, both with three sub-themes (see Table 2 in the Results section).

Table 2

Table 2. Teachers’ reflections on digital sexual health courses.

Sampling and analysis: students’ attitudes toward addressing sexual health extended (SA-SH-Ext)

All students were invited to respond online to a pre- and post-test survey using the SA-SH-Ext¹, a 27-item survey with Likert scale responses, validated and reliable for Norwegian students in health and welfare educations (Lunde et al., 2020; Lunde et al., 2022). The response options were translated to numbers (1 = disagree, 2 = partly disagree, 3 = partly agree, 4 = agree, 5 = strongly agree). Participation was voluntary, the invitation offered on Canvas and no reminders were sent out. Students were invited to fill in the SA-SH-Ext before and after course participation. Thirty-five students responded to the pre-test, and 17 to the post-test. The two groups were compared using independent t-test in SPSS, as it was unknown to the researchers who the respondents were and therefore the responses could not be paired from the two surveys. The data was first analyzed by author KAJ, before a collective analysis.

Sampling and analysis: learning analytic

Learning analytics was extracted from interaction in the digital tool FeedbackFruits (incorporated in Canvas), which was used for shared reading and commenting on compulsory course literature (scientific articles) in the courses. Using FeedbackFruits was voluntary; the students also had the option of reading the articles on their own, without digital interaction. Data was retrieved from eight articles that were available at the start of the course, with teacher prompts (comments and questions) present in the tool from the launch of the course. Students were encouraged to read a specific article, answer teacher questions posted in the article, post their own comments and respond to at least one comment. They had the opportunity to interact asynchronously, revisiting comments and continuing interaction with fellow students in a flexible manner and informed that the tool anonymized their identity to other students, and that teachers could identify who had posted comments. The FeedbackFruits data was extracted into an Excel file, and anonymized (names replaced with numbers), and individual student was data linked. The data included comments, all of which had time stamps to allow for the identification of interactions across time. All comments were added to a digital whiteboard with lines representing interaction. The positioning of the comments was based on the time the comment was made, which presented a visualization of interaction across time. This allowed exploration of how students interacted, with whom, in what circumstances and when. The data was first analyzed by author DM, before a collective analysis.

Ethics and pre-understanding

The project was approved by the Norwegian Agency for Shared Services in Education and Research, ref.no: 382694, and the RE was consistent with the requirements of the ethics board. Informed consent was obtained from all participants (students and teachers) before data collection. The research group consisted of six persons with various backgrounds and functions in the project: a professor who was a researcher in the Sexual Health project (KAJ); a senior lecturer who had participated in one reference group for the development of one course and was currently leading sexual health courses (ML); an internal junior lecturer (AJ) and an international exchange student (both not previously involved in the Sexual Health project) (KJS); the project leader of the Sexual Health project (GHL), and an external senior lecturer who had participated in one reference group for the development of one course (DM). Together, the research group had student experience, skills in teaching and researching within sexual health, digital teaching, co-production, and work-integrated learning, and a broad competence in research methodology. In addition, the group had a variety of professional backgrounds, such as social education, health psychology, nursing, and physiotherapy, and came from four European countries.

Results

Examination of how the theoretical framework (co-production, work-integrated learning, digital teaching, and digital didactics) and the initial program theory (see Figure 1) matched is presented below, by first looking at the results from each set of collected data separately: (1) Midway course evaluations, (2) Focus group interview, (3) SA-SH-Ext survey, and (4) Learning analytics based on Feedback Fruits data. Thereafter, a summarizing analysis of results from all data sets was made, to create a refined program theory. The refined program theory is presented after the presentation of the separate data set results in Figure 2.

Figure 2

Figure 2. Context, Mech (Mechanism) and Outcome configurations in the refined program theory.

Data set 1: midway course evaluations

The initial program theory (Figure 1) was that the co-production and work-integrated learning components of the development and delivery of the courses would lead to learning that changed professional practice, and that accessibility through digital teaching would improve students’ (i.e., working professionals’) ability to attend the courses and support a broader geographic and professional coverage of students. The initial program theory also included the idea that digital tools would support learning and student interaction. The analysis of the midway course evaluations with students resulted in three main themes (Sharing with others, Learning with others and Useful for oneself and others) with two sub-themes each, see Table 1.

Sharing with others

In the midway course evaluations, students stated that they did talk with others about different subjects from the sexual health course. Course content was shared with colleagues, friends and family and examples were given of different course content that they thought relevant to share. This is seen as an example of how students were involved in work-integrated learning. Examples of lack of interest in the subject of sexual health among students’ colleagues were given, and a some also described a resistance among colleagues to raising the topic in the workplace.

Learning with others

Students stated that they used and appreciated the different digital learning resources offered and experienced them as improving their learning. They stated that they worked through the different course modules in the learning platform Canvas in a chronological fashion. Scheduled group work was described as a resource, and some students had created pairs outside the scheduled study groups and used each other as ‘sparring partners in learning’. Synchronous lectures (during the webinars) were suggested by one student who also wanted lectures to be more related to course tasks and examinations. When asked to name positive elements in the course, they listed various things such as general aspects found in all courses: the chronology in Canvas; different learning resources such as films, quizzes, the FeedbackFruits tool; study groups; updated and interesting literature and course-specific aspects (teachers). FeedbackFruits was experienced as both rewarding (a good way to reflect and learn together) and challenging (sometimes difficult to understand and use). When asked about what could be improved in the course, the overall comments concerned improvements that would make the content in Canvas clearer, for instance to specify more clearly compulsory and optional assignments. Specifically, students requested a greater frequency of online meetings and lectures, which indicates a wish for more contact time with other students and teachers. In relation to assignments, students requested a higher word count number in group assignments and a reduction in the percentage needed for succeeding in course quizzes, suggesting quite specific changes to improve their learning experience.

Useful for oneself and others

When asked about how the knowledge gained from the course could be used in daily work or practice, students said they used this knowledge in their daily work; either specific knowledge (subspecialties within the courses) or general knowledge (on sexual health). They said the course had been relevant in relation to their various practices (meeting patients, clients and students of various ages and in different settings) and in their overall life, and that the course had led to increased confidence and understanding, and to new insights. As an example, a student wrote: ‘In work with people with mental illness or substance use I find the topic [prevention, detection and follow-up of sexual abuse] very relevant’. They said that taking the course evoked various emotions, for instance curiosity to learn more, and feeling safer approaching the topic of sexual health with clients, patients or colleagues. One student said:

‘I think the course definitely strengthens my competence as a nurse and the threshold for addressing various issues around sexual health will be lower’.

Students mentioned feeling more able to work with sexual health issues, and that they understood the topic sexual health better. A few claimed that their studies had already, or would, benefit patients and service users. One student was taking a course during parental leave, and shared their enthusiasm for returning to work to use the new knowledge:

‘I'm looking forward to starting work again, with new enthusiasm and new knowledge, so that I can use it! Going through the subject initiates a number of reflections in one's head, and one sees things in a different way and with a more curious eye.’

The midway course evaluation data indicate that work-integrated learning worked in the courses and underscores the importance of having opportunities to discuss course content both with peer students and in workplace settings. It also shows that digital tools can enhance learning, but at the same time can be a barrier to learning, if experienced as difficult.

Data set 2: focus group interview

The initial program theory (Figure 1) was that the digital format of the courses could enable positive change in practice through the digital didactics used. The analysis of the focus group interviews with teachers resulted in two main themes (Self-reflections and Students differ) with three sub-themes each, see Table 2.

Self-reflections

Teachers described digital teaching as both challenging (e.g., difficult to teach in a digital environment, and difficult to adjust the teaching to the different students’ needs) and rewarding (e.g., educational and exciting). This is illustrated by statements such as ‘I find it educating.’ followed by ‘Me too. It’s been a steep learning curve, and challenging. But exciting and fun.’ Digital teaching was described as different from other ‘physical’ forms of teaching and as requiring other types of competence: technical, digital, professional, academic, and organizational. At the same time, it was reported that there were many similarities with traditional forms of teaching. They further stated that digital pedagogical relations required clarity and talked about having an overall control–being like a ‘conductor,’ ensuring, for instance, that the information to the students was clear. Management and organization skills were also highlighted, and the need for arenas where teaching experiences could be exchanged between teachers was mentioned.

Students differ

Teachers reflected on students’ technical skills, and that it was easy to overestimate how technical or digitally skilled students are: ‘Right, yeah, you have already said this, and there has been a great variety in how much technical skills they have.’ Teachers saw varying digital and technical skills (i.e., digital literacy) among students as hindering when organizing learning activities that could fit everyone. It was also stressed that digital teaching could create a distance between student and teacher; ‘So, I think the threshold is higher for the students, to contact the teacher. And maybe they feel unsure, and that they ask stupid questions.’ Teachers talked about having to facilitate students’ sense of community to a greater extent in the digital courses compared to their ordinary in-person teaching. They also stated that the sexual health students were different in terms of motivation and interest in the topic, educational background and work experience, but had a common denominator as they all lacked competence in addressing sexual health issues:

‘I think, as [other informant] touched upon, that they do have some previous competence. I believe that, generally, when you choose the master’s level, you might have some insights, and that’s why you choose to take a course, or a master’s, or whatever. But I think it's extra special with sexual health, because it’s still taboo.’

The focus group interview data accentuate the importance of digital didactics when teaching digital courses, as well as ensuring knowledge of students’ digital literacy, to enable them to optimize their learning.

Data set 3: the SA-SH-Ext survey

The initial program theory (Figure 1) was that co-production in development and delivery of courses, along with work-integrated learning components would lead to changes in practice and that patients’/clients’ needs would be better met. No items showed decreased competence, and four items had significant changes between pre- and the post-test:

• Item 7, ‘I feel comfortable about discussing sexual health issues with future clients with cognitive disability’, increased from a mean of 2.83 to a mean of 3.65, p = 0.003.

• Item 11, ‘I feel comfortable about discussing sexual health issues with future clients regardless of their sexual orientation’ increased from a mean of 3.63 to a mean of 4.47, p = 0.021.

• Item 13, ‘I am unprepared to talk about sexual health with future clients’ decreased from a mean of 2.26 to a mean of 1.41, p = 0.004.

• Item 26, ‘I have sufficient competence to talk about sexual health with my future patients’ increased from a mean of 2.69 to a mean of 3.71, p = 0.035.

The data indicate that after the courses, students were better prepared and comfortable discussing sexual health issues with clients with cognitive disability, and with clients regardless of sexual orientation. Students also felt more prepared to talk about sexual health with future clients and considered themselves to have sufficient competence to talk about sexual health. These results are in line with the qualitative data (midterm evaluation from students and focus group with teachers) and follow the initial program theory, where co-production in the development and delivery of the courses and the work-integrated learning components provide learning about sexual health that can lead to change in practice. The results should however be treated with caution considering the small group sizes.

Data set 4: learning analytics

The initial program theory (Figure 1) was that opportunities for student interaction via FeedbackFruits would facilitate critical appraisal and engagement with learning material, and help students gain different perspectives and deeper learning. Through reviewing this FeedbackFruits interaction data we see diverse student approaches (Figure 3). There were 2–12 student views for each of the eight assigned articles with 2–28 student comments. The interval between the original teacher’s prompt and a reply from each student ranged from 5 h to 90 days, approximately 40 days. The duration of the first interaction, determined between the first and last comment, ranged from 4 to 240 min. There were seven instances where an additional interaction occurred, with a second interval ranging between 5 h to 17 days. The second interaction duration was between 4 and 67 min. There were only two examples of an original post provided by a student.

Figure 3

Figure 3. Visualization of learning analytics from the scientific articles in FeedbackFruits. Circles represent comments and lines represents replies.

The result indicates that the flexible learning approach achieved through FeedbackFruits provides opportunities for students to asynchronously engage with learning activities. However, rather than rich student interaction, we saw diverse ways in which a few students engaged with the learning material in relation to time, frequency and duration, an interaction that was limited to teacher-student interaction. The emerging challenge identified concerned the teachers’ instructions for using FeedbackFruits, it may be that clearer direction is required so that students reply to another student’s comment.

Refining the program theory

Based on the findings from the four data sources, the initial program theory was revised through an iterative analysis. The refined program theory is presented with a model adapted from Dalkin et al. (2015) that divides the concept of mechanism into its basic parts of resources and reasoning. By comparing the identified CMO configurations in the initial program theory (Figure 1) with our findings and relevant research, conclusions could be drawn and used as a basis to refine the program theory presented in Figure 2.

The findings suggest that co-produced content is relevant for practice and engaging for students (Outcomes 1 and 2). Combined with digital teaching, the co-produced content provides opportunities for flexible learning and applying relevant learning in practice. Digital didactics provides flexibility for interaction, which in turn can contribute to work-integrated learning, as students felt more comfortable talking about the subject matter with others (Outcomes 3 and 4). The combination of these outcomes suggests that students found the courses relevant for practice, were comfortable talking about the subject in practice and applying learning in practice (Outcome 5), and that change in professional practice is possible (Outcome 6).

Discussion

As there is criticism that evaluations fail to indicate why and how a program works and in which contexts a program could be scaled up (Van Belle et al., 2023), we have presented how the program theory worked in this specific context. As the methodology of RE is still developing and used in new settings (Van Belle et al., 2023), this study can contribute to this development through the use in continued higher education. The study intended to identify benefits and barriers experienced by involved students and teachers and provide learning on how to improve future digital courses in sexual health for professionals to promote change in professional practice. The findings can therefore be of relevance for developers of courses and teachers in higher education, both within the field of sexual health promotion and when evaluating co-produced courses in other types of education. Additionally, the findings provide an example of how higher education regarding sexual and reproductive health and rights for all can be delivered in times when it is being globally politicized (Grahn and Holm, 2025; Lo, 2024). The studied context consisted of co-produced courses in Norway (Areskoug Josefsson and Lunde, 2024) where stakeholders have been part of designing the courses. Considering the resource demands and complexity of co-production (Bell et al., 2021; Bilous et al., 2018), the findings of this study contribute with additional learning of outcomes from investing in co-production in higher education. However, course design needs to consider that each learning context has its own characteristics which affects educational needs (Theelen and van Breukelen, 2022).

The results show that students engage with the course material, and learning outcomes indicate a change in professional practice. However, the learning analytics based on FeedbackFruits show a lack of interaction between the students indicating a need to further encourage collaborative learning to overcome the risks of insufficient learning outcomes from digital courses described by Frenk et al. (2022). It is probable that teachers need additional support in how to use interactive documents and to prompt the students’ critical reading, to optimize use of the tool (Hasan and Rinaldi, 2021). As asynchronous student and teacher interaction is essential in online courses (Theelen and van Breukelen, 2022), knowledge and skills of how teachers can support this interaction is crucial. Thus, future course design must address both engagement and student interaction, as well as ensuring teachers’ digital literacy skills. Results from the midway course evaluations showed that students considered the course material informative, relevant, and engaging. This positive response might be related to the courses being co-produced and delivered with various stakeholders involved.

The work-integrated learning component of the courses was embedded in their designs, following recommendations for work-integrated learning (Campbell et al., 2021). The integration was thus the responsibility of the students and based on them bringing their work experiences into course discussions and assessments and sharing course material at their workplace. This shift of responsibility, to students, differs from traditional work-integrated learning where responsibility lies with the university (Campbell et al., 2021). The courses were interprofessional and interdisciplinary, regarding both students and teachers, and this was, along with the work-integrated learning component, intended to promote real-world competence of students, which the results support. The results from the learning analytics based on FeedbackFruits do not show interaction between students as planned and expected; instead, the interprofessional discussions took place during webinars and in group assignments. This indicates that teachers’ involvement with interactive learning tools needs to increase. A pilot exploration of FeedbackFruits, from a teacher and student perspective, has shown promising results concerning student interaction (Areskoug Josefsson et al., 2023) and previous research has also shown positive results of student interaction and collaboration with interactive documents (Hasan and Rinaldi, 2021). But, as our findings showed mainly interaction between student and teacher, it highlights the risk of superficial data presentation and false conclusions drawn from learning analytics, as described by Seufert et al. (2019). FeedbackFruits had not previously been used by the teachers which may have limited their ability to prompt the discussions, illuminating the need for digital literacy among teachers. Increased digital literacy among teachers can also support the use of more features within Feedbackfruits, such as for example AI feedback to students. Using AI for feedback can further students’ engagement with the taught topic and effectively manage teacher’s workload (Lo et al., 2025). The SA-SH Ext survey results indicate possibilities for changes in professional practice with an increased ability to meet patients’/clients’ needs and to provide equal care. The positive change measured is comparable with results from a two-week interprofessional physical course in sexual health at bachelor level (Gerbild et al., 2018) and classroom training with a patient educator (Felter, 2020).

Findings from the focus group interview showed that teachers considered it easy to overestimate digital competence among students and that digital teaching creates a distance between teacher and student. This is in line with Erlam et al. (2021) who point out that digital teaching can lead to fewer social interactions with students, technical problems, and lack of equal opportunities due to low digital literacy. Diverse levels of digital experience and literacy among students and teachers should be addressed, to improve digital didactics and ensure equal learning opportunities.

Strengths, limitations, and future directions

The collected data includes both anonymous and non-anonymous data, which have pros and cons, but adds to the strength of the study. Anonymous data gives the informant a free way of adding information without having to risk engaging with the recipient. When there is non-anonymous data, the informant must take a greater responsibility when reflecting on how the information is going to be met by other persons in the discussion. The limited sample of persons in the focus group is a weakness, as the small number may have made teachers less willing to share their experiences. The focus group interview guide focused specifically on digital teaching, digital didactics, and learning, and not on all elements of our program theory, which can be seen as a limitation. This was a pragmatic choice as the digital teaching and digital didactics were considered important resources and mechanisms, with potential to highly influence other factors. Additionally, that far from all students chose to participate in midterm evaluation and the pre- and post-test survey is not surprising given that survey fatigue is a global trend, also affecting higher education (Fass-Holmes, 2022). Still, this needs to be taken into consideration when interpreting the results.

The limitation of evaluating a specific set of courses in a time frame of one semester was weighed against the importance of evaluating them close to their development stage and thus ensuring the originality of the co-produced courses. A longer time frame would have meant a greater risk of the courses being altered by teachers, as part of continuous course development. In addition, sexual health promotion is a field where policies, politics and social media can have great influence, and a postponed evaluation could potentially have influenced the results since social changes may not be revealed when conducting a RE (Van Belle et al., 2023). Lastly, the diversity in the research group, with teachers, researchers, and an exchange student, is seen as an asset as all brought valuable knowledge and experiences to this study.

Conclusion

This realist evaluation supports the value of co-producing development and delivery of digital master-level courses in sexual health promotion for health and welfare professionals, and the added value of work-integrated learning in this setting. However, the mechanisms found are dependent on digital didactics and interaction between students, between students and teachers, and students and their workplaces to provide optimal opportunities for learning and applying learning in practice. Confirmation or contradiction of the findings should be tested in other settings, as realist evaluation is an iterative process.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by the Norwegian Agency for Shared Services in Education and Research, ref.no: 382694. The studies were conducted in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required from the participants or the participants' legal guardians/next of kin in accordance with the national legislation and institutional requirements. Participating in the study was an active choice made by adult participants that had received written study information.

Author contributions

KA: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Supervision, Writing – original draft, Writing – review & editing, Funding acquisition, Project administration. ML: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Supervision, Writing – original draft, Writing – review & editing. AJ: Conceptualization, Data curation, Formal analysis, Investigation, Writing – original draft, Writing – review & editing. GL: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Resources, Writing – original draft, Writing – review & editing. KS: Conceptualization, Formal analysis, Investigation, Writing – original draft, Writing – review & editing. DM: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Supervision, Visualization, Writing – original draft, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. This study was supported by The Norwegian Directory for Health (Helsedirektoratet) and The Department for Behavioural Sciences, Faculty for Health Sciences, Oslo Metropolitan University.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The authors declare that no Gen AI was used in the creation of this manuscript.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^https://www.researchgate.net/publication/359826660_Students_Attitudes_towards_Addressing_Sexual_Health_Extended_SA-SH-Ext

References

Areskoug Josefsson, K., Haarr, K. H., Eriksen, S. S., and Brossard Børhaug, F. (2022). Using digital, universal, and intercultural didactics to improve higher education—a project protocol for a Norwegian interactive and collaborative improvement study concerning Master’s level courses in “theory of science, research methods, and research ethics” [study protocol]. Front. Educ. 7:851783. doi: 10.3389/feduc.2022.851783

Crossref Full Text | Google Scholar

Areskoug Josefsson, K., Lona, I., and Lunde, G. H. (2023). Bedre lesing og forståelse av pensum med støtte av digital sosial annotering. [in English: improve curriculum reading and comprehension with the support of digital social annotation]. Norsk Pedagogisk Tidsskrift 107, 283–287. doi: 10.18261/npt.107.3.8

Crossref Full Text | Google Scholar

Areskoug Josefsson, K., and Lunde, G. H. (2024). Learning from coproducing digital courses in sexual health in higher education in Norway. Learn. Organ. 32, 93–108. doi: 10.1108/TLO-11-2022-0143

PubMed Abstract | Crossref Full Text | Google Scholar

Areskoug-Josefsson, K., Schindele, A. C., Deogan, C., and Lindroth, M. (2019). Education for sexual and reproductive health and rights (SRHR): a mapping of SRHR-related content in higher education in health care, police, law and social work in Sweden. Sex Educ. 19, 720–729. doi: 10.1080/14681811.2019.1572501

Crossref Full Text | Google Scholar

Arvidsson, A., Grander, A., and Lindroth, M. (2024). School health-care team members’ reflections of their promotion of sexual and reproductive health and rights (SRHR): important but neglected. Sexual Reproduct. Healthcare 39:100950. doi: 10.1016/j.srhc.2024.100950

PubMed Abstract | Crossref Full Text | Google Scholar

Bell, S., Lee, R., Fitzpatrick, D., and Mahtani, S. (2021). Co-producing a community university knowledge strategy. Front. Sustain. 2:661572. doi: 10.3389/frsus.2021.661572

PubMed Abstract | Crossref Full Text | Google Scholar

Bilous, R., Hammersley, L., Lloyd, K., Rawlings-Sanaei, F., Downey, G., Amigo, M., et al. (2018). ‘All of us together in a blurred space’: principles for co-creating curriculum with international partners. Int. J. Acad. Dev. 23, 165–178. doi: 10.1080/1360144X.2017.1412973

Crossref Full Text | Google Scholar

Björkas, R., Lindroth, M., and Richert, T. (2024). Child welfare workers' understanding of young people’s sexuality when conducting assessments due to substance use problems. Child Youth Serv. Rev. 164:107885. doi: 10.1016/j.childyouth.2024.107885

Crossref Full Text | Google Scholar

Braun, V., and Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qual. Res. Sport, Exerc. Health 11, 589–597. doi: 10.1080/2159676X.2019.1628806

Crossref Full Text | Google Scholar

Campbell, M., Russell, L., Thomson, K., Tunny, R., Smith, L., and McAllister, L. (2021). The construction and testing of a framework to assure the institutional quality of work-integrated learning. Int. J. Work-Integrated Learn. 22, 505–519.

Google Scholar

Dalkin, S. M., Greenhalgh, J., Jones, D., Cunningham, B., and Lhussier, M. (2015). What’s in a mechanism? Development of a key concept in realist evaluation. Implement. Sci. 10:49. doi: 10.1186/s13012-015-0237-x

PubMed Abstract | Crossref Full Text | Google Scholar

Edström, K. (2008). Doing course evaluation as if learning matters most. Higher Educ. Res. Develop. 27, 95–106. doi: 10.1080/07294360701805234

Crossref Full Text | Google Scholar

Endler, M., Al-Haidari, T., Benedetto, C., Chowdhury, S., Christilaw, J., El Kak, F., et al. (2022). Are sexual and reproductive health and rights taught in medical school? Results from a global survey. Int. J. Gynecol. Obstet. 159, 735–742. doi: 10.1002/ijgo.14339

PubMed Abstract | Crossref Full Text | Google Scholar

Erlam, G. D., Garrett, N., Gasteiger, N., Lau, K., Hoare, K., Agarwal, S., et al. (2021). What really matters: experiences of emergency remote teaching in university teaching and learning during the COVID-19 pandemic. Front. Educ. 6:639842. doi: 10.3389/feduc.2021.639842

PubMed Abstract | Crossref Full Text | Google Scholar

Fass-Holmes, B. (2022). Survey fatigue— what is its role in undergraduates’ survey participation and response rates? J. Interdiscipl. Stud. Educ. 11, 56–73.

Google Scholar

Felter, C. E. (2020). Assessing DPT Students' self-perceived readiness to discuss sexual health before and after instruction from a patient-educator. J. Physical Therapy Educ. 34, 19–27. doi: 10.1097/JTE.0000000000000112

Crossref Full Text | Google Scholar

Flodén, J. (2017). The impact of student feedback on teaching in higher education. Assess. Eval. High. Educ. 42, 1054–1068. doi: 10.1080/02602938.2016.1224997

Crossref Full Text | Google Scholar

Folkman, A. K., Åker, T. H., and Areskoug Josefsson, K. (2023). Forhandlinger i samskaping og utvikling av digitale emner i høyere utdanning: Erfaringer fra en observasjonsstudie. Nordisk tidsskrift for helseforskning 19, 1–23.

Google Scholar

Frenk, J., Chen, L. C., Chandran, L., Groff, E. O. H., King, R., Meleis, A., et al. (2022). Challenges and opportunities for educating health professionals after the COVID-19 pandemic. Lancet 400, 1539–1556. doi: 10.1016/s0140-6736(22)02092-x

PubMed Abstract | Crossref Full Text | Google Scholar

Gerbild, H., Larsen, C. M., Rolander, B., and Areskoug-Josefsson, K. (2018). Does a 2-week sexual health in rehabilitation course Lead to sustained change in Students' attitudes? -a pilot study. Sex. Disabil. 36, 417–435. doi: 10.1007/s11195-018-9540-1

PubMed Abstract | Crossref Full Text | Google Scholar

Grahn, M., and Holm, M. (2025). Trans inclusive no more? Allies and adversaries of gender self-identification in Sweden. Eur. J. Polit. Gender 8, 234–241. doi: 10.1332/25151088Y2024D000000052

Crossref Full Text | Google Scholar

Hasan, D., and Rinaldi, M. (2021). Stimulating active participation with class materials using interactive document. e-mentor 90, 84–90. doi: 10.15219/em90.1525

Crossref Full Text | Google Scholar

Keller, C., and Hrastinski, S. (2009). Towards digitally literate university teachers. Nordic J. Digit. Lit. 4, 104–115. doi: 10.18261/ISSN1891-943X-2009-02-04

Crossref Full Text | Google Scholar

Kenny, B., O'Connor, D., Sugden, E., Tang, C. Y., Tannous, C., and Thyer, E. (2023). Engaging industry in health professions’ education: benefits and challenges. Int. J. Environ. Res. Public Health 20:6131. doi: 10.3390/ijerph20126131

PubMed Abstract | Crossref Full Text | Google Scholar

Lindskog, A., Lindroth, M., Holmgren, K., and Gunnarsson, B. (2024). Balancing on a slack line – staffs’ experiences of talking about sexuality and sexual health with patients cared for in forensic psychiatry in Sweden. Front. Psych. 15:1450377. doi: 10.3389/fpsyt.2024.1450377

PubMed Abstract | Crossref Full Text | Google Scholar

Lo, N. P. K. (2024). The paradox explained: the advancement of LGBTQ rights in the context of growing illiberalism. J. Infrastruct. Policy Develop. 8:8859. doi: 10.24294/jipd8859

Crossref Full Text | Google Scholar

Lo, N., Wong, A., and Chan, S. (2025). The impact of generative AI on essay revisions and student engagement. Comput. Educ. Open 100249. doi: 10.1016/j.caeo.2025.100249

Crossref Full Text | Google Scholar

Lunde, G. H., Bakke, A., and Areskoug Josefsson, K. (2020). Piloting a research-oriented teaching model in a bachelor programme for social educators – a way to increase competence in research methodology and sexual health? UNIPED 43, 260–274. doi: 10.18261/issn.1893-8981-2020-03-08

Crossref Full Text | Google Scholar

Lunde, G. H., Blaalid, L., Gerbild, H., and Areskoug Josefsson, K. (2022). Assessment of the psychometrics of the Students' attitudes towards addressing sexual health extended (SA-SH-Ext) questionnaire for social educator students. Sex Med 10:100507. doi: 10.1016/j.esxm.2022.100507

PubMed Abstract | Crossref Full Text | Google Scholar

Masterson, D., Areskoug Josefsson, K., Robert, G., Nylander, E., and Kjellström, S. (2022). Mapping definitions of co-production and co-design in health and social care: a systematic scoping review providing lessons for the future. Health Expect. 25, 902–913. doi: 10.1111/hex.13470

PubMed Abstract | Crossref Full Text | Google Scholar

Masterson, D., Lindenfalk, B., Kjellström, S., Robert, G., and Ockander, M. (2024). Mechanisms for co-designing and co-producing health and social care: a realist synthesis. Res. Involv. Engage. 10:103. doi: 10.1186/s40900-024-00638-3

PubMed Abstract | Crossref Full Text | Google Scholar

Ministry of Health and Care Services. (2016). Snakk om det! Strategi for seksuell helse (2017–2022). [in English: talk about it! Strategy for sexual health (2017-2022)].

Google Scholar

National Institute for Health and Care Research. (2021), Guidance on co-producing a research project, April 2021. Available online at: https://www.learningforinvolvement.org.uk/content/resource/nihr-guidance-on-co-producing-a-research-project/ (Accessed August 16, 2024)

Google Scholar

Norström, A. V., Cvitanovic, C., Löf, M. F., West, S., Wyborn, C., Balvanera, P., et al. (2020). Principles for knowledge co-production in sustainability research. Nat. Sustain. 3, 182–190. doi: 10.1038/s41893-019-0448-2

Crossref Full Text | Google Scholar

Olsson, A. K., Bernhard, I., Arvemo, T., and Lundh Snis, U. (2021). A conceptual model for university-society research collaboration facilitating societal impact for local innovation. Eur. J. Innov. Manag. 24, 1335–1353. doi: 10.1108/EJIM-04-2020-0159

Crossref Full Text | Google Scholar

Ossian Nilsson, E. (2019). Innovative learning and innovative learning spaces. Asian J. Distance Educ. 14, 98–116.

Google Scholar

Pawson, R. (2013). The science of evaluation: A realist manifesto. Bodwin, Cornwall: MPG Books Group.

Google Scholar

Pawson, R., and Tilley, N. (1997). “An introduction to scientific realist evaluation” in Evaluation for the 21st century: A handbook. eds. E. Chelimsky and W. R. Shadish (United States of America: Sage Publications), 405–418.

Google Scholar

Røe, Y., Wojniusz, S., and Bjerke, A. H. (2022). The digital transformation of higher education teaching: four pedagogical prescriptions to move active learning pedagogy forward [perspective]. Front. Educ. 6:784701. doi: 10.3389/feduc.2021.784701

Crossref Full Text | Google Scholar

Schellekens, L. H., Kremer, W. D. J., Van der Schaaf, M. F., Van der Vleuten, C. P. M., and Bok, H. G. J. (2023). Between theory and practice: educators’ perceptions on assessment quality criteria and its impact on student learning [original research]. Front. Educ. 8:1147213. doi: 10.3389/feduc.2023.1147213

Crossref Full Text | Google Scholar

Seufert, S., Meier, C., Soellner, M., and Rietsche, R. (2019). A pedagogical perspective on big data and learning analytics: a conceptual model for digital learning support. Technol. Knowl. Learn. 24, 599–619. doi: 10.1007/s10758-019-09399-5

Crossref Full Text | Google Scholar

Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., et al. (2021). A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ 374:n2061. doi: 10.1136/bmj.n2061

PubMed Abstract | Crossref Full Text | Google Scholar

Solberg, A. S., and Areskoug Josefsson, K. (2024). Sexual and reproductive health and rights content in higher education in Norway – a quantitative document analysis. Sex Educ. 1–17. doi: 10.1080/14681811.2024.2370305

PubMed Abstract | Crossref Full Text | Google Scholar

Sunnemark, L., Sunnemark, F., Dahlquist, K., Gahnström, E., Assmo, P., and Piper, L. (2023). Bridging theory and practice through work-integrated learning (WIL): critical perspectives on the conceptualisations of WIL at a university in Sweden. Critical Stud. Educ. 65, 403–420. doi: 10.1080/17508487.2023.2294462

PubMed Abstract | Crossref Full Text | Google Scholar

Svae, G. B., Hassel, B., and Søndenaa, E. (2023). People with intellectual disabilities and harmful sexual behaviour: Professionals' views on the barriers to prevent harm. J. Appl. Res. Intellect. Disabil. 36, 176–185. doi: 10.1111/jar.13048

PubMed Abstract | Crossref Full Text | Google Scholar

Theelen, H., and van Breukelen, D. H. J. (2022). The didactic and pedagogical design of e-learning in higher education: a systematic literature review. J. Comput. Assist. Learn. 38, 1286–1303. doi: 10.1111/jcal.12705

Crossref Full Text | Google Scholar

Van Belle, S., Abejirinde, I.-O., Ssennyonjo, A., Srinivas, P. N., Hebbar, P., and Marchal, B. (2023). How to develop a realist programme theory using Margaret Archer’s structure–agency–culture framework: the case of adolescent accountability for sexual and reproductive health in urban resource-constrained settings. Evaluation 29, 259–275. doi: 10.1177/13563890231185167

Crossref Full Text | Google Scholar

Wong, G., Westhorp, G., Manzano, A., Greenhalgh, J., Jagosh, J., and Greenhalgh, T. (2016). RAMESES II reporting standards for realist evaluations. BMC Med. 14:96. doi: 10.1186/s12916-016-0643-1

PubMed Abstract | Crossref Full Text | Google Scholar