Supporting Higher Education—Hospital Transition through Blended Learning

Introduction and Purpose

Blended learning is a pedagogical approach which is designed to enhance complementarity between face-to-face and online learning (Dziuban et al., 2004). According to Qian et al.’s (Qian et al., 2016, p. 8) systematic review and meta-analysis “Blended learning appears to have a consistent positive effect in comparison with no intervention and appears to be more effective than or at least as effective as nonblended instruction for knowledge acquisition in health professions.” In line with Hew and Cheung (2014) and Kich (2016), we are not particularly concerned with the mere ratio of delivery modes. What we consider crucial in blended learning is the effectiveness of the pedagogic design in taking advantage of both environments to improve the student–student as well student–teacher interaction and collaboration. Such interaction is crucial in education that focuses on learning outcomes and specifically in competence-based medical education (CBME) which promotes the “integration of knowledge, skills, and behaviors in practice” (Snell and Frank, 2010, Bourgeois et al., 2015). According to the CBME approach, “competence” cannot be reduced to a list of learning objectives or reductionist performance as it implies a broad objective (Snell and Frank, 2010; Parent et al., 2013). The paper describes two research and educational tools that can be instrumental within a blended learning approach in tracking, discussing, and reflecting on the professional development of students in the context of Italian hospitals. The two tools are complementary to each other in offering data that take into account the international frameworks and debate on the identification and description of the competences that health professionals are required to meet.

The purpose of this paper is to present two exploratory studies that have a potential for identifying research-based blended learning tools for CBME. The studies focus on the development of a group of medical students’ professional image (in Genova and Rome) during their hospital internship. The paper addresses three related questions:

1. How does their professional representation change in the transition from university to hospital?

2. Which competences emerge from the students’ reports?

3. Which competences are still under represented?

Competence-based medical education has been the subject of international studies and frameworks such as Tuning (Marz et al., 2013) and CanMEDS (Frank et al., 2015). For the purpose of this study, we are referring to CanMEDS’ seven key areas of competence as developed by the Royal College of Physicians and Surgeons of Canada. The core physician competence is that of medical expert, which includes all the necessary biomedical expertise. However, CanMEDS provides evidence that this is just one component of a physician’s practice. The competences related to the roles of communicator, collaborator, health advocate, leader, scholar, and professional are equally important for a competent medical practitioner (Frank et al., 2015).

Blended learning offers a favorable setting to present CBME-related problems through shared reporting and discussion of critical incidents (Fitzgerald, 2000). Online tools (Danver, 2016) such as forums, microblogging, threaded discussion support flexible ways of presenting selected case studies as well as trainees’ first-hand experience and to have learners responding to them individually and through group activities. A constructivist approach (Canipe, 2016) encouraging reflection (Gbenga, 2016) and based on the critical examination of the developing ideas seems to have strong potential for supporting students in developing a deeper understanding of the competences related to physician roles and the ways they should be prioritized and related to each other in clinical practice.

Method

Two exploratory studies were implemented using the questionnaire and database activities of the e-learning Moodle platform. Each study was centered on a specific research and education tool: the situational judgment test (SJT, as a questionnaire collecting behavioral data) and reflective writing (RW, as database offering personal reflections).

Situational judgment test is a performance prediction tool. It presents various scenarios describing a critical incident, a dilemma, or a problem requiring relevant competences to be solved (Weekley and Ployhart, 2006; Christian et al., 2010; Metcalfe and Dev, 2014). For each situation described, there are multiple and alternative actions. The test taker has to make judgments about the course of actions presented and choose the most appropriate response using a forced-choice or Likert-scale format (Rockstuhl et al., 2015). Nine experts were asked to assess the content validity of the tool by rating scenarios and alternative courses of actions (Lynn, 1986). Scenarios and actions incorporated into the SJT had at least 80% of the experts agreeing on their validity (Voutilainen and Liukkonen, 1995).

According to Sandars (2009), “Reflection is a metacognitive process that creates greater understanding of self and situations to inform future action” (p. 685). In medical education, including blended learning environments, there has been an increase in RW, critical incident reports based on experience as educational tools in which reflection becomes an opportunity to learn, to develop a therapeutic relationship and professional practices (Rowe et al., 2012).

Participants

In the first study, medical students attending two Italian Universities (Genoa and Rome “La Sapienza”) were involved. 320 students had access to the computer-based SJT survey and 241 students completed it: 66 first year students from the University of Genoa, 175 from the University of Rome (19 first year, 48 third year, 60 fifth year, and 48 sixth year students). Data were collected in an anonymous way. The SJT survey and its goals (including this research project) were presented to each group of students during a plenary meeting. Because of its educational value, after completion the survey results were presented and discussed with the students in another plenary session.

The second study involved 20 medical students (equally divided between female and male) attending the third year at the University of Rome. They wrote their reflective reports in three different phases: at the beginning, during, and at the end of their first hospital internship. RW is a regular component of the third year course Introduction to Medicine and students are supported by a mentor through one-to-one interactions. For the purpose of this study, texts were anonymized by the mentor and so were any reference to dates, locations, and patients.

An ethics approval was not required according to both national regulation on data protection and respect of privacy and sensitive data (d.l. 196/2004) and institutional regulations, because the survey data were anonymized for research purposes. The consent of the participants was implied by virtue of survey completion. For the same reason, explicit consent was not necessary for the 20 students from the University of Rome, who wrote their narratives as an ordinary assignment and had their writings anonymized before being used for this study.

Data Collection and Analysis

The SJT survey was designed by a team of six experts. They opted for the inclusion of 14 scenarios to have 2 for each CanMEDS key competence area. For each scenario, students are presented with five alternative courses of action. Scenarios and alternative actions were identified on the basis of the enabling competences as presented within the CanMEDS framework. The test taker has to judge and rank the enabling competences according to the priority that the student attributes to the various competences in relation to the specificities of the given scenario. The following is an example of a scenario included in the SJT survey:

A 52-year-old woman is hospitalized because of a very advanced stomach cancer. There is no possibility of surgical or chemotherapy. She has acute pain and cannot be fed. When talking to her, you would like to have:

• Ability to communicate her clinical situation

• Ability to make her understand that you will take care of her

• Ability to manage her care system, even after the hospitalization

• Ability to explain how you will coordinate with other specialist colleagues

• Ability to give her information on relevant studies.

The data were collected through the Moodle platform and analyzed with SPSS software.

During the General Surgery course, students were invited to write a reflective text at the beginning, during, and at the end of their hospital internship, according to the following prompts:

• Report your interpretation of the signs and symptoms that you have observed.

• Report your emotions: how did you feel? In which situation? Why? Did your emotions affect your behavior? How will emotions affect you as physician?

• Report examples of good or bad professional practices.

• Report and describe the communicative situation.

The 60 RWs were uploaded on Moodle using database. A content analysis with NVivo was conducted to identify relevant personal reflections that refer back to CanMEDS key competence areas.

The two datasets are available at Zenodo (https://zenodo.org/).

Results and Discussion

Two SJT outputs were elaborated. The first one is a radar graph presenting the mean score for each CanMEDS competence area. The final score was calculated by weighting the students’ answers on a scale from 0 to 100 according to the priority ranking as previously defined by the expert group. A t-test to compare university and gender group means was conducted. The analysis shows that there is statistical evidence that gender means are significantly different in the communication competence area (Figure 1).

FIGURE 1

Figure 1. Students overall scores in the seven CanMEDS competence areas.

In addition, a graph is provided to each student indicating the individual score and the overall group score. By highlighting the competence areas that are taken into account, feedback is provided that supports the student’s reflection concerning the individual sense-making process in relation to the various scenarios (Figure 2).

FIGURE 2

Figure 2. Example of single student scores in the seven CanMEDS competence areas.

Weekley and Ployhart (2006) suggest addressing the black box of situational judgment by asking the test taker to describe her/his judgment reasoning. Each student receives her/his own individual graph with some questions to support her/him in making her/his reasoning explicit. This second analysis is still in progress and also involves the use of focus groups.

Two qualitative analyses were conducted on the 60 reflective texts. A top-down analysis using the CanMEDS competence area (Table 1) shows that communication is the most frequently described (97 references) competence area by students during their internship.

TABLE 1

Table 1. References to CanMEDS in student reflective texts.

The second analysis focused on the communication competence area alone. The analysis of students’ narratives provides relevant examples of different ways to approach patients and to collect and elaborate information in relation to clinical reasoning. Narratives also help to identify critical aspects in conceptualizing disease and illness issues and the role of clinicians’ emotions and the way they are taken into account in decision-making.

Psychological and Feeling Dimension

Three types of sentences are relatively frequent in students’ RW and they address the psychological dimension of physician–patient communication (Rosenstein, 2016). Such sentences concern physician’s psychology (for a total of 18 references), patient’s psychology (8), and the ways patient’s psychology triggers respectful attitudes and behaviors by physicians and trainees (14). An underlying question is related to this type of RW: what role do emotions play within the communication process involving trainees and patients? (Branch et al., 2001).

Stress and communication control are two commonly debated issues concerning physician psychology.

From the students’ writings, it is evident that the traineeship plays a crucial role in exposing many of them to physician–patient interaction, which is experienced as a stressful situation. Many of them do not feel comfortable with such interaction and feel they need to elaborate it to be better equipped to face such interactions in their prospective career. The following excerpt from a female student exemplifies this significant and recurrent experience:

I was holding in my hand the form with the questions we were supposed to ask the patient. I felt very shy. Talking with the patient and trying to address the patient’s problem I realised how important and how necessary it is to be a competent physician¹ (female student).

As the amount of contact with patients and their families increases, some students are also becoming more aware of the need to develop their empathic attitude and to better understand what type of empathic communication is appropriate within a clinical environment. As one male student writes:

the physician should try to understand the concerns of the relatives and the questions that they are asking themselves. S/he should be able to explain the patient’s condition and the next steps in a kind way while at the same time s/he should be able to act in a rational way and to distinguish between being compassionate and being emotional² (male student).

Such RW offers blended learning tutors room for encouraging trainees to explore and to become aware of the relation between empathic-cold attitude and open-control communication in their daily interactions with patients. Previous evidence in this field shows that physicians with high perceived control tend to use more open communication and be rated as more empathic by patients while physicians with low perceived control would use a controlling communication style (Silvester et al., 2007).

For this aim, it is crucial to pay attention to and record elements of patient psychology, a type of narrative writing that is less frequent among students compared to writings addressing physician psychology. When trainees focus on the psychology of patients, it is their motivation that they tend to consider as a crucial aspect in terms of being able to cope with their illness, as the following extract by a male student highlights:

The hospital brings me in contact with patients that are not feeling well. Nonetheless they try to look beyond their present condition. They try to come to terms with their illness not as something that is here to stay but rather as an aspect of their life that they have to face and to experience in their everyday life without losing hope³ (female student).

Strategic Thinking

Demonstrating such an empathic attitude depends on the physician’s ability to communicate with an accessible and patient friendly language and to choose when to avoid a paradigmatic approach (Bruner, 1990) in favor of adopting and welcoming a narrative approach (as 10 of the students’ excerpts reveal). This ability is often at the core of the possibility to identify and to nurture bonding factors between the physician and the patient. Labov and Waletzky (1967) identify two components of the narrative structure that have an educational potential in guiding trainees in reviewing and reflecting upon their experiences with hospital narrative accounts. Namely, Labov and Waletzky (1967) suggest organizing reflection upon what is being narrated by asking “what happened” and “why it is worth telling.”

Some students show an awareness of the first question as a proper trigger to break the ice with the patient especially in difficult situations.

The patient was nervous. Therefore, in order to try to relax her I asked her to tell me about the incident dynamic⁴ (male student).

It is also a “device” that suggests that the trainee adopt an active listening attitude, a category that is identified in 14 of the students’ texts. The active listening capacity is one of the key concepts of the CanMEDS communicator competence area and often it goes hand in hand with the ability to pay attention to the diversity and uniqueness of patients (8 excerpts), to gather contextual information (20), and to identify and seek solutions to trainee–patient problematic relations (10).

The second question (“why it is worth telling”) is at the core of the patient centered CanMEDS communication approach that explicitly suggests physician–patient shared decision-making (Neville et al., 2015, p. 5). This is translated in the following “enabling competence”: “Elicit and synthesize accurate and relevant information, incorporating the perspectives of patients and their families” (Neville et al., 2015, p. 6). Such strategic thinking on the part of the physician seems to be a possible “core element” of a blended learning process based on experiential learning (a concept addressed in 14 students’ texts) and aimed at enhancing students’ awareness that

verbal and nonverbal communication should aim at a patient centred approach, i.e. listening in a holistic way to her/him as an active and participative subject in the healing process. Within this process it is necessary to welcome the patient, to ensure that s/he is comfortable. We should be able to understand deeply her/his problems and requests⁵ (male student).

In fact, in relation to managing clinical information, the patient is conceptualized by different students in a dichotomous way.

A first group of students adopts a physician-centered position and a paternalistic attitude:

After a 30-minute talk, he remembers to tell us that he was operated on for some bladder papilloma some thirty years ago. No Mr Ivo! You’re betraying my confidence, you can’t lie to me, but patients often do⁶ (male student).

Other students adopt a person-centered approach based on the autonomy of an informed patient.

He knows exactly what he is talking about, he even explains to us things that we have not studied yet⁷ (female student).

These two positions are not mutually exclusive. They are to be found on a continuum at the core of the current debate on the decision-making process in health care (Bomba, 2017; Hamilton et al., 2017), which offers further opportunities for triggering and scaffolding student reflection through a blended learning process.

Toward Negative Capability

This reflective dimension seems particularly important to acknowledge situations where physicians are confronted with their personal and professional limits. As one female student writes

Will I be able to understand when the time comes to act for the patient’s sake, doing nothing, to admit being powerless and to resign serenely?⁸ (female student).

This personal awareness also implies an awareness of the patient’s context and relatives. This is revealed in the following excerpts from a female student’s reflective text:

The emotions I felt in front of that bed were many and diverse. To begin with I felt “out of place” and “interfering” in relation to such a dramatic situation. In the end, we had to try to accept what was going on and to respect their (i.e. the patient’s and family’s) decision even if at the beginning it had appeared irrational to us⁹ (female student).

Meeting patients and relating to the way patients are experiencing their illness confronts students with their capacity to judge when it is time to not take a further initiative and therefore to enact a sort of “negative capability.” The RWs offer students a transitional space to reflect on such negative capability, leaving aside the usual “positive capability,” the one who supports “decisive action.” “Negative capability supports ‘reflective inaction,’ that is, the ability to resist dispersing into defensive routines when leading at the limits of one’s knowledge, resources and trust” (Simpson et al., 2002, p. 1209).

Conclusion

The way students address critical incidents, their self-report, and narrative texts can be purposefully organized within a blended educational environment to play a generative learning role. The tools tested through the two studies presented here contribute to scaffolding the students’ reflective practice in relation to physicians’ core competences and for supporting faculties in designing higher education curricula (Johna et al., 2014, Lempp and Seale 2004).

In addition, such tools allow medical education research to explore the ways students experience the transition between higher education and health practices. In turn, an understanding of this transition from a professional development perspective helps medicine to focus on ways to perceive and to approach the complexities of illness, i.e., the ways clinicians can better formulate their roles with respect to the patients and clinical reasoning, both in its technical dimensions and in terms of meaning (Charon, 2004).

Author Contributions

The paper was jointly written by the two authors.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors would like to thank Prof. Fabrizio Consorti (University “La Sapienza,” Rome), Prof. Antonella Lotti (University of Genoa), and Prof. Francesca Helm for their collaboration. They would also like to thank Dr. Andrea Sciandra who provided support and supervision with quantitative analysis.

Footnotes

1. ^Avevo in mano il modulo con le domande da fare, ma ero molto intimidita. Parlando col paziente e di fronte al suo problema, ho capito quanto era importante e necessario essere competente come medico.
2. ^Il medico dovrebbe comprendere la preoccupazione e le domande dei familiari e gentilmente spiegare le “mosse” e le condizioni del paziente, ma al contempo stesso dovrebbe essere razionale e pronto a scindere la compassione con l’emotività.
3. ^Nell’ospedale si incontrano infatti persone che nonostante il loro malessere cercano di guardare oltre e affrontare la malattia non come qualcosa di immobile e sempre presente, ma come un aspetto della loro vita che va affrontato e vissuto nella quotidianità senza perdere la speranza.
4. ^Notando una discreta agitazione della paziente le chiedo di raccontarmi meglio la dinamica dell’incidente per cercare di farla un po’ rilassare.
5. ^La comunicazione verbale e non verbale deve tendere a un approccio incentrato sull’ascolto del paziente nella sua globalità, soggetto attivo e partecipe nel processo di cura della malattia. In questo percorso è necessario accogliere il paziente, metterlo a proprio agio, essere in grado di capire a fondo le sue problematiche e le sue richieste.
6. ^Dopo mezz’ora di chiacchierata si ricorda di dirci di essere stato operato a dei papillomi vescicali trent’anni fa. No signor Ivo! Così tradisce la mia fiducia, non può mentirmi, ma i pazienti spesso lo fanno.
7. ^Sa perfettamente ciò di cui sta parlando, spiegando, addirittura, lui a noi alcune cose che ancora non abbiamo studiato.
8. ^Saprò essere in grado anche io di capire quando sarà il momento di agire nel bene del paziente non facendo nulla, di ammettere di essere impotenti e di rassegnarsi, serenamente.
9. ^Le emozioni provate di fronte a quel letto sono state tante e diverse; partendo dal disagio iniziale del sentirsi “di troppo” e “di disturbo” davanti ad una situazione così drammatica, fino al tentativo finale, anche da parte nostra, di accettare quello che stava accadendo e di rispettare la loro decisione, che all’inizio sembrava così illogica.

References

Bomba, P. (2017). Supporting the patient voice: building the foundation of shared decision-making. J. Am. Soc. Aging 41, 21–30.

Google Scholar

Bourgeois, J. A., Hategan, A., and Azzam, A. (2015). Competency-based medical education and scholarship: creating an active academic culture during residency. Perspect. Med. Educ. 4, 254–258. doi: 10.1007/s40037-015-0218-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Branch, W. T., Kern, D., Haidet, P., Weissmann, P., Gracey, C. F., Mitchell, G., et al. (2001). The patient-physician relationship. Teaching the human dimensions of care in clinical settings. J. Am. Med. Assoc. 286, 1067–1074. doi:10.1001/jama.286.9.1067

CrossRef Full Text | Google Scholar

Bruner, J. (1990). Acts of Meaning. Cambridge, MA: Harvard University Press.

Google Scholar

Canipe, S. (2016). “Constructivism,” in The SAGE Encyclopedia of Online Education, ed. S. L. Danver (Los Angeles: SAGE), 247–251.

Google Scholar

Charon, R. (2004). Narrative and medicine. N. Engl. J. Med. 350, 862–864. doi:10.1056/NEJMp038249

CrossRef Full Text | Google Scholar

Christian, M. S., Edwards, B., and Bradley, J. C. (2010). Situational judgment tests: constructs assessed and a meta-analysis of their criterion-related validities. Pers. Psychol. 63, 83–117. doi:10.1111/j.1744-6570.2009.01163.x

CrossRef Full Text | Google Scholar

Danver, S. L. (ed.) (2016). The SAGE Encyclopedia of Online Education. Los Angeles: SAGE.

Google Scholar

Dziuban, C. D., Hartman, J. L., and Moskal, P. D. (2004). Blended Learning, Vol. 7. Educause Center for Applied Research (ECAR) Research Bulletin. Available at: https://www.educause.edu/ir/library/pdf/ERB0407.pdf

Google Scholar

Fitzgerald, M. (2000). “Establishing cultural competency for health professionals,” in Anthropological Approaches to Psychological Medicine, eds V. Skultans and J. Cox (London: Jessica Kingsley), 184–200.

Google Scholar

Frank, J. R., Snell, L., and Sherbino, J. (eds) (2015). CanMEDS 2015 Physician Competency Framework. Ottawa: Royal College of Physicians and Surgeons of Canada.

Google Scholar

Gbenga, A. E. (2016). “Reflexive learning,” in The SAGE Encyclopedia of Online Education, ed. S. L. Danver (Los Angeles: SAGE), 947–950.

Google Scholar

Hamilton, J. G., Lillie, S. E., Alden, D. L., Scherer, L., Oser, M., Rini, C., et al. (2017). What is a good medical decision? A research agenda guided by perspectives from multiple stakeholders. J. Behav. Med. 40, 52–68. doi:10.1007/s10865-016-9785-z

PubMed Abstract | CrossRef Full Text | Google Scholar

Hew, K. F., and Cheung, W. S. (2014). Using Blended Learning. Berlin: Springer.

Google Scholar

Johna, S., Woodward, B., and Patel, S. (2014). What can we learn from narratives in medical education? Perm. J. 18, 92–94. doi:10.7812/TPP/13-166

CrossRef Full Text | Google Scholar

Kich, M. (2016). “Blended learning,” in The SAGE Encyclopedia of Online Education, ed. S. L. Danver (Los Angeles: SAGE), 137–142.

Google Scholar

Labov, W., and Waletzky, J. (1967). “Narrative analysis,” in Essays on the Verbal and Visual Arts, ed. J. Helm (Seattle: University of Washington Press), 12–44.

Google Scholar

Lempp, H., and Seale, C. (2004). The hidden curriculum in undergraduate medical education: qualitative study of medical students’ perceptions of teaching. BMJ 329, 770–773. doi:10.1136/bmj.329.7469.770

CrossRef Full Text | Google Scholar

Lynn, M.R. (1986). Determination and quantification of content validity. Nursing Research 35, 382–385. doi:10.1097/00006199-198611000-00017

CrossRef Full Text | Google Scholar

Marz, R., Dekker, F. W., Van Schravendijk, C., O’Flynn, S., and Ross, M. T. (2013). Tuning research competences for Bologna three cycles in medicine: report of a MEDINE2 European Consensus Survey. Perspect. Med. Educ. 2, 181–195. doi:10.1007/s40037-013-0066-z

PubMed Abstract | CrossRef Full Text | Google Scholar

Metcalfe, D., and Dev, H. (2014). Situational Judgement Test, 2nd Edn. Oxford: Oxford University Press.

Google Scholar

Neville, A., Weston, W., Martin, D., Samson, L., Feldman, P., Wallace, G., et al. (2015). “Communicator,” in CanMEDS 2015 Physician Competency Framework, eds J. R. Frank, L. Snell, and J. Sherbino (Ottawa: Royal College of Physicians and Surgeons of Canada), 16–17

Google Scholar

Parent, F., Jouquan, J., and De Ketele, J. M. (2013). CanMEDS and other “competency and outcome-based approaches” in medical education: clarifying the ongoing ambiguity. Adv. Health Sci. Educ. Theory Pract. 18, 115–122. doi:10.1007/s10459-012-9402-z

CrossRef Full Text | Google Scholar

Qian, L., Weijun, P., Fan, Z., Rong, H., Yingxue, L., and Weirong, Y. (2016). The effectiveness of blended learning in health professions: systematic review and metaanalysis. J. Med. Internet Res. 18, e2. doi:10.2196/jmir.4807

CrossRef Full Text | Google Scholar

Rockstuhl, T., Ang, S., Ng, K.-Y., Lievens, F., and Van Dyne, L. (2015). Putting judging situations into situational judgment tests: evidence from intercultural multimedia SJTs. J. Appl. Psychol. 100, 464–480. doi:10.1037/a0038098

PubMed Abstract | CrossRef Full Text | Google Scholar

Rosenstein, A. H. (2016). Understanding the psychology behind physician attitudes, behaviors, and engagement as the pathway to physician well-being. J. Psychol. Clin. Psychiatry 5, 00312. doi:10.15406/jpcpy.2016.05.00312

CrossRef Full Text | Google Scholar

Rowe, M., Frantz, J., and Bozalek, V. (2012). The role of blended learning in the clinical education of healthcare students: a systematic review. Med. Teach. 34, 216–221. doi:10.3109/0142159X.2012.642831

PubMed Abstract | CrossRef Full Text | Google Scholar

Sandars, J. (2009). The use of reflection in medical education: AMEE Guide No. 44. Med. Teach. 31, 685–695. doi:10.1080/01421590903050374

PubMed Abstract | CrossRef Full Text | Google Scholar

Silvester, J., Patterson, F., Koczwara, A., and Ferguson, E. (2007). “Trust me…”: psychological and behavioral predictors of perceived physician empathy. J. Appl. Psychol. 92, 519–527. doi:10.1037/0021-9010.92.2.519

CrossRef Full Text | Google Scholar

Simpson, P., French, R., and Harvey, C. (2002). Leadership and negative capability. Hum. Rel. 55, 1209–1226. doi:10.1177/a028081

CrossRef Full Text | Google Scholar

Snell, L. S., and Frank, J. R. (2010). Competencies, the tea bag model, and the end of time. Med. Teach. 32, 629–630. doi:10.3109/0142159X.2010.500707

CrossRef Full Text | Google Scholar

Snell, L. S., and Frank, J. R. (2010). Competencies, the tea bag model, and the end of time. Med. Teach. 32, 629–630. doi:10.3109/0142159X.2010.500707

CrossRef Full Text | Google Scholar

Voutilainen, P., and Liukkonen, A. (1995). Senior Monitor-laadun arviointimittarin sisällön validiteetin määrittäminen (Senior monitor-determining content validity for quality assessment instrument). Hoitotiede. 1, 51–56. [In Finnish, quoted in Hyrkas, K., Appelqvist-Schmidlechner, K., and Oksa, L. (2003). Validating an instrument for clinical supervision using an expert panel. Int. J. Nurs. Stud. 40(6), 619–625].

Google Scholar

Weekley, J., and Ployhart, R. (2006). Situational Judgment Tests: Theory, Measurement, and Application. New Jersey: Lawrence Erlbaum Publisher.

Google Scholar