Yvonne Schaffler1*
Andrea Jesser2
Elke Humer1,2
Katja Haider1
Christoph Pieh1
Thomas Probst3†
Brigitte Schigl4†- 1Department for Psychosomatic Medicine and Psychotherapy, University for Continuing Education Krems, Krems, Austria
- 2Faculty of Psychotherapy Science, Sigmund Freud University Vienna, Vienna, Austria
- 3Division of Psychotherapy, Department of Psychology, Paris Lodron University Salzburg, Salzburg, Austria
- 4Department Psychology and Psychodynamics, Karl Landsteiner University of Health Sciences, Krems, Austria
Background: While most studies assessing psychotherapy efficacy are randomized-controlled trials conducted in research institutions or short clinical treatments, the understanding of psychotherapy effectiveness under regular, clinically representative conditions, particularly in outpatient practice, remains limited. Representative data examining the effectiveness of psychotherapy under real-world conditions in Austria is lacking.
Aims and Methods: This paper introduces a naturalistic observational combined process- and outcome study, implementing a dual-perspective approach through standardised pre- and post-treatment questionnaires and evaluating changes in the therapeutic alliance after each session. Further, semi-structured qualitative interviews aim to illuminate the personal experiences of patients and therapists. The primary objective of the presented study is to discern whether symptoms markedly decrease following therapy. A significant secondary goal is to trace the therapeutic alliance’s evolution from both patient and therapist viewpoints, emphasising the alliance-outcome association and gender dynamics within the pairs. This paper discusses the project’s feasibility after three years and shares key insights.
Discussion: Recruitment for this study has posed substantial challenges due to psychotherapists’ concerns regarding data protection, extensive documentation, and philosophical reservations about the study design. Consequently, we recruited fewer participants than initially planned. Despite these hurdles, qualitative data collection has shown notable success. Given psychotherapists’ busy schedules and reluctance to participate, more potent external incentives or a legal obligation may be necessary to encourage participation in future studies.
1 Introduction
Psychotherapy is a universally recognised and effective strategy for treating mental disorders (1–3). Over time, an extensive body of research has emerged, focused on scrutinising psychotherapy’s impact on informing healthcare policies and refining psychotherapeutic practices (4). Most psychotherapy research focuses on randomised controlled trials (RCTs), aiming to evaluate interventions under ideal, controlled conditions. However, this approach tends to overlook the complexities of routine clinical practice, where outcomes may differ. Critics argue that RCTs operate in environments deviating significantly from real-world psychotherapy and bypass fundamental aspects of everyday therapeutic practices. For example, RCTs often exclude patients with co-morbidities, a common occurrence in routine practice, and patients in RCTs typically lack autonomy in choosing their therapists (5–7). Furthermore, RCTs tend to rely heavily on manual-guided therapy, which could limit spontaneous interventions, possibly compromising the therapeutic relationship (8).
In light of these limitations, there is growing attention towards studies conducted in real-life conditions, known as naturalistic outcome studies. These studies offer an alternative perspective to existing RTCs by tracking individuals undergoing therapy in their everyday environments and documenting their progress over time (9). They provide a grassroots method for building and applying scientific knowledge, helping bridge the gap between science and clinical practice, often cited as an “empirical imperialism” problem. This problem arises when there is a divide between clinical practice and research. In many cases, clinicians may not be involved in research or may feel that the findings of controlled studies do not apply well to their day-to-day work with patients. This disconnect can lead to a lack of valuable input from clinicians in the research process (10). Recent years have witnessed an uptick in uncontrolled naturalistic outcome studies in Germany (11), Switzerland (12), Norway (4), and Denmark (13), and one is currently underway in Brazil (14). Austria lacks representative data on psychotherapy outcomes under the conditions of outpatient psychotherapy in outpatient practice under routine care conditions.
This article introduces a study focused on the processes and outcomes of psychotherapies under clinically representative conditions in Austria. The study’s design considers the learnings from a previous unsuccessful attempt by a different research team. The former attempt, which involved exclusively behavioural therapists, failed due to inadequate resources, limited stakeholder involvement, insufficient incentives, and a prevailing belief among therapists that further practice studies were unnecessary (15). The POPP study (Process and Outcome in Private Practice) (16) detailed in this paper welcomes participation from all psychotherapeutic modalities acknowledged by Austrian psychotherapy legislation. Consequently, it encompasses therapeutic modalities from four major theoretical orientations: Humanistic therapy, Psychodynamic therapy, Systemic therapy, and Behavioural therapy. The POPP study is supported by a team of postdoctoral researchers funded by universities. It ensures a more realistic study duration and broad stakeholder involvement, thereby addressing the limitations encountered in the earlier trial. While we have built upon lessons learned, we needed to revise our initial expectations regarding the number of study participants. Through this publication, we aim to 1) describe the planned methodology and rationale of the research project and 2) explain the ongoing challenges we face in conducting a naturalistic outcome study in Austria despite incorporating lessons learned from a previous attempt.
2 Background
Evidence-based psychotherapy is fundamentally based on collections of empirically supported treatments (ESTs) (16). These provide clinicians access to treatments validated through robust research, typically through RCTs. However, these collections primarily concentrate on differential efficacy, determined by the therapeutic modality used in treatment. Despite more than half a century of research, countless RCTs, and substantial monetary investment, the effects of specific psychotherapy modalities on mental disorders have proven to be only moderate (3). The focus on the efficacy of a therapeutic modality may also skew the broader perspective, as it overlooks substantial evidence indicating that the effectiveness of treatment largely hinges on the person of the therapist (a) (17), the complex dynamics of the therapy relationship (b) (18), and on the person of the patient (c) (17, 19).
Therapist variables (a), which include technical and relational skills, responsiveness, and attentiveness to the patient’s emotional experiences, contribute significantly to the outcome (17, 20–22). Research indicates that therapists who excel at forming strong therapeutic alliances often achieve better treatment outcomes (23). The overall magnitude of therapists’ effects tends to be greater in naturalistic settings than in RCTs (24). The therapeutic alliance (b), defined by the bond between the therapist and patient, agreed therapeutic objectives, and collaborative tasks, is pivotal for patient transformation during psychotherapy (25–27). This dynamic aspect of therapy, which evolves over time, is frequently viewed as a predictor of therapeutic outcomes. Even though the therapeutic alliance’s correlation with the outcome is not too high (r=.28), various studies underscore its significance (28). In this context, the specific and common factors models of psychotherapy are worth mentioning. While the specific factors model assumes that ingredients specific to each therapy approach contribute most to therapy outcome (e.g., specific therapeutic interventions), the common factors model assumes that factors realised in all psychotherapies contribute most to the outcome (e.g., the therapeutic relationship or alliance). The specific factors model has also been named the medical model of psychotherapy, and the common factors model has been developed into the contextual model of psychotherapy (29). However, Cuijpers et al. (2019) (30) argue that no specific or common factor in psychotherapy has been sufficiently researched or theorised to be recognised as an evidence-based mechanism of change. According to research from a decade ago, patients and their psychotherapists hold disparate views concerning the therapeutic alliance and its aspects. Patients tend to attribute greater importance to factors such as support, joint engagement in the therapeutic process, and potential strain in the alliance. At the same time, therapists often concentrate on goals and tasks (31). However, this understanding of differing perspectives has been challenged and expanded by recent studies and meta-analyses. Current research indicates a strong correlation between the evaluations of the alliance by patients and therapists, particularly in adult populations. For instance, Flückiger et al. (2018) (28) provide a comprehensive meta-analytic synthesis demonstrating this alignment in adult psychotherapy, emphasising the interconnectedness of patient and therapist perspectives in forming a successful therapeutic alliance. Similarly, Laws et al. (2017) highlight the convergence in therapeutic alliance ratings between patients and therapists and its significant relation to treatment outcomes in chronic depression (32). On the other hand, several studies in naturalistic settings examining whether congruence is linked with different types of outcomes have yielded mixed results (33).
Several patient factors (c), too, have been demonstrated to predict treatment outcomes, including age, gender, ethnicity, partnership status, comorbidity, self-reported disability, and personality disorder traits, among others (17, 19, 34). While Luborsky et al.’s early literature review (1971) (35) suggested better outcomes when therapist and patient genders align, Flückiger et al.’s (2018) above-mentioned meta-analytic synthesis on the alliance in adult psychotherapy (28) concluded that individual patient characteristics, including gender, generally have a small impact on treatment outcomes compared to other factors, such as the therapeutic alliance. This suggests that while gender may play a role in psychotherapy outcomes, it is likely overshadowed by the quality of the therapeutic relationship and other elements of the therapeutic process. However, while gender might influence the outcome only minimally, there are nuanced ways gender can influence therapy. For example, a study analysing data from over 17,000 students and 200 therapists at a US university counselling centre from 1996 to 2008 found that male patients engaged in longer treatment periods with male therapists, with comparable results to their female counterparts (36). Also, a preference has been observed among female patients for same-gender therapists, particularly for sensitive topics (37, 38). A study involving over 300 patients demonstrated that some therapists, more than others, can work more effectively with one gender over another. Some psychotherapists achieved better outcomes working with female patients, while others had more success with male patients. However, for some therapists, no significant difference was observed in their effectiveness with patients of different genders (39). One Swiss study showed that female therapists’ contributions were perceived as more supportive, while male therapists were viewed as more analytical (40). Austrian research involving 1,357 patients linked risky developments in therapy primarily to male-therapist-female-patient pairings (41). Qualitative insights suggest gender influences therapist-patient dynamics, with young female therapists sometimes feeling uneasy with older male patients due to flirtation and confrontation proving more difficult in female-female pairs. Male-male pairs tend to experience early rivalries; however, once a good relationship is established, it becomes easier for male patients to discuss topics such as sexuality. The study also identifies topics that are more challenging to address in mixed-gender dyads: sexuality, desire, and other issues related to corporeality (42). Such dynamics impact the therapeutic process, highlighting the importance of understanding gender’s role in therapy. While existing research provides some understanding, it does not elucidate the reasons behind observed differences (36), neglects critical aspects of the therapy process, such as the evolving therapeutic relationship (40), or lacks depth in exploring phenomena related to different gender combinations. Combining quantitative and qualitative data in a mixed methods design is necessary to acquire a comprehensive understanding, ensuring both therapist and patient perspectives are represented (43).
3 Design and methods
3.1 Aims and research questions
The main objective of the POPP study is to evaluate the outcome and process of psychotherapies amidst the complexity and fluidity of naturalistic conditions over time at varying lengths. A distinct emphasis is placed on examining the therapeutic relationship, focusing on the collaborative and trustful bond between a therapist and a patient (the alliance), and gender dynamics. Importantly, the study also includes data from patients who end their therapy prematurely for various reasons, offering an inclusive and thorough perspective on the psychotherapeutic process. In particular, the POPP study will answer the following research questions:
1) Given an open-ended, naturalistic, and representative psychotherapy setting (outpatient psychotherapy practice), what are the rates and magnitudes of diagnostic, symptomatic, and interpersonal change (the outcomes gauged through pre- and post-therapy comparisons)?
2) How does the therapeutic alliance evolve in outpatient psychotherapy practices from the viewpoints of both patients and therapists? This includes contrasting these dual perspectives within each therapeutic orientation and potentially even within individual therapeutic modalities, provided sufficient data exists. For a discrete evaluation of a therapeutic modality, a minimum of ten patient-therapist pairs are required.
3) How does the therapeutic alliance connect to the therapy’s outcome from both the patient’s and therapist’s standpoints?
4) How are therapy outcomes influenced by therapist demographics (e.g., age, gender), professional experience, personality traits, and the congruence of therapist-patient characteristics?
5) Which interventions are perceived as beneficial or detrimental to the psychotherapy process from the perspectives of patients and therapists?
6) What impact does the gender pairing within the therapeutic dyad (female-female, male-female, male-male, female-male) have on the therapeutic alliance, therapy outcome, or premature termination of therapy?
It is crucial to clarify that the POPP study does not aim to juxtapose the efficacy of the four theoretical orientations or their numerous therapeutic modalities. Due to the inherent design of a naturalistic outcome study, there is no control group, and participants are not assigned randomly to different treatment modalities. Consequently, there might be intrinsic variations amongst individuals who opt for different therapeutic modalities. These variations impede direct comparisons (44).
3.2 Setting
The POPP study, designed to reflect the prevalent psychotherapy conditions in Austria, offers an authentic insight into the field. Austria, an early adopter of psychotherapy legislation, enacted a law in 1991 (BGBL 361/1990) defining psychotherapy as a scientifically-based treatment for behavioural disorders and psychological suffering on a psychosocial or psychosomatic basis (45). The study operates within Austria’s mental health system, featuring 23 accredited psychotherapeutic modalities categorised into four broader theoretical orientations: humanistic, psychodynamic, systemic, and behavioural. As of 2021 (46), the distribution of psychotherapists according to their theoretical orientation was as follows: 38% practised Humanistic therapy, 26% were Psychodynamic therapists, 24% practised Systemic therapy, and 12% were Behavioural therapists.
The Austrian model exemplifies an uncommonly diverse blend on a global scale and sets Austria apart from regions dominated by only a few mainstream modalities, such as psychoanalysis and behavioural therapy. Notably, this model provides an array of entry pathways and multiple training opportunities in psychotherapy (47, 48). The POPP study allows us a closer look into, and a deeper understanding of, the particular characteristics of Austria’s unique psychotherapy model.
3.3 Design
Launched in 2020, the POPP project utilises a sophisticated mixed-methods research design, synergising quantitative and qualitative methodologies. It involves a wide-ranging participant pool, encompassing patients with various and combined symptoms and undergoing treatment durations ranging from as brief as three sessions to as extended as four years. Furthermore, the study aims to incorporate these patients’ corresponding psychotherapists to align and compare their data.
The quantitative part of the study involves pre- and post-therapy online surveys taken by patients and their therapists. Both groups are also requested to assess the therapeutic alliance after each session, enabling real-time process monitoring. Therapists document their cases using a specific case report format.
The qualitative part of the study encompasses interviews with patients and their respective psychotherapists. These interviews occur randomly at several therapy stages, either at the initial, the mid-point, or the concluding phases, whereby individuals are only asked once during the process. These interviews, guided by a semi-structured interview guide, probe deeply into individual experiences.
During the analysis process, quantitative and qualitative data forms corroborate each other. The qualitative data contribute valuable insights into the psychosocial subtleties and dynamics of therapy that might potentially elude the quantitative data. The quantitative analysis will primarily scrutinise the relationship between the therapeutic alliance and therapy outcomes across different gender pairings, considering the perspectives of both patients and therapists. Furthermore, the qualitative data, in an independent capacity, will shed light on the subjective experiences of patients and therapists alike.
This research project represents a collaborative endeavour involving the University of Continuing Education Krems (UWK) and the Karl Landsteiner Private University for Health Sciences in Krems. Research approval for the study was obtained from the Data Protection Office of the University of Continuing Education Krems (UWK) as well as from the Ethics Committee of the University of Continuing Education Krems (UWK), Austria (EK GZ 28/2018-2021). The timeline of the study can be accessed in Figure 1.
Figure 1 Research procedure. The survey follows the course of psychotherapy and ends individually with its conclusion. It includes various processes with different lengths, therapeutic approaches, and symptoms.
3.4 Recruitment of participants
The success of the POPP study hinges on psychotherapists actively participating and being ready to include two to three of their patients. Psychotherapists eligible for involvement must be officially licensed and listed in the Austrian Federal Ministry for Social Affairs, Health Care, and Consumer Protection’s directory, demonstrating their qualification to provide individual outpatient psychotherapy to adult clients in their psychotherapy practice; group therapies are not incorporated in this study.
To ensure a broad and representative sample of patients, psychotherapists are instructed to consecutively invite all their patients in the nascent therapy stages between their first and third sessions to partake in the study. They should invite on a consecutive basis without bias towards the severity, type of symptoms, or personal preferences. Psychotherapists must record the reasons if an invited patient either declines participation or is not extended an invitation, thereby aiding in detecting recruitment biases.
The study aimed to register a minimum of 100 therapists and 300 patients to secure representation across all four theoretical orientations and the 23 therapeutic modalities recognised in Austria. In the spirit of collaboration and inclusive research, all professional therapeutic associations in Austria have been invited by our research team to contribute a minimum of ten psychotherapists each, to facilitate individual data evaluation of their respective therapeutic modalities.
Numerous professional psychotherapy associations, including the Austrian Federal Association for Psychotherapy (ÖBVP), assist with recruitment by disseminating information via emails, newsletters, and social media. The research team has also briefed therapists at various scientific meetings and congresses.
A central aspect of our recruitment strategy involves arranging multiple online information workshops throughout the year. These workshops aim to demystify the research design, clarify the role of the psychotherapists, and provide information about incentives. Psychotherapists who choose to participate in the study will receive a certification of five training hours per patient per year, which will count towards the obligatory 150 hours of further training over a five-year period, as mandated by Austrian law.
This study presents a unique opportunity for psychotherapists to contribute to the research community, examine and potentially improve their therapeutic practice by reflecting on their therapeutic alliance after each session, and reinforce the effectiveness of their therapeutic modalities, especially those currently lacking substantial empirical support.
Psychotherapists interested in participating in the study are encouraged to contact our research team for further information and workshop dates. Basic information about the study is also available online, hosted by the Karl Landsteiner Private University1. The dates and the link to the online info workshops are also provided via a Facebook site2.
Upon contacting the research team via the designated email address to request study materials, psychotherapists receive comprehensive informational resources in digital format or as printed materials. These resources contain thorough details about the study, handouts intended for the patients, and case report forms. Patients are then invited by their therapists to participate in the study and provided with these informational materials.
The psychotherapist’s and the patient’s materials include secure hyperlinks to the Research Electronic Data Capture (REDcap) platform (49), where they can complete quantitative surveys after granting their electronically informed consent. Patients interested in contributing to the study’s qualitative interviews can sign up through a separate online contact form.
3.5 Quantitative measures
3.5.1 Data protection and security
Once patients register via REDCap, they are asked to create a code composed of the first letter of their last name and birth date (e.g., M1790). This code, along with the name of the treating psychotherapist, is provided by the patients as well as by the therapist in each online questionnaire. Using this method, the data from different measurement points and individuals (patient/psychotherapist) can be linked, allowing for a more streamlined data management process.
3.5.2 Patient questionnaires at the beginning of psychotherapy
The research protocol includes a series of standardised questionnaires to gather information on various parameters. Demographic data such as age, gender, level of education, marital status, and occupation are collected. Additionally, data concerning the funding of the psychotherapy sessions, utilisation of psychopharmacological medications, instances of sick leave, or work incapacity due to mental disorders are collected. Another inquiry assesses patients’ awareness of the therapeutic method utilised by their psychotherapist. To assess therapy pre-post outcome the following measures are used:
Primary outcome: Positive Mental Health Scale (PMH) (50)
Secondary outcomes: Patient Health Questionnaire (PHQ-9) (51), Generalised Anxiety Disorder Scale (GAD-7) (52), and the World Health Organisation Disability Assessment Schedule (WHODAS) (53).
To explore gender aspects, the Social Roles Questionnaire (SRQ) (54) is implemented. Personality dimensions are assessed using the Big Five Inventory Short version from the Socio-Economic Panel (BFI-SOEP) (55) and the Short-Form Personality Traits Assessment (SASPD) (56). Lastly, patient expectations are examined through the Outcome Expectations Scale (OES) (57).
3.5.3 Therapists’ questionnaires at the beginning of psychotherapy
We gather demographic data from therapists and use the BFI-SOEP (55) and SRQ (54) tools to evaluate their personality dimensions and gender aspects in the context of their social roles.
3.5.4 Assessment of the helping alliance and change
The alliance is assessed after each psychotherapy session using the Helping Alliance Questionnaire (HAQ) (58–60). The HAQ includes two versions: one for patients and one for therapists. Each version consists of items that participants rate based on their perceptions of the therapeutic alliance. Eleven questions cover aspects such as the degree of agreement on therapy goals, the helpfulness of the therapy, and the level of mutual trust and respect. In the context of this study, patients and therapists are asked to complete the HAQ (59) after each psychotherapy session. This ongoing assessment allows for the tracking of changes and fluctuations in the therapeutic relationship over time and can provide insight into the role this relationship plays in the overall therapy outcome. The maximum likelihood estimation (ML) will be used to handle missing data in multilevel models. We chose the HAQ over other more frequently used instruments, such as the WAI (61), because it is better validated in German than other instruments, especially because the therapist version is also validated (60, 62). We also plan to use the HAQ as a measure of change during therapy, as it captures changes in well-being after each session compared to the start of therapy. Since the HAQ includes two different scales: (a) assessing satisfaction with the therapeutic relationship and (b) evaluating satisfaction with therapy outcomes, with a primary focus on treatment outcome satisfaction, the second scale (b) will also be used to track changes from the beginning to the end of therapy.
3.5.5 Patients’ questionnaires at the end of psychotherapy
At this stage, we utilise a range of standardised questionnaires to assess the following outcomes for patients. These encompass the PMH (50), the PHQ-9 (51), the GAD-7 (52), and the WHODAS (53). The personality dimensions are analysed using the SASPD (56). In this phase, we also investigate the therapeutic modality implemented during the psychotherapy sessions, as assessed by the Multitheoretical List of Therapeutic Interventions instrument (MULTI 30) (63). Furthermore, any negative effects are recorded using the Negative Effects Questionnaire (NEQ) (63). Periodically, we send emails to therapists, requesting them to remind their patients about completing the questionnaires during the course of therapy. We will analyse the missing data, including the number and percentage of missing data points, the number and percentage of variables with missing data, and the number and percentage of participants with missing data. We will further clarify whether the missing data is missing completely at random, missing at random, or not missing at random and whether the missing data patterns are monotone or intermittent. Based on this assessment, we will decide whether to remove cases with missing data. Full information maximum likelihood and multiple imputation are possible options for estimating missing data if the assumptions are met (e.g., missing completely at random) and are generally preferred over casewise deletion (64, 65).
3.5.6 Therapists’ questionnaires at the end of psychotherapy
To ensure comprehensive data collection, psychotherapists involved in the study complete a Case Report Form for each consecutively admitted patient until they secure two to three patients for the study. This form is submitted to the research team upon completion of the study. It includes details such as the therapist’s name, patient code, distribution and receipt of study materials, patient participation status, and reasons for non-participation, if applicable. It also records diagnoses, session dates, supervision details, treatment conclusions, reasons for non-standard treatment termination, and study discontinuation. This Case Report Form captures pertinent data during psychotherapy and streamlines the completion of the Psychotherapist Questionnaire at the end of treatment by pre-populating it with relevant data.
As the instruments used vary between psychotherapists and patients, Table 1 provides an overview delineating which tools are employed with each group.
Table 1 delineates the specific instruments employed for patients and therapists.
3.5.7 Quantitative analysis
In our study, we plan to utilise t-tests for dependent samples to assess changes in psychological distress through pre- and post-intervention comparisons. Additionally, we intend to employ multilevel models to analyse the dynamics of therapeutic relationships over multiple measurement points. Multilevel models will also be applied to explore potential predictors of therapy outcomes; both patient and therapist variables will be considered.
3.6 Qualitative measures
3.6.1 Qualitative interviews
Unlike the general study, where therapists are enrolled first, the recruitment for additional qualitative interviews initiates with patients already participating. Patients who consent to the quantitative survey can further opt to participate in a qualitative interview. They can register to be interviewed face-to-face by filling out a secure contact form on the UWK’s homepage, including their phone number, email, and county. Each time after completing their post-session questionnaire to assess the helping alliance (HAQ) (59), patients receive comprehensive information about the possibility of registering for a face-to-face interview in REDCap. Furthermore, psychotherapists involved in the study are encouraged to inform their patients about the opportunity to partake in qualitative interviews.
This study’s interview methodology delves into patients’ and therapists’ subjective impressions of the therapeutic process and the therapeutic relationship. Using a semi-structured qualitative interview approach (52), therapists and patients share their experiences by answering reflective and open-ended questions face-to-face with interviewers recruited from psychotherapy students at the UWK. The interview guide, designed for both patients and psychotherapists by BS, YS, and AJ, starts with motivations at the beginning of therapy (66). It explores the evolution of therapy goals from start to present and prompts reflection on pivotal moments and impactful dialogues within the therapeutic process. Moreover, it aims to elicit insights into what participants perceive as the most advantageous facets of therapy and probes their interpretation of the transformations and achievements they attribute to the therapy process. Notably, the guide emphasises the therapist’s role and delves into the implications of gender dynamics within the therapeutic relationship. Finally, the interview facilitates the expression of any hardships or obstacles that participants may have encountered throughout their therapy, encouraging an open dialogue about the challenges inherent in the therapeutic relationship and process.
With the therapy process’s different stages in mind, the study team has developed specific interview guides for each phase: First from the initial session to the fifth, then mid-therapy starting with the sixth session until the end of therapy and in instances of therapy discontinuation. While the patient and therapist interview guides maintain a similar thematic focus, each is uniquely tailored to capture the distinct perspective of the therapy process from the viewpoint of the patient or therapist. In this way, we aim to collect interviews from all stages of the therapy (beginning, middle, and end).
Before the interviews, patients are provided with an information form and a privacy statement, and written consent for participation is secured. This also includes the patient’s consent for their therapist to be interviewed. Upon receiving approval, the same interviewer approaches the therapist to conduct a complementary interview on the therapeutic process. These follow-up interviews ideally occur within two weeks of the patient interview to capture the same phase of the process. Therapists, similar to patients, receive a privacy statement before their interview, and their written consent for participation is obtained.
3.6.2 Qualitative analysis
Our methodology involves dividing the sample into distinct sub-samples. These groups are defined based on different stages of the therapeutic process, such as early, middle, and late stages. Additionally, we consider gender as another critical factor, dividing the sample by gender or gender pairings, which allows us to examine dynamics within same-gender dyads and mixed-gender dyads. The primary objective of our qualitative analysis is not to generalise our findings to broader populations. Instead, we focus on developing detailed maps of variation within the data. These maps aim to capture and reflect the diverse practices and experiences related to the phenomena under study (67). For instance, this approach enables us to explore variations in how patients seek therapists in private practice (66) or their views and experiences with therapists of the same or opposite gender. We also closely examine therapists’ perspectives, including their views and experiences with interventions or what patients and therapists find helpful at different stages. To extract these nuanced insights, our analysis relies on category-based methods in qualitative research. This approach is instrumental in identifying recurring patterns and themes across the data, encompassing patients, therapists, dyads or subgroups in terms of stages or gender.
3.6.3 Data collection process and data management
A crucial collaborative aspect of this project is the inclusion of psychotherapy students at the UWK in the research process with the aim to train scientist-practitioners. This will provide the students with an early-career model based on conducting scientifically rigorous and clinically relevant studies. It will help them identify clinically relevant questions later in their work (22). The UWK, offering postgraduate training in psychosocial interventions, involves their Master’s students from multiple psychotherapy programs in data collection and analysis. These students, working towards their certification as academically trained psychotherapists under the Austrian Psychotherapy Act (BGBL 361/1990), can utilise project data for their Master’s thesis in return for their involvement. These students are recruited during research seminars and via notifications to those enrolled in the University’s psychotherapy programs. Interested students are briefed about the study in an informational session, after which they sign a confidentiality agreement.
When patients participating in the quantitative survey also confirm their interest to partake in a qualitative interview, a Master’s student contacts a few patients (typically three to four) to schedule interviews at a location chosen by the patients. After the interview, they seek the patient’s consent to interview their psychotherapist.
As they conduct and transcribe the interviews independently, students receive supervision from staff members available for questions. We also offer focus groups and troubleshooting support. These interactions are essential for identifying and addressing methodological challenges and refining interview guidelines.
The gathered data is stored on a secure university drive and shared collectively with the students. Each case folder must contain consistent documents: an interviewer’s memo, an audio file, a transcript, and a signed consent form. We also keep a keyword list in an Excel file documenting the interview contents, ensuring uniformity in data format and digital document presentation. Depending on their Master’s thesis project requirements, students can create their subsets of data after submitting their collected data to the pool.
3.6.4 Focus groups
Qualitative data collection was conducted by two separate waves of students, who later engaged in focus group discussions. This process was established to oversee the research procedure and maintain the quality of the findings. These focus groups, led by two staff members, provide a platform for interviewers to discuss their insights regarding the process’s feasibility and their overall interview experiences. As of now, two waves of data collection and their corresponding focus group sessions have been completed.
4 Recruitment process
4.1 Qualitative and quantitative data
Despite our extensive efforts and incentives to motivate psychotherapists to participate in the study, the recruitment process has emerged as the most significant challenge, as can be seen in Figure 2. By now, we have acquired data from 93 therapists and 173 patients, and we are far from realising the goal of securing a minimum of 10 psychotherapists for each therapeutic modality. The study’s conclusion, initially scheduled for 2023, has been deferred to December 2024 due to the slow pace of data collection. This means we included new psychotherapists in the study until mid-2023 and collect data until the end of 2024. The therapy, of course, goes on even after the study is finished. Most psychotherapists were admitted in the first year, and the average psychotherapy time in private practice of 2-4 years was considered in our planning.
Figure 2 Overview of the recruiting process for psychotherapists.
Contrary to the difficulties experienced in the broader recruitment for the study, soliciting qualitative interviews has been noteworthy successful. Over the course of two waves carried out in the summers of 2021 and 2022, we have successfully completed 43 patient interviews and 41 psychotherapist interviews. During the initial wave, 6 students conducted interviews with 21 patients and 17 therapists. This was followed by a second wave, where an additional 8 students interviewed 22 patients and another 17 therapists. Notably, a subset of therapists was interviewed twice, with each occasion focusing on a distinct patient.
The comprehensive dataset thus compiled is particularly impressive, given that these interactions occurred amidst the constraints posed by the COVID-19 pandemic.
4.2 Difficulties with incomplete data and data management procedures
While we have collected over 3,452 process surveys from therapists until today (January 2024), patient submissions amount to 1,990. Missing process survey data from patients underscores the importance of having a simple and user-friendly process for data collection. Furthermore, we assume that a stronger commitment to the study on the part of the therapists would also have led them to regularly remind their patients to enter the data for the process survey. This happened too little.
One frequent difficulty encountered is patients’ inconsistent usage of codes when they login to complete the questionnaires compared to when the interviewers contact them via phone. This inconsistency hampers the data merging process. We partly ascribe these issues to the pathology patients grapple with. This pathology also introduced challenges for the interviewers, as voiced by one of them during a focus group:
“Occasionally, the pathology influenced the dynamics of the interviews in a bizarre way. For instance, it was unclear whether one patient was speaking in metaphors or experiencing psychosis. That was quite draining.” (researcher 3, focus group 1)
However, there have been instances where psychotherapists failed to use consistent codes as well, indicating that mental pathology cannot be the sole reason for such confusion.
4.3 Working with student researchers
Working with student researchers presented great advantages and only minor challenges due to their diverse experiences with scientific research and qualitative methodologies. While many have social or health sciences backgrounds, others have never worked in a scientific field, impacting the interview process and data quality. However, due to their psychotherapy training, these student researchers are adept at conversational techniques. They have learned to form a strong rapport with their counterparts and reflect on the information they receive. All of them were already working with patients under supervision, which positively influences the openness of interviewees and the depth of conversation. However, during focus groups, it became apparent that students often struggled to differentiate between their roles as therapeutic interviewers and researchers. They found it challenging to disengage from a therapeutic conversation and follow the interview guide.
During a focus group, one researcher expressed,
“I constantly had to remember, ‘ah, here’s my questionnaire, and I’m here as a researcher.’ Sometimes it would have been emotionally appropriate to ask questions or summarise, but then there was the INTERVIEW GUIDE. I felt as if I was juggling multiple roles at once.” (researcher 1, focus group 1)
A recurrent challenge faced by our student researchers pertains to the expansive geographical scope of participant recruitment, encompassing all regions of Austria. As a result, students often needed to travel significant distances to interview patients and psychotherapists. Despite our endeavours to align the students’ places of residence with the locations of our study participants, such coordination was not always feasible.
5 Discussion
5.1 Recruitment of participating psychotherapists
Even though Austrian law requires a scientific approach to psychotherapy, a 2018 study found Austrian therapists to be more resistant to process and outcome monitoring than their counterparts in Belgium, the UK, and Australia. These therapists expressed apprehension about complex procedures, the perceived minimal treatment benefits, and potentially detrimental effects on the therapist-patient relationship (68). As we already mentioned in the introduction, similar problems were encountered in previous psychotherapy research efforts (15).
Despite our comprehensive and strategic efforts, which encompass refining our research approach, dedicating significant human resources, and actively involving stakeholders (7) in the context of the POPP study, we persistently encounter challenges that impede our recruitment process. Throughout the series of 22 informational workshops from October 2020 to June 2023, we have identified several concerns expressed by potential participants that could discourage their involvement:
Concerns about the study have been notably focused on data protection and potential disruption of the therapeutic process. Therapists worry about guaranteeing their patients’ confidentiality and the possible negative effects of ongoing data collection, particularly post-session questionnaires, on their therapeutic relationships. They also expressed concerns about interpreting individual data and gauging their therapeutic efficacy on a personal level.
There were also concerns about the strain filling out questionnaires and process monitoring puts on patients.
The complexity of the study information, as necessitated by data protection regulations, emerged as a further barrier to participation. Therapists expressed doubts about their patients’ capacity to understand such intricate details. Furthermore, the study’s complex design was occasionally misconstrued as a randomised controlled trial (RCT), evoking apprehensions about reducing psychotherapeutic processes to symptom-oriented constructs. Particularly, therapists from humanistic modalities emphasised their focus on less tangible aspects like personality development and enhancement of self-perception, contending that these elements could not be accurately captured through standardised questionnaires. In this vein, previous research on the attitudes of Austrian psychotherapists on process and outcome monitoring has found that cognitive-behavioural therapists generally held favourable views; therapists employing humanistic-existential modalities, on the other hand, expressed negative attitudes (68). This attitude is anchored in broader, epistemological objections to psychology’s characterisation as a quantitative science and its aspiration to produce “objective” explanations for psychological phenomena (69). Despite these findings, however, most participating psychotherapists belong to humanistic psychotherapy modalities.
Despite our consistent efforts during the workshops to clarify that our study did not aim to compare the effectiveness of various therapeutic modalities, we probably could not alleviate these concerns sufficiently. Given Austria’s endorsement of 23 psychotherapy modalities and the upcoming revision of the Psychotherapists Act, there is a prevalent sense of unease about potential regulatory changes. Particularly, smaller professional associations expressed fears that their modalities may not be deemed sufficient, thus jeopardising their future inclusion. This, coupled with a perceived sense of inferiority against larger, mainstream approaches such as behavioural therapy, likely dissuades therapists from participating in research more broadly.
We think the efforts to explain our study design during workshops were largely unsuccessful due to a widespread unfamiliarity with the concept of scientific research. As pointed out by Castonguay et al. (2014), many clinicians had not had the opportunity to work with a supervisor who was both a practising psychotherapist and a researcher. This gap in early career training means they lacked exposure to a role model who could identify the most appropriate research methods to answer the most relevant research questions (10).
The intensive documentation needed for participation was another deterrent, in line with findings from Laireiter and Kopetzky (2014) (15).
Some therapists expressed challenges in persuading new patients to join the study. They felt it was inappropriate to ask patients to contribute additional effort for the sake of psychotherapy, especially since the patients were already paying for the therapists’ services. However, therapists who firmly believed in the study’s value reported facing no issues enlisting patients. This indicates that the therapists’ personal hesitations might serve as a barrier to motivating patient involvement in the study.
Qualitative data collection proceeded more smoothly than quantitative data, indicating less therapist resistance. This inference comes from the noticeable increase in qualitative interview registration following email communication urging therapists to encourage their patients to participate in qualitative interviews.
Finally, unlike many therapists, patients probably did not share the same apprehensions concerning data protection, the complexities of the study’s framework, or the affiliated duties. Additionally, they value the distinctive introspective possibility offered by these personal interviews. Almost unanimously, every patient approached by our student interviewers for a deeper explanation regarding qualitative interview participation (after registration for the qualitative survey via RedCap) agreed to participate, with only a few exceptions.
5.2 Data management procedures
The issue with patients and therapists using different codes when logging in and during interviews indicates a need for clearer communication and an easier procedure for creating a patient code.
5.3 Working with student researchers
The challenges interviewers face, as reflected in the focus group sessions, highlight the need for support systems for the interview team. Handling challenging situations, such as deciphering whether a patient is speaking in metaphors or experiencing psychosis, can be draining. Since our student interviewers are being trained as future psychotherapists, they did handle such situations well. Moreover, since we provided regular sessions for reflection and support via email and telephone, they could address pressing issues immediately.
From the outset, the project prioritised data quality, aiming to build a robust database to facilitate a wide array of future research inquiries. Several quality assurance strategies have been implemented to ensure this high-quality data, including:
● Providing methodological training and support materials for student interviewers.
● Initiating early data documentation using templates to capture the extensive research context surrounding the interview situation.
● Ensuring opportunities for interviewer reflexivity.
Standardisation of the management of qualitative data proved to be advantageous. Given the complexity of a longitudinal mixed-methods dataset, understanding the data structure, familiarising oneself with documentation routines, and tracking documents for deposition and their scheduled dates can be challenging. We thus recommend having at least one person oversee data management, processing, and archiving (70).
5.4 Limitations
Certain limitations should be acknowledged. Despite comprehensive efforts to enhance the feasibility of the POPP study, certain limitations impacting the recruitment process persist. The resistance among Austrian psychotherapists, influenced by concerns about data protection, potential disruption of the therapeutic process, and the strain on patients, poses a significant feasibility challenge. The complexity of study information, particularly related to data protection regulations, further hampers participant understanding and engagement. Efforts to clarify the study design during workshops faced difficulties due to widespread unfamiliarity with scientific research concepts among therapists. The study’s sample was comprised only of therapists interested in the study, thus limiting its generalizability. Although therapists were advised to inform all consecutive patients about the study, adherence to this protocol couldn’t be verified. Hence, the possibility of therapists inviting only a specific subset of patients, such as those with less severe illnesses, cannot be discounted, which could potentially limit patient-side generalizability. Further limitations include missing data, especially among patients. While the baseline assessments experienced the least amount of missing data, there was a noticeable increase in missing data in the session-based questionnaires and those conducted post-therapy. Although we periodically send emails to therapists, requesting them to remind their patients about completing the questionnaires during the course of therapy, a significant amount of data will be missing, which needs to be handled appropriately based on the final percentage of missing data as well as the distribution of missing data. Possible inaccuracies due to the use of erroneous codes by patients or therapists represent an additional limitation, which could affect the interpretation of study results.
Another limitation regards the collection of qualitative interviews. Our objective is to gather qualitative interviews from all stages of therapy (beginning, middle, and end). However, realising this goal presents certain challenges. A key limitation is the unpredictability of each patient’s therapy duration. This uncertainty is evident when conducting interviews, as we cannot foresee the total length of the therapy process for each patient. Additionally, the process of conducting interviews is time-intensive, and given our limited personnel resources, it becomes impractical to accurately manage the distribution of interviews across different therapy phases. Therefore, these factors combined hinder our ability to evenly distribute the collection of interviews throughout the various stages of therapy. Consequently, we observe a predominance of interviews originating from the middle phases of therapy. This imbalance restricts our ability to compare experiential patterns characteristic of early therapy stages, late stages and processes with premature termination, limiting the scope of our analysis.
A notable limitation arises in the collection of consistent outcome measurements during the practice study, primarily stemming from the challenges posed by varying start times and the decentralised locations of patients. The nature of real-world practice settings introduces a level of unpredictability, making it difficult to ensure uniformity in the timing of outcome measurements across different treatment trajectories.
6 Conclusion
Despite employing various strategies and incentives to engage psychotherapists in the POPP study, participant recruitment has proven to be a significant challenge. The slower-than-anticipated data collection rate has led us to extend the study’s deadline to December 2024. The reluctance of Austrian psychotherapists to participate can be attributed to practical concerns, such as data security and extensive documentation, as well as philosophical disagreements with the study’s design and quantitative research methods. The above problems suggest that many therapists are unfamiliar with up-to-date methodological and statistical advances because they are not trained in research. Revision of the Austrian law for psychotherapy could change that.
Given the demanding schedules of psychotherapists, dedicating sufficient time to meet the requirements of a naturalistic outcome study is daunting. Although it was posited that Austria’s therapeutic diversity could be maintained only if each of its 23 therapeutic modalities substantiates its effectiveness, most modalities have not motivated at least ten therapists to participate. This suggests that more robust external motivators are needed to increase participation in future nationwide studies evaluating psychotherapy’s effectiveness. Implementing a legal mandate for participation in scientific studies could significantly augment Austria’s psychotherapy research data. Another approach could involve exposing young psychotherapists in training to research during their studies to foster enthusiasm and engagement in research. Enriching their scientific training in conjunction with patient work may provide additional benefits.
Ultimately, examining outpatient psychotherapy within the context of a therapist’s practice confronts many challenges, primarily attributable to the restricted control over participants compared to clinical trials. Given these conditions, therapists’ roles transform into critical conduits for disseminating information about the study to patients, a role typically played by the hosting institution in a conventional clinical trial setting. This dynamic might provide a credible rationale for the paucity of such essential studies in the field. Despite these complexities, we maintain optimism that the results will contribute to a deeper understanding of the processes and outcomes of psychotherapies under the condition of routine outpatient care in Austria.
Ethics statement
The studies involving humans were approved by Ethics Committee of the University of Continuing Education Krems (UWK), Austria (EK GZ 28/2018-2021). The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
Author contributions
YS: Conceptualization, Data curation, Writing – original draft, Investigation, Writing – review & editing. AJ: Conceptualization, Data curation, Writing – original draft, Writing – review & editing. EH: Data curation, Investigation, Methodology, Project administration, Writing – review & editing. KH: Visualization, Writing – review & editing. CP: Writing – review & editing. TP: Methodology, Writing – review & editing. BS: Methodology, Visualization, Writing – original draft, Writing – review & editing.
Funding
The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article. We did not receive any funding for this project.
Acknowledgments
The authors thank the Austrian Federal Association for Psychotherapy (ÖBVP) for inviting their members to participate in the surveys. Further thanks deserve the participating psychotherapists and patients for their time and efforts in participating in this extensive study.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Footnotes
- ^ https://opencampus.kl.ac.at/de/system/files/Pr%C3%A4sentation_POPP%20f%C3%BCr%20Homepage.pdf
- ^ https://www.facebook.com/POPPStudie
References
1. Correll C, Carbon M. Efficacy of pharmacologic and psychotherapeutic interventions in psychiatry to talk or to prescribe: is that the question? JAMA Psychiatry. (2014) 71:624–6. doi: 10.1001/jamapsychiatry.2014.288
2. Huhn M, Tardy M, Spineli LM, Kissling W, Förstl H, Pitschel-Walz G, et al. Efficacy of pharmacotherapy and psychotherapy for adult psychiatric disorders: A systematic overview of meta-analyses. JAMA Psychiatry. (2014) 71:706–15. doi: 10.1001/jamapsychiatry.2014.112
3. Leichsenring F, Steinert C, Rabung S, Ioannidis JPA. The efficacy of psychotherapies and pharmacotherapies for mental disorders in adults: an umbrella review and meta-analytic evaluation of recent meta-analyses. World Psychiatry. (2022) 21:133–45. doi: 10.1002/wps.20941
4. Nordmo M, Sønderland NM, Havik OE, Eilertsen DE, Monsen JT, Solbakken OA. Effectiveness of open-ended psychotherapy under clinically representative conditions. Front Psychiatry. (2020) 11:384. doi: 10.3389/fpsyt.2020.00384
5. Seligman MEP. The effectiveness of psychotherapy: The consumer reports study. Am Psychol. (1995) 50:965–74. doi: 10.1037/0003-066X.50.12.965
6. Philips B, Falkenström F. What research evidence is valid for psychotherapy research? Front Psychiatry. (2021) 11:625380. doi: 10.3389/fpsyt.2020.625380
7. Shadish WR, Navarro AM, Crits-Christoph P, Jorm AF, Nietzel MT, Robinson L, et al. Evidence that therapy works in clinically representative conditions. J Consult Clin Psychol. (1997) 65:355–65. doi: 10.1037/0022-006X.65.3.355
8. Tschuschke V, Crameri A, Koemeda M, Schultess P, Von WA, Weber R. Fundamental reflections on psychotherapy research and initial results of the naturalistic psychotherapy study on outpatient treatment in Switzerland (PAP-S). Int J Psychother. (2010) 14:23–35.
9. Sales C, Goncalves S, Fragoeiro A, Noronha S, Elliott R. Psychotherapists openness to routine naturalistic idiographic research? Ment Health Learn Disabil Res Pract. (2007) 4:145–61. doi: 10.5920/mhldrp.2007.42145
10. Castonguay LG, Barkham M, Lutz W, McAleavey AA. “Practice-oriented research: approaches and applications.,”. In: Bergin and garfield’s handbook of psychotherapy and behavior change, Sixth Edition. Wiley, Hoboken (2014). p. 85–133.
11. Strauss BM, Lutz W, Steffanowski A, Wittmann WW, Boehnke JR, Rubel J, et al. Benefits and challenges in practice-oriented psychotherapy research in Germany: The TK and the QS-PSY-BAY projects of quality assurance in outpatient psychotherapy. Psychother Res. (2015) 25:32–51. doi: 10.1080/10503307.2013.856046
12. Crameri A, Koemeda M, Tschuschke V, Schulthess P, von Wyl A. Ergebnisqualität ambulanter Psychotherapie : Ergebnisse aus der Grundversorgung in der Schweiz. Psychotherapie-Wissenschaft. (2014) 4:96–107.
13. Lauritzen LR, Faye Jacobsen C, Nielsen J, Lunn S, Mathiesen BB, Falkenström F, et al. Common factors, Responsiveness and Outcome in Psychotherapy (CROP): study protocol for a naturalistic prospective cohort study of psychotherapy in Denmark. BMJ Open. (2023) 13:e072277. doi: 10.1136/bmjopen-2023-072277
14. Gonçalves L, Barbisan GK, Rebouças CDAV, Da Rocha NS. Longitudinal investigation of psychotherapy outcomes (LIPO): Description of the study protocol. Front Psychiatry. (2019) 10:212. doi: 10.3389/fpsyt.2019.00212
15. Laireiter A-R, Kopetzky U. Die Wirksamkeit von Psychotherapie in der Praxis. Erkenntnisse aus einer missglückten Praxisstudie. In: Riess G, editor. Praxisorientierte Psychotherapieforschung. Verfahrensübergreifende, patientenorientierte Aspekte und Kompetenzentwicklung. Tagungsband zur wissenschaftlichen Tagung 2014. Österreichisches Bundesinstitut für Gesundheitswesen (ÖBIG, Wien (2014). p. 14–22.
16. Woody SR, Sanderson WC. Manuals for empirically supported treatments: 1998 update. Clin Psychol Rev. (1998) 51:17–21.
17. Lutz W, de Jong. K, Rubel. JA, Delgadillo J. “Measuring, predicting, and tracking change in psychotherapy.,”, in: Bergin and Garfield’s handbook of psychotherapy and behavior change (2021). John Wiley & Sons. Available online at: https://psycnet.apa.org/record/2021-81510-004 (Accessed January 17, 2024).
18. Norcross JC, Lambert MJ. Evidence-based psychotherapy relationship: the third task force. Psychother Relat that Work. (2019) 1:1–23. doi: 10.1093/MED-PSYCH/9780190843953.003.0001
19. Joutsenniemi K, Laaksonen MA, Knekt P, Haaramo P, Lindfors O. Prediction of the outcome of short- and long-term psychotherapy based on socio-demographic factors. J Affect Disord. (2012) 141:331–42. doi: 10.1016/J.JAD.2012.03.027
20. Beutler LE, Malik ML, Alimohamed S, Harwood TM, Talebi H, Noble S, et al. “Therapist variables.,”. In: Lambert MJ, editor. Bergin and garfield’s handbook of psychotherapy and behavior change. Wiley, New York (2004). p. 227–306.
21. Clarkin JF, Levy KN. “The influence of client variables on psychotherapy. .,”. In: Lambert MJ, editor. Bergin and Garfield’s handbook of psychotherapy and behavior change. New York: Wiley (2004). p. 194–226.
22. Castonguay L, Hill CE. How and why are some therapists better than others? Understanding therapist effects. Castonguay LG, Hill CE, editors. Washington, DC: American Psychological Association (2017).
23. Del Re AC, Flückiger C, Horvath AO, Wampold BE. Examining therapist effects in the alliance–outcome relationship: A multilevel meta-analysis. J Consult Clin Psychol. (2021) 89:371–8. doi: 10.1037/ccp0000637
24. Wampold BE, Owen J. “Therapist effects: History, methods, magnitude, and characteristics of effective therapists.,”, in: Bergin and Garfield’s handbook of psychotherapy and behavior change (2021). John Wiley & Sons. Available online at: https://psycnet.apa.org/record/2021-81510-009 (Accessed January 17, 2024).
25. Horvath AO, Del Re AC, Flückiger C, Symonds D. Alliance in individual psychotherapy. Psychotherapy. (2011) 48:9–16. doi: 10.1037/a0022186
26. Martin DJ, Garske JP, Davis MK. Relation of the therapeutic alliance with outcome and other variables: A meta-analytic review. J Consult Clin Psychol. (2000) 68:438–50.
27. Bordin ES. The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory Res Pract. (1979) 16:252–60.
28. Flückiger C, Del Re AC, Wampold BE, Horvath AO. The alliance in adult psychotherapy: A meta-analytic synthesis. Psychotherapy. (2018) 1:8050. doi: 10.1037/pst0000172
29. Wampold B, Imel ZE. The great psychotherapy debate. The evidence for what makes psychotherapy work. 2nd ed. New York and London: Routledge (2015). doi: 10.4324/9780203582015
30. Cuijpers P, Reijnders M, Huibers MJH. The role of common factors in psychotherapy outcomes. Annu Rev Clin Psychol. (2019) 15:207–31. doi: 10.1146/ANNUREV-CLINPSY-050718-095424
31. Bachelor A. Clients’ and therapists’ Views of the therapeutic alliance: similarities, differences and relationship to therapy outcome. Clin Psychol Psychother. (2013) 20:118–35. doi: 10.1002/cpp.792
32. Laws HB, Constantino MJ, Sayer AG, Klein DN, Kocsis JH, Manber R, et al. Convergence in patient–therapist therapeutic alliance ratings and its relation to outcome in chronic depression treatment. Psychother Res. (2017) 27:410–24. doi: 10.1080/10503307.2015.1114687
33. Castonguay L, Barkham M, Youn SL, Page AC. “Practice-based evidence—Findings from routine clinical settings,” in: Bergin and Garfield’s handbook of psychotherapy and behavior change (2021). Available online at: https://psycnet.apa.org/record/2021-81510-006 (Accessed January 17, 2024).
34. Gergov V, Prevendar T, Vousoura E, Ulberg R, Dahl HSJ, Feller C, et al. Sociodemographic predictors and moderators of treatment outcomes of psychotherapeutic interventions for young people with mental disorders: A systematic review. Adolesc Res Rev. (2023) 1:1–23. doi: 10.1007/S40894-023-00228-6/METRICS
35. Luborsky L, Auerbach A. H., Chandler M., Cohen J., Bachrach H. M. Factors influencing the outcome of psychotherapy: A review of quantitative research. Psychol Bull. (1971) 75:145–85. doi: 10.1037/h0030480
36. Lambert MJ. Does client-therapist gender matching influence therapy course or outcome in psychotherapy? Evidence based Med Pract. (2016) 2:108. doi: 10.4172/2471-9919.1000108
37. Fowler WE, Wagner WG. Preference for and comfort with male versus female counselors among sexually abused girls in individual treatment. J Couns Psychol. (1993) 40:65–72. doi: 10.1037/0022-0167.40.1.65
38. Jones EE, Zoppel CL. Impact of client and therapist gender on psychotherapy process and outcome. J Consult Clin Psychol. (1982) 50:259–72. doi: 10.1037/0022-006X.50.2.259
39. Owen J, Wong YJ, Rodolfa E. Empirical search for psychotherapists’ gender competence in psychotherapy. Psychotherapy. (2009) 46:448–58. doi: 10.1037/a0017958
40. von Wyl A, Crameri A, Koemeda-Lutz M, Tschuschke V, Schulthess P. Praxisstudie ambulante Psychotherapie Schweiz (PAP-S): ein spezifischer Blick auf den Therapieprozess und Genderfragen. Psychotherapie Forum. (2021) 1–2. doi: 10.1007/s00729-021-00175-4
41. Leitner A, Märtens M, Koschier A, Gerlich K, Liegl G, Hinterwallner H, et al. Patients’ perceptions of risky developments during psychotherapy. J Contemp Psychother. (2013) 43:95–105. doi: 10.1007/s10879-012-9215-7
42. Schigl B. Psychotherapie und Gender. Konzepte. Forschung. Praxis. Welche Rolle spielt die Geschlechtszugehörigkeit im therapeutischen Prozess. Wiesbaden: Springer Fachmedien (2018).
43. Schigl B. Gender als notwendige Perspektive psychotherapeutischen Handelns. Psychotherapie Forum. (2019) 23:18–24. doi: 10.1007/s00729-019-0109-y
44. Lambert MJ. Bergin and garfield’s handbook of psychotherapy and behavior change. Sixth Edition. Lambert MJ, editor. Hoboken: Wiley (2013).
45. Heidegger KE. Psychotherapy in Austria - European association for psychotherapy. Available online at: https://www.europsyche.org/situation-of-psychotherapy-in-various-countries/Austria/ (Accessed April 1, 2023).
46. Probst T, Haid B, Schimböck W, Reisinger A, Gasser M, Eichberger-Heckmann H, et al. Therapeutic interventions in in-person and remote psychotherapy: Survey with psychotherapists and patients experiencing in-person and remote psychotherapy during COVID-19. Clin Psychol Psychother. (2021) 28:988–1000. doi: 10.1002/CPP.2553
47. Löffler-Stastka H, Gelo O, Pleschberger I, Hartmann A, Orlinsky DE, Rønnestad MH, et al. Psychotherapie-Berufsausbildung in Österreich: Basis- und soziodemographische Hintergrunddaten aus einer SPRISTAD-Pilotstudie (Society of Psychotherapy Research Interest Section on Therapist Training and Development). Z Psychosom Med Psychother. (2019) 65:341–52. doi: 10.13109/zptm.2019.65.4.341
48. Löffler-Stastka H, Gelo O, Pleschberger I, Schröder T, Orlinsky DE, Rønnestad MH, et al. Psychotherapieausbildung in österreich. Psychopraxis Neuropraxis. (2018) 21:227–31. doi: 10.1007/s00739-018-0508-9
49. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, Neal LO, et al. The REDCap consortium: building an international community of software platform partners. J BioMed Inform. (2019) 95:1–24. doi: 10.1016/j.jbi.2019.103208.The
50. Lukat J, Margraf J, Lutz R, Der Veld WM, Becker ES. Psychometric properties of the positive mental health scale (PMH-scale). BMC Psychol. (2016) 4:1–14. doi: 10.1186/s40359-016-0111-x
51. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med. (2001) 16:606–13. doi: 10.1046/j.1525-1497.2001.016009606.x
52. Spitzer RL, Kroenke K, Williams JW, Löwe B. A brief measure for assessing generalized anxiety disorder: The GAD-7. Arch Intern Med. (2006) 166:1092–7. doi: 10.1001/archinte.166.10.1092
53. Üstün TB, Chatterji S, Kostanjsek N, Rehm J, Kennedy C, Epping-Jordan J, et al. Developing the world health organization disability assessment schedule 2.0. Bull World Health Organ. (2010) 88:815–23. doi: 10.2471/BLT.09.067231
54. Baber KM, Tucker CJ. The social roles questionnaire: A new approach to measuring attitudes toward gender. Sex Roles. (2006) 54:459–67. doi: 10.1007/s11199-006-9018-y
55. Schupp J, Gerlitz J-Y. Big five inventory-SOEP (BFI-S). Zusammenstellung sozialwissenschaftlicher Items und Skalen (ZIS). (2008). doi: 10.6102/zis54 (Acessed June 28, 2023).
56. Moran PA, Olajide K, Munjiza J, Moran P, Connell LO, Newton-howes G, et al. Development and psychometric properties of the standardized assessment of severity of personality disorder ( SASPD ). J Pers Disord. (2017) 32(1):44–56. doi: 10.1521/pedi_2017_31_285
57. Vîslă A, Constantino MJ, Newkirk K, Ogrodniczuk JS, Söchting I. The relation between outcome expectation, therapeutic alliance, and outcome among depressed patients in group cognitive-behavioral therapy. J Soc Psychother Res. (2016) 28:446–56. doi: 10.1080/10503307.2016.1218089
58. Hatcher RL, Gillaspy JA. Development and validation of a revised short version of the Working Alliance Inventory. Psychother Res. (2006) 16:12–25. doi: 10.1080/10503300500352500
59. Luborsky L, Barber JP, Siqueland L, Johnson S, Najavits LM, Frank A, et al. The revised helping alliance questionnaire (HAq-II) : psychometric properties. J Psychother Pract Res. (1996) 5:260.
60. Eich HS, Kriston L, Schramm E, Bailer J. The German version of the helping alliance questionnaire: Psychometric properties in patients with persistent depressive disorder. BMC Psychiatry. (2018) 18:107. doi: 10.1186/s12888-018-1697-8
61. Flückiger C, Del AC, Wampold BE, Horvath AO. The alliance in adult psychotherapy: A meta-analytic synthesis. Psychotherapy. (2018) 55:316–40. doi: 10.1037/pst0000172
62. Nübling R, Kraft M, Henn J, Kriz D, Lutz W, Schmidt J, et al. Testing the psychometric properties of the helping alliance questionnaire (HAQ) in different health care settings. PPmP Psychotherapie Psychosomatik Medizinische Psychol. (2017) 67:465–76. doi: 10.1055/S-0043-111083/ID/R0826-0039/BIB
63. Solomonov N, McCarthy KS, Gorman BS, Barber JP. The multitheoretical list of therapeutic interventions - 30 items (MULTI-30). Psychother Res. (2019) 29:565–80. doi: 10.1080/10503307.2017.1422216
64. Enders CK. Applied missing data analysis (2010). Guilford Press. Available online at: https://psycnet.apa.org/record/2010-13190-000 (Accessed January 17, 2024).
65. Schafer JL, Graham JW. Missing data: Our view of the state of the art. Psychol Methods. (2002) 7:147–77. doi: 10.1037/1082-989X.7.2.147
66. Schaffler Y, Probst T, Jesser A, Humer E, Pieh C, Stippl P, et al. Perceived barriers and facilitators to psychotherapy utilisation and how they relate to patient’s psychotherapeutic goals. Healthcare. (2022) 10:2228. doi: 10.3390/healthcare10112228
67. Levitt HM. Qualitative generalization, not to the population but to the phenomenon: reconceptualizing variation in qualitative research. Qual Psychol. (2021) 8:95–110. doi: 10.1037/qup0000184
68. Kaiser T, Schmutzhart L, Laireiter AR. Attitudes of Austrian psychotherapists towards process and outcome monitoring. Administration Policy Ment Health Ment Health Serv Res. (2018) 45:765–79. doi: 10.1007/s10488-018-0862-1
69. Barry KL. “Chapter 6 –brief humanistic and existential therapies.,”, in: Treatment improvement protocol (TIP) series (1999). Substance Abuse and Mental Health Services Administration (US). Available online at: https://www.ncbi.nlm.nih.gov/books/NBK64939/ (Accessed June 28, 2023).
Keywords: psychotherapy, longitudinal study, gender, therapeutic relationship, helping alliance, naturalistic outcome study, outpatient practice, psychotherapy effectiveness
Citation: Schaffler Y, Jesser A, Humer E, Haider K, Pieh C, Probst T and Schigl B (2024) Process and outcome of outpatient psychotherapies under clinically representative conditions in Austria: protocol and feasibility of an ongoing study. Front. Psychiatry 15:1264039. doi: 10.3389/fpsyt.2024.1264039
Received: 20 July 2023; Accepted: 16 February 2024;
Published: 06 March 2024.
Edited by:
Brian Littlechild, University of Hertfordshire, United KingdomReviewed by:
Paola Arnaboldi, Independent researcher, Lugano, SwitzerlandPhil Mollon, Institute of Psychoanalysis, United Kingdom
Vera Gergov, The Finnish Psychological Association, Finland
Copyright © 2024 Schaffler, Jesser, Humer, Haider, Pieh, Probst and Schigl. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Yvonne Schaffler, yvonne.schaffler@donau-uni.ac.at
†These authors share senior authorship