Counseling and psychotherapy researchers have found two important results related to the working alliance between client and therapist. First, researchers using meta-analysis have established an overall effect size of approximately 0.30 between scores on working alliance and outcome measures (Horvath and Symonds, 1991; Martin et al., 2000; Horvath and Bedi, 2002; Del Re et al., 2012; Flückiger et al., 2012, 2018). A reasonable interpretation of these consistent findings is that a correlational and possible causal relation exists between alliance and psychotherapy outcome. Second, client-completed alliance measures frequently evidence a ceiling effect early in therapy, suggesting that many clients quickly develop a strong affiliation with their therapist (Tryon et al., 2007; Hall et al., 2012; Reese et al., 2013; Babatunde et al., 2017; Paap et al., 2019; Meier and Feeley, 2021). Ceiling effects refer to a set of scores clustering toward the top of the range for an item, subscale, or total scale score.

Because traditional test development procedures attempt to identify and eliminate items whose score distributions are skewed, ceiling effects are relatively uncommon in psychological tests. The use of Likert response formats with more than 2 response options typically leads to item scores with higher reliability estimates, item-total correlations, and factor loadings (Maul, 2017) as well as fewer ceiling or floor effects. Developers of alliance measures, including the frequently employed Working Alliance Inventory (WAI; Horvath and Greenberg, 1989), followed traditional test construction procedures designed to maximize variability in item response and subsequent aggregated scale scores (Meier and Feeley, 2021). Given the procedures employed in creating alliance measures, it remains unclear why ceiling effects are commonly observed on client-rated alliance measures.

Some researchers favor measurement problems as the chief explanation for ceiling effects on alliance scales (Paap et al., 2019; Baldwin and Goldberg, 2021). Describing the effects of restricted range on alliance-outcome correlations, Baldwin and Goldberg (2021, p. 28) wrote that “we believe that alliance measures may not be sufficiently sensitive to discriminate variability in the alliance.” Paap et al. (2019) also favored a technical explanation, suggesting that ceiling effects result from a failure to construct a comprehensive response format for test items as well as client characteristics such as socially desirable responding.

On the other hand, ceiling effects on client-rated alliance measures may be a consequence of theoretical factors related to the development of the therapeutic relationship (Meier and Feeley, 2021). Using data from two previously published meta-analyses, Meier and Feeley produced 92 estimates of ceiling effects based on 37 studies with 6,439 participants. They found moderate to large ceiling effects across multiple measures of client-rated alliance (e.g., the WAI and SRS) as well as time of administration. While the working alliance has typically been defined in terms of theoretical content such as tasks, goals, and bond (Bordin, 1976, 1994), Meier and Feeley concluded that for many clients, alliance development is not gradual or incremental, but occurs abruptly, non-linearly, and early in therapy. Meier and Feeley suggested that a key element is a threshold structure where clients shift to an experience of the therapeutic relationship as established.

Given that many clients may wish to offer positive feedback to their therapist on alliance measures, social desirability, where individuals provide desired responses in relation to the assessment purpose, is an individual differences variable of interest. Research conducted by Reese et al. (2013) and Sturgiss et al. (2019), however, failed to find significant correlations between measures of social desirability and alliance. In contrast, Osborne and Blanchard (2011) studied the effect of random responding on outcome measures used to evaluate an educational intervention. They created a simple random responding scale with items that should be answered in a particular direction by 0 or 100% of the respondents. Deviations from the expected response allowed identification of random responders, who presumably lacked motivation to provide valid responses. Osborne and Blanchard found that random responders evidenced (a) lower scores on study tests and (b) failed to improve on scores intended to assess change from pre- to post-test after an educational intervention intended to improve student learning and retention. Non-random responders had higher scores and did show growth on pre-post scores.

This study sought to replicate ceiling effect findings on alliance measures using a diverse sample and methodological conditions. To extend previous findings with ceiling effects, an archival dataset was sought for this study with a sample different from that typically employed in alliance-outcome studies; Meier and Feeley (2021) noted that one of the limitations of their review was that study samples consisted primarily of female, younger, and Caucasian clients. In addition, the archival data will be explored for patterns of item responses that might reflect one or more individual differences variables that could potentially explain ceiling effects on alliance scores (Osborne and Blanchard, 2011; Reese et al., 2013; Sturgiss et al., 2019).

Table 1 displays descriptive statistics, scale score range, coefficient alpha, and intercorrelations among all measures. Coefficient alpha for all scales exceeds 0.80, indicating acceptable internal consistency. Convergent validity for the alliance and expectations measures, respectively, is supported by the 0.60 correlation between WA-CO and WA-PO scores and the 0.63 correlation between EXP-CO and EXP-PO scores. Correlations between the alliance and expectations ratings for parole officers (r = −72) and for counselors (r = −0.63), however, exceed the correlation between alliance ratings of parole officers and counselors (r = 0.60) or expectations ratings of parole officers and counselors (r = 0.63). This suggests that the scale target (i.e., parole officer or counselor) influences scores more than alliance or expectations content per se.

Correlations in Table 1 indicate that increasing age (but not grade) is positively associated with alliance and expectations ratings for counselors and parole officers. The DUPS has the highest correlation with age, such that older respondents report more concerns with drug problems. NAS scores evidence no statistically significant correlations except with DUPS, indicating that more intense negative affect is associated with an assessment of greater drug problems. SAS’s General Linear Models procedure, useful with unequal sample sizes, was employed to compare gender and race (i.e., Black and Caucasian participants, the largest racial groups), the independent variables, on the six scale scores, the dependent variables. Participants with missing data were not included in the statistical analyses. For gender, no statistically significant results were found. Regarding race, statistically significant results were found for DUPS [F_(1,415) = 31.81, p < 0.001] and NAS [F_{(1, 419)} = 5.35, p < 0.03], but not for WAI-CO, WAI-PO, EXP-CO, or EXP-PO. Scores for Caucasian respondents were higher for DUPS and NA.

This study replicated previous research that found evidence for ceiling effects on measures of working alliance where scores clustered near the top (positive) of the scale. Mean scores of the alliance with both counselors and parole officers were close to 1 SD unit near the top of the respective scales. In contrast, a normal distribution was apparent for scores on a measure of drug use problems, while scores on a negative affect scale suggested a slight floor effect. This positive clustering of scores on alliance measures resulted from ratings made by mandated, not self-referred, clients. This replication is strengthened by the finding of a floor effect on a reversed score expectations measure, indicating that positive evaluations of counselors and parole officers were not simply a result of the valence of item wording and an acquiescence bias.

Investigation of individual differences in item responding found evidence for response consistency where respondents endorsed the same response from the 5-point response format across all items on the scale. On the WA-CO and WA-PO, for example, 44 and 46% of all respondents, respectively, endorsed the same response across the 7-item scales. If this consistent responding resulted from respondents’ lack of motivation, this item response behavior should have been spread equally across response options 1, 2, 3, 4, and 5. The nodes of consistent responding, however, occurred more frequently on response options toward the positive end of the scale for WA-CO, WA-PO, EXP-CO, and EXP-PO. That is, consistent responding occurred in such a manner that provided high appraisals, reflected in total scores, for counselors and parole officers.

Individuals who responded consistently to items across scales comprised 30% of WA-CO/WA-PO scores, 2% of DUPS/NAS scores, and 36% of EXP-CO and EXP-PO scores. Thus, consistent responding appeared more frequently on scales rating counselors and parole officers, but less so on the drug use and negative affect scales. This finding suggests that consistent responding is not a result of an individual differences variable, but depends upon the specific scale. Additionally, correlations between alliance and expectations ratings for parole officers and counselors exceeded the correlation between alliance ratings for parole officers and counselors, indicating that the scale target (i.e., parole officer or counselor) influenced scores more than alliance or expectations content.

While this study’s results argue against respondents as unmotivated or random responders, social desirability remains a possible, but less likely explanation. Some respondents may have had doubts about the confidentiality of their responses, for example, believing that their scores may influence the parole process. If social desirability is an individual differences variable reflected by consistent responding, it should have been present across all scales. Consistent responding was absent, however, for DUPS and NAS. Study findings are more compatible with a theoretical explanation that the alliance is primarily a holistic experience for many clients. Bordin’s (1976, 1994) model of the working alliance implies that clients consider tasks, goals, and bond when assessing an alliance with a particular therapist. The holistic hypothesis, in contrast, suggests that alliance development can be rapid, implicit, and affect-based; clients cross a subjective threshold to experience the alliance as established. Similarly, Flückiger et al. (2018, p. 2) defined the alliance as “the holistic collaborative aspects of the therapist-client relationship.” They reported that the factor structure of WAI, California Psychotherapy Alliance Scale (CALPAS; Marmar et al., 1986), and Helping Alliance Questionnaire (HAQ; Alexander and Luborsky, 1987) share a common component of a “confident collaborative relationship” (2018, p. 3).

While this study provided further evidence of ceiling effects on working alliance and similar measures, future research should attempt to replicate consistent responses on alliance measures. Ceiling effects offer a potential marker of the holistic connection at the scale level, while consistent responding may provide a marker at the item level. Researchers may wish to conduct qualitative studies to investigate how individuals who provide consistent responses to alliance items perceive the alliance as well as how they interpret particular alliance items. In any event, distinguishing between social desirability and holistic hypotheses for high scores on alliance measures may be difficult in that these two constructs may produce similar effects. Finally, the use of archival data prevents hypothesis confirmation bias and effects from investigator expectancies, but also presents alternative explanations relative to choice of measures, varied timing of measurement administration, and participant selection and motivation. Four of the six measures in this study were homemade, for example, and timing of the administration of these measures may also have influenced observed results.

Publicly available datasets were analyzed in this study. This data can be found here: University of Michigan Inter-University Consortium for Political and Social Research (http://www.icpsr.umich.edu).

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

The author confirms being the sole contributor of this work and has approved it for publication.

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.