The Representativeness of Participants With Severe Mental Illness in a Psychosocial Clinical Trial

Introduction: Cardiovascular morbidity and mortality are increased in severe mental illnesses (SMI). Trials of psychosocial health interventions to improve physical health in SMI, including in treatment-resistant schizophrenia, have shown some benefit. However, the representativeness of participants in such trials has not been determined. Method: We utilized an anonymised case register to determine if participants in a randomized controlled trial (RCT) of a novel psychosocial health intervention aiming to improve physical health in SMI had similar severity of illness to eligible non-participants. A retrospective database analysis was performed, using Health of the Nation Outcome Scale (HoNOS) data from the sample of patients participating in the IMPaCT (Improving Physical health and reducing substance use in Psychosis) RCT (n = 293) compared to all eligible participants with a psychotic illness (n = 774). Results: The mean total HoNOS score in the eligible comparator population (Mean = 9.09, SD = 5.8, range = 0–30) was significantly greater than that of the IMPaCT RCT participants (Mean = 7.16, SD = 4.7, range = 0–26), (t = 3.810, p = 0.006), as was the degree of overall illness severity and functional impairment, as measured by HoNOS. Conclusion: This study shows for the first time that the patient population participating in an RCT of a lifestyle intervention for those with SMI had a better mental health status at entry to the trial, than the total eligible population, although there was no difference in physical health needs. This has relevance to the applicability of RCTs of lifestyle interventions in service planning and suggests that when people are more unwell, greater effort may be needed to include them in psychosocial interventions. A more careful and focused recruitment approach should be followed to improve the participation of the more severely ill patients in psychosocial interventions in order to enhance the external validity of such studies.


INTRODUCTION
People with severe mental illness (SMI), such as schizophrenia, schizoaffective disorder and bipolar affective disorder, have reduced life expectancy compared to those in the general population (1)(2)(3). Most of this excess mortality is due to physical illnesses, with cardiovascular disease prominent (4,5). Metaanalyses have demonstrated that targeted behavioral and nonspecific psychosocial interventions can be beneficial in reducing antipsychotic induced weight gain and improving metabolic parameters (6)(7)(8). However, in order to determine the external validity of these findings (i.e., the extent to which the results can be generalized to clinical practice), we must determine how the extent to which participants in such studies are representative of people with SMI.
Similar concerns have been raised in relation to inclusion criteria for RCTs investigating the efficacy of clozapine in treatment resistant schizophrenia (TRS), where a broader range of patients are included than those meeting strict criteria for treatment resistance, with inclusion of treatment intolerant and non-refractory cases (9,10).
Although there are examples of studies that have followed up non-randomized patients in clinical trials making it possible to assess and describe the generalizability of the results (11), few studies involving people with SMI have compared the characteristics of participating and non-participating patients, thus limiting the external validity of study findings. One difficulty in doing so lies in the ethical challenges of obtaining clinical data pertaining to individuals who have not consented to participate in the study (12,13). The generalisability and real world translation of research in SMI is further limited by nonparticipation which may be selective (13). For example, at the more severe spectrum of mental illness, people may lack the capacity to consent to research, reducing the representativeness of research samples (14). Participation rates for studies in SMI are thought to be low, although this has not been as widely documented as the high dropout rates in this population (15). A recent survey of a large representative sample of people with psychotic disorders identified that 65% (n = 773) of those approached consented to participate in research, with older people less likely to do so (16).
Obstacles to participation include illness severity (17)(18)(19), fear of worsening of mental state due to participation (20), and concerns about adverse treatment effects (21). Negative symptomatology in patients with schizophrenia, with its inherent poor motivation and communication difficulties, may further reduce participation and limit the applicability of study findings (13).
While there is extensive work aimed at improving the participation of eligible patients (22) and identifying barriers to patient participation in mental health research (23), little is known about the clinical characteristics of non-participants. In particular, no such data exists for studies of non-pharmacological interventions to improve metabolic parameters in SMI, an important gap in the evidence on which to plan services.
We set out to determine the clinical characteristics of patients eligible to participate in the Improving Physical health and reducing substance use in Psychosis (IMPaCT) randomized controlled trial (RCT) (24) and compare these to the participating group using the Health of the Nations Outcome Scale (HoNOS) scores (25). We hypothesized that overall, individuals who were eligible to participate in the RCT would be more severely ill and functionally impaired as assessed by Health of the Nations Outcome Scale (HoNOS) scores (25), compared to those who agreed to participate.

MATERIALS AND METHODS
This current study is a secondary analysis of the Improving Physical health and reducing substance use in Psychosis (IMPaCT) cluster randomized controlled trial (RCT) (Trial registration: ISRCTN58667926) (24,26). The IMPaCT study is a multicentre, two arm, parallel cluster randomized controlled trial (RCT) of a psychosocial health promotion intervention (IMPACT Therapy) in people with a diagnosis of SMI (24). The patient-tailored IMPACT Therapy aimed to target one or more health behaviors from a pre-defined list that includes cannabis use; alcohol use; other substance use; cigarette smoking; exercise; diet and diabetic control, prioritizing those identified as problematic by the patient, taking a motivational interviewing (MI) and cognitive behavioral therapy (CBT) approach. Participants were permitted to start the communitybased IMPACT Therapy as soon as they were well-enough to attend, even if they were in-patients, to mirror clinical practice (24,26).
The aim of this current study is to determine whether participants in the IMPaCT RCT were representative of the target population in respect of levels of illness severity and functional impairment.

Setting
The South London and Maudsley NHS Foundation Trust (SLaM) is one of the largest providers of secondary mental health care in Europe. It provides mental healthcare across four London boroughs (Croydon, Lambeth, Lewisham, and Southwark) to a population of 1.2 million. Since 2006, electronic records have been used across all SLAM services. Since 2008 the Case Register Interactive Search (CRIS) system has been developed to allow for the search and retrieval of anonymised electronic clinical records of patient data (27). Over 250,000 cases are currently represented on CRIS in the form of detailed anonymised clinical information (28). The protocol for this case register has been described in an open-access publication (27).

Participants and Recruitment
In the IMPaCT RCT, to maximize inclusiveness, care coordinators from continuing care/recovery teams, community rehabilitation, assertive outreach, and community forensic teams were recruited in random order and the eligible patients of consenting care co-ordinators likewise approached for inclusion in the study in random order. The inclusion criteria for IMPaCT service user participants were as follows: male or female aged between 18 and 65 years old; community patients with a primary diagnosis of a non-affective psychotic disorder (ICD10 diagnostic criteria: F20.0-F29.0) or an affective psychotic disorder (F30-33). Exclusion criteria included a primary diagnosis of learning disability; a first episode of psychosis; serious physical illness that could impact metabolic measures and substance misuse; pregnant or up to 6 months post-partum; or receiving intensive care for a medical or terminal condition.
In this present study, the comparator population comprised the patients meeting the same inclusion criteria who were on the overall caseload of each of the participating care coordinators. The IMPaCT group were therefore a subset of the comparator population.
Individuals on the comparator caseload were included in the analysis if they had been assessed by a mental health professional using the Health of the Nations Outcome Scale (HoNOS) at least once in a time period of plus or minus 6 months from the date of the recruitment of the care coordinator to the IMPaCT RCT. If the individual had required an acute psychiatric hospital admission during that period they were excluded from the comparator population. Further, if no HoNOS score was completed in the 12 months study period, then the patient was excluded.

Outcome Measures
The primary measure of interest was the total HoNOS scale score, as this was routinely collected clinically and so was available in the pseudoanonymised clinical sample as well as the research sample. We compared HoNOS in patients participating in the IMPaCT RCT compared to that of all patients with SMI in the caseload of their care coordinators. Sociodemographic characteristics were not taken for comparison, as this would have potentially identified study non-participants.
A total score from the 12 items gives a measure of illness severity. Individual items and subscales in the 4 domains can be analyzed to assess their relative contributions to global functioning. Each item is measured on a 5-point Likert scale where; 0 = no problem within the period rated, 1 = sub-threshold problem, 2 = mild but definitely present, 3 = moderately severe, and 4 = severe to very severe, making 48 the highest possible score on the HoNOS (29).
Individuals in the total caseload who had a HoNOS score assessed during the 12 months study period were included. In the analysis we used the earliest HoNOS completed closest to the recruitment date of the relevant care coordinator to the RCT. HoNOS scores for individuals who had a hospitalization or an admission to a high intensity community support team [such as a Home Treatment Team (HTT)] within 6 months before or after the recruitment date were excluded to reduce the likelihood of us merely demonstrating that a short-term deterioration in mental state reduces participation in community-based trials. We did not exclude participants in the IMPaCT study group who were admitted to hospital or to a Home Treatment Team (HTT) following recruitment to the trial, although trial recruitment solely took place in the community.
The HoNOS scores of interest for this analysis were total HoNOS scale scores, subscale scores and individual itemized HoNOS scale scores. Items 9, 10, 11, and 12 in particular were assessed as a measure of functional impairment.
The means of total HoNOS scores from the overall care coordinator caseload and IMPaCT RCT participants were used as a proxy measure of global illness severity. Secondary outcomes were the "functional impairment" HoNOS subscale, which was composed of: item 9, "impairments in interpersonal relationships" (such as social withdrawal); item 10-"impairments in activities of daily living" (such as washing, dressing, mobility, and use of transport); item 11-"deficits in the quality of living conditions" (including absence of basic necessities such as heat and light); and item 12-"impairments in occupational functioning" (including the ability to engage in occupational and recreational activities).
In the study, HoNOS ratings were conducted by mental health professionals directly involved in the care of patients care for the IMPaCT non-participants and by clinical researchers for the IMPaCT study participants. It was not possible to measure interrater reliability for HoNOS scores between the clinical staff and the IMPaCT clinical researchers.

Statistical Analysis
For this analysis, a paired-samples t-test was used to compare: 1. total HoNOS scores from the overall caseload data set and from the IMPaCT RCT participant data set 2. HoNOS subscale scores (items 9, 10, 11, and 12) in the overall caseload data set and the IMPaCT RCT participant data set 3. individual HoNOS item scores between the overall caseload group and the IMPaCT RCT participants group.
The statistical package SPSS version 24 was used for the analyses and all t-tests were two-tailed with statistical significance set at an alpha level of p ≤ 0.05.

RESULTS
The IMPaCT RCT recruited 293 eligible participants from within SLaM, who were on the caseload of 68 care coordinators. In this present study, using CRIS, we identified on the overall caseload of each of the 68 eligible care coordinators, a total comparator population comprising 1,109 patients who met the inclusion criteria, including having a primary diagnosis of a psychotic disorder. Of these 1,109 patients, 19 were excluded due to incomplete HoNOS scoring in the 12 month period from the time of the RCT recruitment, giving a HoNOS completion rate of 98.3%, and leaving a total of 1,090 in the comparator group. A further 316 (21.7% of those with a psychotic disorder) were excluded having had a hospitalization or admission to a HTT over the study period, leaving 774 patients in the comparator group for analysis. The mean total HoNOS score in the target population (n = 774) (Mean = 9.09, SD = 5.8, range = 0-30) was significantly greater than the mean total HoNOS score for the IMPaCT RCT participating group (n = 293) (Mean = 7.16, SD = 4.7, range =0-26), (t = 3.810, p < 0.001). Comparison of the individual HoNOS item scores between those participating in the RCT and those on the care coordinators total caseload is shown in Table 1.
There were significantly increased scores for HoNOS item 1 (overactive, aggressive, or agitated behavior regardless of cause), item 4 (cognitive problems), item 8 (symptoms due to other mental or behavioral problems), item 9 (problems with relationships), item 10 (problems with activities of daily living), and item 11 (problems with living conditions) in the comparator group when compared to the IMPaCT participants (see Table 1). There was no significant difference in the HoNOS scores on the physical health item between the comparator group and the IMPaCT study group [Mean difference (MD) =0.24; t = 1.408, p = 0.161].
There was a significant increase in the HoNOS subgroup score for items 9,10,11, and 12 in the comparator group [mean HoNOS score for items 9,10,11, and 12 = 3.35 (SD = 2.6)] compared to the IMPaCT group [mean HoNOS score for items 9,10,11, and 12 = 2.41 (SD = 2.3) (t = 3.945, p < 0.001]. This indicates that the comparator group were more functionally impaired than the IMPaCT study participants. The frequency of responses to each of the HoNOS items by group is shown in Table 2.

DISCUSSION
To the best our knowledge, this is the first time that comparative levels of illness severity and functional impairment in a large, eligible non-participating group, and participating sample of a psychosocial health intervention RCT in SMI have been explored. This study demonstrates that both illness severity and functional impairment were increased in the non-participating population compared to the participating group. The overall health status was better in the study population, and the less severely ill patients were recruited to this trial.
Of interest, the levels of physical health problems were similar between both groups. This is relevant, as the IMPaCT trial intervention was designed to effect physical health improvements in the study population. The comparability between physical health impairment in both groups suggests that this did not drive participation selection bias, as would have been indicated by either increased severity of physical illness in the nonparticipating comparator group or indeed, by people with more physical health problems electing to participate. Instead, it appears there was an equivalent physical health need, but that other factors accounted for the difference in uptake of the research opportunity.
Limitations of this study need to be considered. It was not possible to assess for the effect of gender, age, ethnicity, and duration of illness of the participating and non-participating patient populations. Our inability to investigate factors that may be predictive of non-trial participation is a limitation, information that would be informative to improve the design of future RCTs. Studies have indicated that older age (16,23) and ethnicity (30), specifically black ethnicity (16), are barriers to recruitment in mental health studies, factors which may be related to illness severity. Clinical data that may have impacted on study involvement, such as duration of illness and number of psychiatric hospitalizations, were not available in the pseudoanonymised comparator sample. However, the two study populations were comparable on clinical symptoms such as hallucinations/delusions, and depression, indicating that active symptoms of mental illness were not impacting on the participation rates between the groups. Data were obtained retrospectively, which may limit the generalisability of the findings. However, the recruitment of a patient population for a prospective study would be difficult, and the method used has enabled us to for the first time compare these groups in a behavioral intervention in SMI.  Mild to very severe problem 183 (24) 48 (16) In the analysis, we used only a single HoNOS score based on the first HoNOS assessment in the relevant study period. This precludes a more encompassing assessment of fluctuating symptom profiles over time. The use of the HoNOS provides a behavioral assessment of functioning at the level of individual items, but does not allow for assessment of discrepancies between behavior and inner experience. It may be that in a population of individuals with SMI that the significantly increased functional impairment in the inclusive comparator group is related to negative symptomatology (31,32), but this is something that we were not able to assess.
In this study, HoNOS scores measured by care coordinators for the comparator population were compared with those measured by the IMPaCT RCT researchers. HoNOS is reported to show a moderate inter-rater reliability, but this is improved with training in HoNOS completion (33). Both care coordinators and researchers in the IMPaCT RCT study were trained in the use of the HoNOS, with the expectation that this would support inter-rater reliability.
Strengths of this study include the size and comprehensiveness of the sample. We were able to access the clinical records of over 1,000 community dwelling individuals with psychotic disorders. We looked at data specifically relating to individuals with SMI living in the community and purposefully excluded those who were hospitalized over the study period. This aids the applicability of our study findings to ambulatory research and enhances the generalisability of the study findings. Due to the comprehensiveness of the search tool, we were able to identify a patient population that reflects the characteristics of patients seen in standard clinical practice thus further enhancing the validity of our findings.
An additional important finding in this study was the high rate of HoNOS completion in the sample of community dwelling patients with SMI (98% completion rate). This finding demonstrates the clinical utility of the HoNOS and a high level of acceptability for its use in community mental health settings, although this was likely enhanced in UK practice by HoNOS measures being used for funding models. These findings are however mirrored in other community samples, such as in New Zealand where high completion rates of the HoNOS are documented, more so than in inpatient settings (95 and 79% respectively) (34).
The greater illness severity in the comparator group as compared to the participating group is unlikely to be a result of the recruitment and randomization method in the IMPaCT RCT. An important recruitment factor in the IMPaCT RCT was the use of relatively wide inclusion criteria, including dual diagnosis, and complex patients.
We cannot tell from these data whether the more ill patients would have participated in the psychosocial health promotion intervention were it not part of a clinical trial. If research itself were the barrier, it raises questions as to whether published research into psychosocial interventions for physical health is applicable to the patients with the greatest impairment in health and social functioning. This mirrors more broad concerns regarding the representativeness of RCTs of treatment interventions in SMI, and how this impacts on translation to real world clinical practice. Observational data can help in this regard. Clinical implementation trials for physical health problems in SMI are required (35), which may be pragmatic large scale trials to ensure broad inclusion criteria, heterogeneous populations and assessment of the real world effectiveness of psychosocial interventions for physical health (36). There remains limited data on what factors predict entry to psychosocial intervention trials in SMI. An increased awareness of this may aid increased knowledge of when medication based and/or psychosocial interventions are preferable to psychosocial interventions alone.
Our findings have implications for future RCTs of psychosocial intervention in SMI. The study finding that individuals with greater functional impairment are less likely to participate in RCTs should lead to focused interventions to increase their participation. This is required in order to ensure that trial results can be confidently translated into clinical practice.

AUTHOR CONTRIBUTIONS
JL and FG conceptualized the study. JL, SS, RM, and FG contributed to the design. JL, RW, SN, and HS conducted the data collection. JL and RW undertook the statistical analysis. JL wrote the first draft of the manuscript. All authors contributed to and have approved the final manuscript.