Social determinants of health (SDoH) significantly impact patients' health and quality of life. Housing status is a key SDoH and ending homelessness among United States Veterans is a national priority for the Department of Veterans Affairs (VA), which provides a breadth of health and housing services for homeless-experienced Veterans (HEVs). To evaluate the effectiveness of VA homeless services, assessing short- and long-term housing outcomes is essential. However, to date, most studies that assess housing outcomes require collecting repeated patient-reported measures of housing status, which are costly and challenging to obtain.

In VA and other integrated healthcare systems, the Electronic Health Record (EHR) is a potentially valuable source of data regarding longitudinal housing outcomes. However, using EHR data for this purpose is challenging due to measurement error, missing data, and other complexities (Botsis et al., 2010; Wells et al., 2013; Glicksberg et al., 2018) which can bias outcomes assessed using longitudinal analyses (Lin et al., 2004; Pullenayegum and Lim, 2016; Lokku et al., 2021). SDoH are often recorded in the EHR using free-text clinical narratives (Organization, 2004; Gundlapalli et al., 2013, 2015; Peterson and Gundlapalli, 2015; Conway et al., 2019; Chapman et al., 2021; Hatef et al., 2022; Lybarger and Yetisgen, 2023; Tsai et al., 2022), and several studies have developed methods for extracting housing data from clinical texts (Gundlapalli et al., 2013; Conway et al., 2019; Chapman et al., 2021; Hatef et al., 2022; Lybarger and Yetisgen, 2023). In VA, one such system is Relative Housing Stability in Electronic Documentation (ReHouSED) (Chapman et al., 2021), a Natural Language Processing (NLP) system developed to extract housing stability from the EHR to evaluate VA's homelessness prevention and rapid rehousing program. ReHouSED demonstrated higher validity for identifying homeless status compared to International Classification of Diseases 10^th Edition (ICD-10) codes, a set of standardized codes representing clinical diagnoses and symptoms published by the World Health Organization (Organization, 2004).

However, there are several challenges in applying ReHouSED to study housing outcomes. First, the system may need to be adjusted for particular patient cohorts or evaluating specific services. It was originally designed for HEVs engaged in a rapid rehousing program; HEVs enrolled in other homeless services may have different EHR note structures or linguistic patterns. Second, missing data may cause bias when using ReHouSED for measuring outcomes. Information is only recorded in the EHR when patients present for care, which may occur more frequently for some patients than others. This produces observations at highly irregular intervals rather than the fixed, regularly spaced assessments that are ideal for longitudinal data collection, which can lead to biased analyses unless methods account for missing data (Pullenayegum and Lim, 2016; Lokku et al., 2021). Third, measurement error is ubiquitous in studies that use EHR data, particularly when using NLP to extract information from complex free text. While NLP is often designed to improve upon the shortcomings of structured data, misclassification is still present. This is especially true for complex variables such as longitudinal housing outcomes.

In the rapid rehousing context, ReHouSED achieved moderate accuracy (average positive predictive value and sensitivity of 65.3 and 68.1, respectively) and expert annotators achieved modest inter-annotator agreement (Cohen's Kappa = 0.7) (Chapman et al., 2021), demonstrating the complexity of the task. The accuracy of housing status classification can potentially be improved by combining NLP classifications with other EHR variables (e.g., ICD-10 codes) (Gundlapalli et al., 2015; Peterson and Gundlapalli, 2015; Wang et al., 2016; Nelson et al., 2018; Tsai et al., 2022). However, the accuracy of these data elements, as well as the best combination of indices, is not well-studied, in part due to the challenge of constructing a reference standard.

We aimed to develop a “best practice” for assessing longitudinal housing instability using observational EHR data as part of a quality improvement initiative targeting VA's Grant and Per Diem (GPD) case management aftercare program (hereafter, “Aftercare”). In this program, VA partners with community-based homeless service agencies to provide 6 months of case management for HEVs undergoing housing transitions (e.g., from institutional settings to independent housing). For a cohort of Aftercare patients in Southern California, we collected patient-reported housing history for a 2-year period. We then extracted six indicators of housing instability from the VA EHR: clinical note classifications of housing status obtained using ReHouSED tailored for this cohort (Chapman et al., 2021); ICD-10 codes for homelessness; notations of homeless service use found in outpatient visits; inpatient admissions associated with homelessness (e.g., residential treatment programs); a universal screening tool to assess housing instability; and data from VA's homeless registry. We compare the validity of each indicator of housing instability, considering the patient-reported data as a gold standard, and discuss implications for evaluations of housing interventions.

Table 1 summarizes demographics for patients who provided self-reported housing history (“interviewed”) vs. those who did not. Among interviewed patients, most (63.9%) were >60 years old and 85.2% were male. Over half (54.1%) were African American. Among the entire cohort, the most common psychiatric diagnoses were major depressive (27.2%) and post-traumatic stress disorders (24.9%), with a smaller proportion of patients demonstrating evidence of drug use (13.2%), alcohol use (16.1%), or psychotic spectrum disorders (4.9%).

We compared patient-reported housing history with clinical and administrative data regarding housing status for a cohort of homeless-experienced VA patients. Our goal was to compare the validity of different data elements to identify best practices for assessing longitudinal housing outcomes using EHR data. Among the small number of patients who experienced housing instability in our cohort, most had EHR documentation of their housing status. Using NLP to supplement standard structured data elements with information recorded in clinical notes NLP led to more complete assessment of longitudinal housing outcomes. This is an important finding with methodologic implications for optimizing the validity of assessing patients' longitudinal housing outcomes using EHR data when patient-level data collection is not feasible due to sample size or resource constraints.

In these analyses, sensitivity and specificity varied by EHR extraction method. NLP generally had higher sensitivity than structured EHR data for capturing repeated occurrences of housing instability, but demonstrated lower specificity than some structured elements. ICD-10 codes, which are often used in epidemiologic studies, had lower sensitivity and specificity than most other indicators, including NLP. Combining NLP and ICD-10 codes increased sensitivity but decreased specificity. These findings build on prior work with ReHouSED in a distinct cohort of VA patients engaged in rapid rehousing (Chapman et al., 2021); at the patient-month level, both analyses provide evidence that ReHouSED performs better than ICD-10 codes in measuring housing instability.

The VA EHR contains data elements for documenting housing instability that are unique to VA. In particular, outpatient administrative data had higher specificity than NLP and higher sensitivity than ICD-10 codes. Combinations of these three elements could be used to tailor definitions to improve sensitivity or specificity as appropriate for a particular cohort or analysis. Additionally, while encounter- and month-level performance varied across different data elements, patient-level sensitivity was similarly high for NLP, ICD-10 codes, and outpatient administrative data, suggesting structured data may be sufficient for constructing coarse definitions of housing instability (e.g., identifying patients with a history of housing instability at any point in time).

When patients received services recorded in HOMES, those episodes of housing instability were captured with high sensitivity and specificity. However, this dataset does not capture an important segment of the population that is disengaged from VA homeless services; our data suggests that quality improvement leaders and researchers using HOMES to assess housing outcomes should consider complementing this data with EHR data elements. These findings parallel prior work (Tsai et al., 2022) comparing estimated prevalence of homelessness across VA, which found that utilizing multiple EHR data elements can improve ascertainment of housing instability.

When deciding how to define housing instability using EHR data, we suggest that specific analytic goals and the underlying prevalence of housing instability be taken into consideration. Analyses examining cohorts with low prevalence of housing instability, as we had here, may demand high specificity to avoid large numbers of false positives. Specificity can be improved by requiring multiple data elements to show evidence of housing instability or by favoring more specific data elements. When high sensitivity is more desirable, using NLP or the union of multiple data elements may be more effective. Attention should also be given to missing data, as EHR data depends on patients presenting for care. Patients experiencing housing instability may use care more frequently, leading to an imbalance in the degree of observation for stably and unstably housed patients. To avoid biased results, longitudinal analyses of housing instability using EHR data should consider utilizing methods for adjusting for missing data and irregular observations (Lin et al., 2004; Pullenayegum and Lim, 2016; Pullenayegum and Scharfstein, 2022).

This work has limitations. First, we performed these exploratory analyses on a small sample and thus our statistical analyses had had low power. Regardless, the detailed patient-reported housing history we obtained over a 2-year period in patients with homeless experiences is a valuable observational dataset and our findings will inform future work. As with any retrospective analyses using patient-reported data, there is a possibility of recall or recruitment bias in our sample. Interviewed patients differed slightly in terms of race/ethnicity (i.e., interviewed patients were more likely to be African-American than the rest of the cohort and less likely to be Hispanic/Latino). They had similar distributions of psychiatric and substance use diagnoses, although the reported proportions only represent patients receiving clinical services related to these conditions and may not be reliable due to the inaccuracy of ICD-10 coding. To check for possible differential housing instability, we compared the EHR documentation of housing instability between interviewed and non-interviewed patients and found the two groups to be similar in terms of the frequency of documented housing instability, offering some assurance against recruitment bias; however, such bias remains a possibility. Second, we treated each indicator of housing instability as dichotomous. However, accuracy may be improved by factoring information such as the number of notes processed by the NLP during a single encounter or different levels of structured data (e.g., ICD-10 codes indicating risk of homelessness vs. literal homelessness). Third, we examined a cohort of VA patients from one geographic area enrolled in a particular housing program. The observed patterns here of housing instability and EHR documentation may not generalize to other cohorts of Veterans or to populations outside of the VA, who demonstrate different demographic characteristics and documentation patterns. However, documentation of housing and other SDoH is common in clinical texts, and ICD-10 codes are widely used across healthcare systems. We demonstrated here that ReHouSED could be tailored for a new cohort and analysis task, and other work has demonstrated the feasibility of customizing NLP systems developed in VA to be applied in other settings (Chapman et al., 2022). Additionally, this analysis was performed using data from the VA's legacy EHR, VISTA, which is planned to be replaced by Cerner. Future work should compare these findings with data in Cerner to ensure continuing data quality and accuracy.

Longitudinal housing status is an important outcome for patients who have experienced homelessness. For a sample of 61 homeless-experienced VA patients enrolled in a case management program, we found that housing status was documented longitudinally in the EHR using several structured and unstructured data elements. Using NLP to extract information from clinical notes can improve sensitivity for assessing housing outcomes, while incorporating multiple EHR indicators of housing instability achieves higher specificity compared to single indicators. Future work could customize ReHouSED for processing clinical texts within and outside VA for distinct patient cohorts, augmented by other EHR elements. Similar approaches could also be employed to evaluate other SDoH variables longitudinally using NLP.

The datasets presented in this article are not readily available because due to the sensitive and protected nature of this data, the authors are unable to make it available to the public. Requests to access the datasets should be directed to alec.chapman@hsc.utah.edu.

The studies involving human participants were reviewed and approved by VA Central Institutional Review Board, U.S. Department of Veterans Affairs. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

AC contributed to conceptualization, study design, NLP system development, data curation, data analysis, system evaluation, and manuscript preparation. KCo and SG contributed to conceptualization, study design, data curation, system evaluation, and manuscript preparation. SC, TP, and DA contributed to conceptualization, study design, data curation, primary data collection, and manuscript preparation. NJ contributed to conceptualization, study design, data analysis, and manuscript preparation. KCl and JT contributed to data curation and manuscript preparation. RN and AM contributed to conceptualization, study design, NLP system development, data curation, and manuscript preparation. EF contributed to conceptualization, study design, and manuscript preparation. All authors contributed to the article and approved the submitted version.