Undercounts of people with serious mental illness using the Washington Group Short Set questions

Background: Accurately counting Americans with mental health conditions is essential to support program development and appropriate resource allocations, which are often based on prevalence data. Multiple federal surveys use the Washington Group Short Set (WG-SS) questions to identify people with disabilities, including those with mental health conditions. However, the WG-SS questions miss many people with mental illnesses, under-representing this population in US federal survey data. Hence, we sought to explore the degree to which people with serious mental illness are missed.

Methods: We used data from the 2020 National Survey on Health and Disability to assess the rates that respondents with self-reported serious mental illness (SMI) conditions, i.e., major depression, bipolar disorder, schizophrenia, and schizoaffective disorder (n=263), were missed as disabled by the WG-SS questions.

Results: Using the three WG-SS questions suggested by the Washington Group to capture people with mental illnesses, 66.2%, 88.6%, and 96.6% of respondents with SMI were characterized as non-disabled; 58.2% were characterized as non-disabled across the three questions combined.

Discussion: Previous research demonstrated that the WG-SS questions missed almost 60% of respondents with any mental illness. However, the Washington Group states that its question set better captures people with more severe disabilities, so this study focused only on respondents with serious mental illnesses and only on questions that the Washington Group suggests capture people with psychosocial disabilities.

Conclusion: Results indicate that the WG-SS questions miss large percentages of even those with the most severe mental illnesses, who therefore may be substantially undercounted in US federal surveys using these questions. In turn, public mental health programs may be substantially underfunded.

Introduction

Serious mental illness (SMI) is defined as “a mental, behavioral, or emotional disorder resulting in serious functional impairment, which substantially interferes with or limits one or more major life activities” and is estimated to affect approximately 5.5% of adults in the United States (1, 2). The SMI designation originated in the deinstitutionalization movement with a desire to replace terms such as “chronically mentally ill” or “persistent mental illness” with a term that was more indicative of possibilities for recovery (3, 4). Although several mental health conditions could be considered “serious,” the most prevalent diagnoses referenced in reviews of the SMI category have included schizophrenia, schizoaffective disorders, psychotic disorders, bipolar disorder, and major depression (5, 6).

In the US, the Americans with Disabilities Amendments Act defines a disability as “a physical or mental impairment that substantially limits one or more major life activities” and, further, a condition is considered a disability without regard to the ameliorative effects of mitigating measures such as medication, i.e., mitigating measures must be ignored in determining if an impairment substantially limits a major life activity (7). The law also states that an impairment that is episodic or in remission meets the definition of disability if it would substantially limit a major life activity when active. In fact, the appendix to the law specifically mentions major depressive disorder, bipolar disorder, and schizophrenia as examples of impairments that may be episodic but are still considered disabilities (8). Given these definitions above, people with SMI have a disability under US law.

Understanding the prevalence and characteristics of people with serious mental illness is essential for program planning and allocation of funds for services. For example, it is estimated that 26% of adults with SMI are covered by Medicaid (9) and people with SMI generate significantly greater Medicaid expenditures than adults without (10). The American Community Survey (ACS) is a national survey representative at the state and local government level that identifies rates of disability, including SMI, which are used to determine funding allocations and to support planning for federal, state, and local government services, including Medicaid and Medicare services. Additionally, ACS rates of disability are used to identify needs for mental health professional workforce development in compliance with a range of laws including the Social Security Act, the Americans with Disabilities Act, the Older Americans Act, and the Rehabilitation Act (11). ACS data are also used to understand critical gaps in the social, educational, and medical service systems, including for identifying populations at disproportionate risk of experiencing barriers to health care and poor health outcomes and for documenting and addressing discrimination in education and employment. In particular, cutting-edge strategies to understand and ameliorate the current mental health crisis are grounded in data that allow linkages of geographic, household, and individual characteristics within the policy context where people live, such as available through the ACS data (12).

Currently, the ACS uses a set of six yes/no questions, known as the “ACS-6,” to identify people with disabilities. Two of these questions specifically reference functional difficulties potentially due to a mental or emotional condition: Because of a physical, mental, or emotional condition, do you have serious difficulty concentrating, remembering, or making decisions? and Because of a physical, mental, or emotional condition, do you have difficulty doing errands alone, such as visiting a doctor’s office or shopping? While not exclusive to people with mental health conditions, these questions can be used to identify people with SMI (13). However, the Washington Group Short Set (WG-SS) (14) of disability questions was recently proposed to be used in the ACS, replacing the ACS-6 set (15), and is widely used for identifying people with disabilities in other US federal surveys (e.g., the National Health Interview Survey, the National Health and Nutrition Examination Survey, and the National Survey of Family Growth), despite the fact that federal testing indicated use of the WG-SS questions in the ACS would decrease overall disability estimates in the US by at least 40% (16). In contrast to the yes/no questions in the ACS-6, the WG-SS questions use a scaled response, asking how much difficulty respondents have with certain functions (no, some, a lot of difficulty, or cannot do at all; those answering “a lot of difficulty” or “cannot do at all” are counted as having a disability). One rationale for using the WG-SS questions in the ACS is that they are said to be better at identifying people with more severe disabilities. However, recent research, including our own, demonstrates that the WG-SS fails to capture people with severe hearing, vision, and mobility disabilities (17, 18). In fact, multiple studies have documented significant shortcomings of the WG-SS in identifying people with a variety of disabilities, including those with psychiatric disabilities (19, 20). The objective of this paper is to explore the degree to which the WG-SS questions accurately identified people with the most serious mental illnesses and to augment the literature regarding underperformance of the WG-SS questions in identifying people with the most severe disabilities.

Materials and methods

Data

We used data from the 2020 National Survey on Health and Disability (NSHD; n=2,175), a national online survey of American adults ages 18–64 with disabilities administered by researchers at the University of Kansas (21). All potential survey participants were screened with the question, “Do you have a physical or mental condition, impairment, or disability that affects your daily activities and/or that requires you to use special equipment or devices, such as a wheelchair, walker, TDD [telecommunications device for the deaf] or communication device?” Those who answered ‘yes’ to this question were invited to complete the survey and asked the WG-SS disability questions. All respondents were also asked to list all of their disabilities or health conditions in an open-ended response question. From the sample of respondents who answered all disability questions (n=2,099), we selected the 263 respondents who wrote in having major depressive disorder (n=107), bipolar disorder (n=129), or schizophrenia/schizoaffective disorder (n=37).

Measures

While the WG-SS contains six questions, the Washington Group suggests (14) that people with psychosocial disabilities are identified primarily by their answers to the WG-SS questions about cognition (remembering and concentrating), communication, and self-care. Thus, we focused on responses to these three questions (see Table 1 for question phrasing). Washington Group guidance indicates that those who answer “a lot of difficulty” or “cannot do at all” to these questions are considered disabled while people who answer only “some difficulty” or “no difficulty” do not count as disabled. As mentioned, SMI conditions are directly captured via self-report in the NSHD, allowing us to examine responses to these three WG-SS questions by people who reported having SMI. Other authors have documented the validity of self-report of these SMI conditions (22, 23).

Table 1

Table 1. WG-SS psychosocial difficulties reported by NSHD respondents with serious mental illness (SMI)^a (n=263).

Results

Demographics for the NSHD group self-reporting SMI are shown in Table 2. In general, these demographics are similar to those for the overall NSHD sample of people self-reporting disabilities (also shown). As seen in Table 1, each of the three WG-SS questions said to capture people with mental health conditions missed more than half of respondents with SMI, because those who reported “no” or “some” difficulty with these functions would not have been identified as having a disability. Specifically, the question regarding difficulty with remembering or concentrating missed 66.2% of respondents with SMI; the question about difficulty with self-care missed 88.6% of respondents with SMI; and the question about difficulty with communication missed 96.6% of respondents with SMI. In combination, the three questions missed 58.2% of all respondents with SMI.

Table 2

Table 2. Demographics of NSHD respondents with self-reported serious mental illness and those without self-reported serious mental illness.

Discussion

It is possible that not all people living with an SMI diagnosis would generally be considered disabled. Nevertheless, the Americans with Disabilities Act as amended specifies that a disability is any “physical or mental impairment that substantially limits one or more major life activities,” even when the impairments are mitigated by medications or accommodations. Individuals with SMI whose impairments are mitigated may need services and supports to maintain access to appropriate medications and accommodations. Availability and funding of such services and supports are facilitated through accurate prevalence estimates. Other research has recently demonstrated that the WG-SS questions underestimate disability prevalence in the US, including among those with the most severe vision, hearing, and mobility disabilities who may also benefit from services and accommodations (17, 18). Our findings suggest people with serious mental illnesses are also missed.

In a previous study using this dataset, we found that the WG-SS questions missed 58.7% of people reporting any mental illness, including those reporting potentially less severe conditions such as anxiety, personality disorders, or general mood disorders (24). For this study, we focused only on respondents with serious mental health conditions—those who self-reported experiencing major depressive disorder, bipolar disorder, schizophrenia or schizoaffective disorder—and found that the WG-SS questions still missed 58.2% of them. This finding suggests that the WG-SS questions may not more accurately capture people with more severe disabilities as stated by the Washington Group. More importantly, this finding has serious implications for our understanding of the true prevalence of people with these conditions and their experiences in accessing needed health services.

An important reason for the different outcome rates from the NSHD and WG-SS measures is that the WG-SS questions focus only on functional impairment across three domains whereas the NSHD uses a much broader screening question for disability, followed by prompts to write in all disabilities experienced. Functional impairment is not typically the basis of measurement of SMI (5). The current standard, established through the Mental Health Surveillance Study, is to combine functional limitation with assessment of psychological distress and its frequency and social limitations (25). Furthermore, a strength-based approach to assessment emphasizes positive gains, such as building a cohesive sense of self through personal agency, in addition to symptom reduction and functional improvement (26).

Based on these results, using the WG-SS has significant potential to undercount people with SMI. Such undercounting could result in contractions in service provision and eligibility, which can result in adverse outcomes. For example, Simpson et al. (27) concluded that Social Security policy contractions (e.g., reductions in eligibility) are linked to negative subsequent population mental health outcomes.

Limitations in this work are important to note. The NSHD is an internet-based survey, which may make it more accessible to respondents with greater resources and higher functioning. To the extent that people with greater limitations would answer in the affirmative to both the NSHD and WG-SS questions, the rate of under-reporting of the WG-SS may be lower than reported here. On the other hand, the NSHD may underreport people with psychiatric conditions who do not think of themselves as having a disability or psychological distress. Because about half of people with psychiatric conditions do not believe they have a condition that can be helped with healthcare services (28), they might not see this survey with its focus on health as relevant. Additionally, people with SMI in living arrangements supported by informal or formal/paid help may not experience functional disability or psychological distress to the extent of those with less support and thus not report them. Because people who do not see the NSHD as relevant to them would likely also answer that they do not have any of the 3 functional disabilities of the WG-SS question set, the true under-reporting of the WG-SS may be higher than reported here.

Future research should continue developing SMI self-report questions that are not deficit-based and allow for self-categorization of disability and/or open-ended responses to improve specificity of data (29). It has been 22 years since the last nationally representative psychiatric epidemiological survey was completed (30). We need a new one to make efficient investment of funds to understand current population needs and address unmet needs for people with SMI in the face of rising rates of mental distress (31). Precise measurement in existing national surveys is critical to motivate that investment.

Disability questions are used in a range of national surveys. It is vitally important to understand if, and to what extent, they capture people with SMI. This study indicates that the WG-SS disability questions miss many in this population and that more inclusive, and less deficit-based, questions are needed. The broad question used in the NSHD captured many people with self-reported SMI not captured by the WG-SS questions. Other similar broad disability questions are being tested and documented and this study reaffirms the need for their wider use in national surveys (18, 19). As Abdalla and Galea (32) note, the future of mental health epidemiology requires a refinement of definitions and methodologies, including in measurement.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by University of Kansas Institutional Review Board. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

JH: Funding acquisition, Conceptualization, Formal analysis, Writing – original draft, Writing – review & editing, Supervision, Methodology. KT: Writing – original draft, Writing – review & editing. BM: Writing – original draft, Writing – review & editing. NK: Data curation, Project administration, Writing – review & editing, Formal Analysis.

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. The National Survey on Health and Disability (NSHD) is administered by the University of Kansas Institute for Health & Disability PolicyStudies (KU-IHDPS) and funded by the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRRprojects #90IFRE0050 and #90IFRE0089). NIDILRR is a center within the Administration for Community Living (ACL), U.S. Department of Health and Human Services (HHS). The contents of this manuscript, do not necessarily represent the policy of NIDILRR, ACL, or HHS and you should not assume endorsement by the federal government.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declare that no Generative AI was used in the creation of this manuscript.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. National Institute of Mental Health. Mental Illness (2024). Available online at: https://www.nimh.nih.gov/health/statistics/mental-illness (Accessed March 27, 2025).

Google Scholar

2. Substance Abuse and Mental Health Services Administration. Key substance use and mental health indicators in the United States: Results from the 2021 National Survey on Drug Use and Health (HHS Publication No. PEP22-07-01-005, NSDUH Series H-57) (2022). Available online at: https://www.samhsa.gov/data/report/2021-nsduh-annual-national-report (Accessed March 27, 2025).

Google Scholar

3. Goldman HH and Grob GN. Defining ‘mental illness’ in mental health policy. Health Aff. (2006) 25:737–49. doi: 10.1377/hlthaff.25.3.737

PubMed Abstract | Crossref Full Text | Google Scholar

4. Kinter KT. What’s in a name:”Serious”,”Severe”, and “Severe and persistent. Int J Psychosoc Rehabil. (2017) 21:52–4. Available online at: https://www.psychosocial.com/index.php/ijpr/article/view/1653/1497 (Accessed June 16, 2025).

Google Scholar

5. Gonzales L, Kois LE, Chen C, López-Aybar L, McCullough B, and McLaughlin KJ. Reliability of the term “Serious mental illness”: A systematic review. Psychiatr Serv. (2022) 73:255–1262. doi: 10.1176/appi.ps.202100661

PubMed Abstract | Crossref Full Text | Google Scholar

6. Martínez-Martínez C, Richart-Martínez M, and Ramos-Pichardo JD. Operational definition of serious mental illness: heterogeneity in a review of the research on quality-of-life interventions. J Am Psychiatr Nurses Assoc. (2020) 26:229–44. doi: 10.1177/1078390320902823

PubMed Abstract | Crossref Full Text | Google Scholar

7. U.S. Equal Employment Opportunity Commission. ADA Amendments Act of 2008 (2008). Available online at: https://www.eeoc.gov/statutes/ada-amendments-act-2008 (Accessed June 3, 2025).

Google Scholar

8. U.S. Equal Employment Opportunity Commission. Questions and Answers on the Final Rule Implementing the ADA Amendments Act of 2008 (2011). Available online at: https://www.eeoc.gov/laws/guidance/questions-and-answers-final-rule-implementing-ada-amendments-act-2008 (Accessed June 4, 2025).

Google Scholar

9. Centers for Medicare and Medicaid Services. Using Data Analytics to Better Understand Medicaid Populations with Serious Mental Illness . Available online at: https://www.medicaid.gov/state-resource-center/innovation-accelerator-program/iap-downloads/program-areas/data-analytics-smi-2.pdf (Accessed March 27, 2025).

Google Scholar

10. Rowan P, Begley C, Fu S, Turibekov B, and Morgan R. Impact of serious mental illness on medicaid and other public healthcare costs in texas. Adm Policy Ment Health. (2019) 46:498–506. doi: 10.1007/s10488-019-00929-y

PubMed Abstract | Crossref Full Text | Google Scholar

11. US Census Bureau. American Community Survey Handbook of Questions and Current Federal Uses American Community Survey Office (2014). Available online at: https://www2.census.gov/library/publications/2014/acs/acs-federal-uses.pdf (Accessed March 27, 2025).

Google Scholar

12. Alegría M, Xiong M, and Sánchez González ML. The role of social determinants in racial and ethnic mental health disparities: getting it right. Harv Rev Psychiatry. (2025) 33:67–77. doi: 10.1097/HRP.0000000000000421

PubMed Abstract | Crossref Full Text | Google Scholar

13. Miller K and DeMaio TJ. Report of cognitive research on proposed American Community Survey Disability Questions (2006). Available online at: https://www.census.gov/content/dam/Census/library/working-papers/2006/adrm/ssm2006-06.pdf (Accessed March 27, 2025).

Google Scholar

14. Washington Group on Disability Statistics. An introduction to the Washington Group on Disability Statistics Question Sets (2020). Available online at: https://www.washingtongroupdisability.com/fileadmin/uploads/wg/Documents/An_Introduction_to_the_WG_Questions_Sets:2_June_2020_.pdf (Accessed March 27, 2025).

Google Scholar

15. Santos RL. Next Steps on the American Community Survey Disability Questions. Washington DC: U.S. Census Bureau (2024). Available at: https://www.census.gov/newsroom/blogs/director/2024/02/next-steps-on-acs-disability-questions.html.

Google Scholar

16. US Census Bureau. 2023 Research and evaluation report memorandum (2023). Available online at: https://www.census.gov/content/dam/Census/library/working-papers/2023/acs/2023_Steinweg_01.pdf (Accessed June 3, 2025).

Google Scholar

17. Goddard KS and Hall JP. Limitations of the Washington Group Short Set in capturing moderate and severe mobility disabilities. Health Aff Sch. (2025) 3. doi: 10.1093/haschl/qxaf015

PubMed Abstract | Crossref Full Text | Google Scholar

18. Landes SD, Swenor BK, and Hall JP. Performance of the Washington Group questions in measuring blindness and deafness. Health Aff Sch. (2024) 2. doi: 10.1093/haschl/qxae131

PubMed Abstract | Crossref Full Text | Google Scholar

19. Landes SD, Hall JP, Swenor BK, and Vaitsiakhovich N. Comparative performance of disability measures. PloS One. (2025) 20. doi: 10.1371/journal.pone.0318745

PubMed Abstract | Crossref Full Text | Google Scholar

20. Landes SD, Swenor BK, and Vaitsiakhovich N. Counting disability in the National Health Interview Survey and its consequence: Comparing the American Community Survey to the Washington Group disability measures. Disabil Health J. (2024) 17(2):101553. doi: 10.1016/j.dhjo.2023.101553

PubMed Abstract | Crossref Full Text | Google Scholar

21. University of Kansas Institute for Health and Disability Policy Studies. The National Survey on Health and Disability (NSHD) (2025). Available online at: https://ihdps.ku.edu/nshd (Accessed March 27, 2025).

Google Scholar

22. Woolway GE, Legge SE, Lynham AJ, Smart SE, Hubbard L, Daniel ER, et al. Assessing the validity of a self-reported clinical diagnosis of schizophrenia. Schizophr. (2024) 10:1–11. doi: 10.1038/s41537-024-00526-5

PubMed Abstract | Crossref Full Text | Google Scholar

23. Vieira LS, Nguyen B, Nutley SK, Bertolace L, Ordway A, Simpson H, et al. Self-reporting of psychiatric illness in an online patient registry is a good indication of the existence of psychiatric illness. J Psychiatr Res. (2022) 151:34–41. doi: 10.1016/j.jpsychires.2022.03.022

PubMed Abstract | Crossref Full Text | Google Scholar

24. Hall JP, Kurth NK, Ipsen C, Myers A, and Goddard KS. Comparing measures of functional difficulty with self-identified disability: implications for health policy. Health Aff. (2022) 41:1433–41. doi: 10.1377/hlthaff.2022.00395

PubMed Abstract | Crossref Full Text | Google Scholar

25. Aldworth J, Colpe LJ, Gfroerer JC, Novak SP, Chromy JR, Barker PR, et al. The national survey on drug use and health mental health surveillance study: calibration analysis. Int J Methods Psychiatr Res. (2010) 19:61–87. doi: 10.1002/mpr.312

PubMed Abstract | Crossref Full Text | Google Scholar

26. Hamm JA and Lysaker PH. Application of integrative metacognitive psychotherapy for serious mental illness. Am J Psychother. (2018) 71:122–7. doi: 10.1176/appi.psychotherapy.20180033

PubMed Abstract | Crossref Full Text | Google Scholar

27. Simpson J, Albani V, Bell Z, Bambra C, and Brown H. Effects of social security policy reforms on mental health and inequalities: A systematic review of observational studies in high-income countries. Soc Sci Med. (2021) 272:113717. doi: 10.1016/j.socscimed.2021.113717

PubMed Abstract | Crossref Full Text | Google Scholar

28. Kessler RC, Berglund PA, Bruce ML, Koch JR, Laska EM, Leaf PJ, et al. The prevalence and correlates of untreated serious mental illness. Health Serv Res. (2001) 36:987–1007.

Google Scholar

29. Mizock L, Dilts G, Sotilleo E, and Cherry J. Preferred terminology of people with serious mental illness. Psychol Serv. (2024) 21:184–97. doi: 10.1037/ser0000717

PubMed Abstract | Crossref Full Text | Google Scholar

30. Alegria M, Jackson JS, Kessler RC, and Takeuchi D. (2024). Collaborative psychiatric epidemiology surveys (CPES), 2001–2003 [United states], in: Inter-university Consortium for Political and Social Research [distributor], . doi: 10.3886/ICPSR20240

Crossref Full Text | Google Scholar

31. Jackson HJ and Haslam N. Ill-defined: Concepts of mental health and illness are becoming broader, looser, and more benign. Australas Psychiatry. (2022) 30:490–3. doi: 10.1177/10398562221077898

PubMed Abstract | Crossref Full Text | Google Scholar

32. Abdalla SM and Galea S. Key considerations for the future of mental health epidemiology. Am J Epidemiol. (2024) 193:1307–12. doi: 10.1093/aje/kwae114

PubMed Abstract | Crossref Full Text | Google Scholar