COVID-19 Consumer Health Information Needs Improvement to Be Readable and Actionable by High-Risk Populations

Health communicators help promote recommended health behaviors by providing accurate, actionable health information that is easy to read and understand. The COVID-19 public health crisis presents a special challenge to clear health communication because some populations most affected by the virus are also at risk for limited health literacy. We collected 28 consumer COVID-19 materials from the internet using popular search engines. We then assessed the materials for readability, understandability, and actionability using validated tools. Aggregate results suggest that the sample of materials was difficult to read and lacked a number of recommended features that promote a readers' ability to understand and act upon the information. We present these findings, their implications for health equity, and their limitations and then suggest ways to improve future health communication about time-sensitive infectious diseases.

One of the WHO's strategic objectives to respond to this pandemic included public communication about disease risk along with details to counter misinformation 2 . Many organizations have posted information about COVID-19 online in an effort to educate the public about the evolving situation. Likewise, the public is seeking information about the pandemic. A study conducted by the Pew Research Center in late March 2020 found that 7 in 10 US adults had used the internet to learn about COVID-19 5 .
In their communications, public health authorities frequently include actions that community members can take to limit the spread of COVID-19 (e.g., properly washing hands and staying home if ill). To follow public health guidance, consumers must be able to read and understand instructions, but this is a challenge for most adult Americans. Only 12 out of 100 have proficient health literacy skills (Kutner et al., 2006). Being able to read, or general literacy, is important to health literacy, which comprises a larger set of skills. Beyond the ability to read prose narrative, health literacy skills encompass information-seeking, interacting with forms and other documents, and the ability to use numbers to improve health.
Populations at risk for limited health literacy include those who are older and those from minority groups (Kutner et al., 2006). Compared with 12 out of 100 adults in the general population with proficient health literacy, only three out of 100 adults over age 65 fall into this category. Similarly, most racial and ethnic minorities fare worse than their white peers. Among African Americans and Hispanics, just two and four out of 100, respectively, demonstrate proficient health literacy skills (Kutner et al., 2006).
Of note, the same populations are at increased risk for COVID-19 infection and death (Mueller et al., 2020). Adults over 65 make up ∼80% of COVID-19 hospitalizations. Weekly reports continue to show greater numbers of COVID-19 deaths among older adults 6 . Many states are reporting that African Americans experience disproportionate rates of positive COVID-19 cases and death (Abrams and Szefler, 2020) 7 . As an example, an April 2020 report showed that while African Americans comprised only about 30% of the population of the state of Louisiana, they accounted for closer to 70% of the state's COVID-19 deaths (Yancy, 2020). And in Baltimore, where Hispanics comprised 5% of the population, they accounted for 12% of the COVID-19 patient population 8 . A May 29, 2020 report stated that rates of hospitalizations for African Americans were 4.5 times that of 5 Pew Research Center. (2020). Americans turn to technology during COVID-19 outbreak, say an outage would be a problem. Available online at: https://www. pewresearch.org/fact-tank/2020/03/31/americans-turn-to-technology-duringcovid-19-outbreak-say-an-outage-would-be-a-problem/ (accessed April 14, 2020). 6 Centers for Disease Control and Prevention. (2020). Weekly Updates by Select Demographic and Geographic Characteristics. Available online at: https://www.cdc. gov/nchs/nvss/vsrr/covid_weekly/index.htm (accessed June 2, 2020). 7 American Association of Retired Persons. (2020). Blacks, Hispanics Hit Harder by the Coronavirus, Early U.S. Data Show. Available online at: https://www.aarp.org/ health/conditions-treatments/info-2020/minority-communities-covid-19.html (accessed June 2, 2020). 8 The Baltimore Sun. (2020). Latinos disproportionately hurt by coronavirus in Maryland, Baltimore and among Johns Hopkins patients. Available online at: whites, and rates for Hispanics were 3.5 times that of whites 9 . Because these populations are at increased risk for COVID-19 and its complications and are also likely to be limited by health literacy, clear communication with them about COVID-19 is of critical importance.
While often described as a measure of individual capacity, health literacy is influenced not only by individual characteristics, but by the demands or complexities of the health information itself 10 . To address limitations in the health literacy skills of consumers, communicators can follow recommended practices to produce health information that is readable, understandable, and actionable 11,12 .
Our team used validated formulas and tools to formally assess a sample of highly visible online COVID-19 materials retrieved during late March 2020 to determine how readable, understandable, and actionable they were. The results point to a number of techniques that could be used to improve current and future messaging and to better engage individuals in public health behaviors that limit the spread of infectious diseases.

MATERIALS AND METHODS
The purpose of this study was to evaluate online COVID-19 information intended for and easily accessible by the general public.
To locate the content most likely to be viewed by consumers, we used three top search engines (Google, Yahoo!, and Bing) 13 and searched for content using the "incognito" method to prevent previous internet search history from affecting the search results. During the week of March 23, 2020, we entered four terms into each search engine: coronavirus, covid-19, covid19, and covid 19. Within each set of search results, we chose the top six web site links for further review. We excluded all sponsored or promoted content and content from news outlets. In addition to helping avoid bias that may be found in these materials, our selected assessment tool is designed to assess educational materials rather than news or advertisements.
Of the top 72 links initially identified, we removed 44 duplicates. We followed the remaining 28 links to their respective https://www.baltimoresun.com/coronavirus/bs-md-covid-latinos-20200512-s3cjb6swbbfofmmfg7afmj3zw4-story.html (accessed June 2, 2020). 9 Centers for Disease Control and Prevention. (2020). COVIDView Weekly Summary, Key Updates for Week 22, ending May 30, 2020. Available online at: https://www.cdc.gov/coronavirus/2019-ncov/covid-data/covidview/index.html (accessed June 2, 2020). 10 Office of Disease Prevention and Health Promotion. (2020). Health Literacy-Healthy People 2020. Available online at: https://www.healthypeople.gov/2020/ topics-objectives/topic/social-determinants-health/interventions-resources/ health-literacy (accessed April 16, 2020). 11 Plain Language Action and Information Network (PLAIN). (2020). Federal plain language guidelines. Available online at: https://plainlanguage.gov/guidelines/ (accessed April 16, 2020). 12 Agency for Healthcare Research and Quality. (Content last reviewed May 2015). Tip 6. Use Caution With Readability Formulas for Quality Reports. Available online at: https://www.ahrq.gov/talkingquality/resources/writing/tip6. html (accessed April 20, 2020). 13 StatCounter GlobalStats. (2020). Search Engine Market Share Worldwide-April 2020. Available online at: https://gs.statcounter.com/search-engine-market-share (accessed May 8, 2020). web sites. If the material that appeared at the landing page met our inclusion criteria, we included it in our assessment. If it did not, we identified the next available appropriate material within the site, using a left-to-right and top-to-bottom approach. Inclusion criteria were met if the material was clearly directed at the general public and if it was at least 100 words in length. The software we used for readability assessments includes a cautionary note that samples smaller than 100 words do not produce valid readability results. The majority of final materials assessed were posted by public health entities at the state, national, and international levels.
Once the appropriate content was selected for study, we assessed the materials for readability, understandability, and actionability using standardized processes to promote interrater reliability. To assess readability, two trained staff cleaned each material (e.g., removed bullets and extraneous punctuation) and used Seven Formulas software (Micro Power & Light Co., Dallas, TX, USA) to generate results from three validated formulas: Flesch-Kincaid (Kincaid et al., 1975), SMOG (McLaughlin, 1969), and Fry Graph (Fry, 1968). We averaged the scores from the three formulas to arrive at a mean readability score for each material, and used those to arrive at a mean readability score for the entire sample. Further, we categorized each material and the sample into the "easy, " "average, " or "difficult" level 12 .
Recognizing the limitations of readability assessments to judge how understandable or actionable a material is (McGee, 2010), we used the Patient Education Materials Assessment Tool for Print Materials (PEMAT-P) (Shoemaker et al., 2013) to assess those domains. The PEMAT-P includes 17 items to assess understandability and 7 items to assess actionability and produces separate percentage scores for each area.
In accordance with the PEMAT User's Guide, two reviewers independently assessed each material using the PEMAT scoring sheet. For each item, each user assigned a score of 0 (disagree), 1 (agree), and where indicated by the User's Guide, N/A (not applicable). After independently reviewing each material and scoring it across the 24 items, pairs of reviewers met to review and discuss scores. Where reviewers did not agree on an item, the User's Guide was consulted as needed, reviewers looked at the material together, and after discussion arrived at consensus on a final score for each of the 24 items. Results were entered into an Excel spreadsheet, and authors used Excel features to generate mean scores for readability, understandability, and actionability across materials.

Key Findings: Readability
Using the Flesch-Kincaid, SMOG, and Fry readability formula results obtained from Seven Formulas software, we calculated the mean readability score across the 28-material sample at grade 10, which places the mean in the "difficult" category. Of the 28 selections tested, only 2 (7%) were assessed as "easy" (grade 6 or below). Another 10 (36%) were in the "average" range (grades 7 to 9), and the majority (16, or 57%) were assessed as "difficult" (grade 10 or above).

Key Findings: Understandability and Actionability
The PEMAT-P includes 17 items to assess understandability and seven to assess actionability, and it produces a percentage score for each domain. The higher the percentage score, the more understandable or actionable the material is. Our study revealed some important strengths and weaknesses of online consumer information about COVID-19. Results for selected PEMAT-P items are presented in Table 1 along with examples to illustrate each of these concepts. The number of materials scored for each item varied as not all PEMAT-P items were relevant to all materials.
The mean score for understandability across all materials was 70%. Overall, the sample scored well on the items related to quantitative expression. Most materials (92%) presented numbers clearly (e.g., used whole numbers rather than fractions or decimals), and all (100%) avoided requiring readers to perform calculations. Another strength is that 75% of materials presented the content in a logical sequence; that is, in the order in which readers would expect.
With respect to actionability, the mean score across the sample was 79%. Almost all materials (93%) included at least one action readers could take, and 64% addressed readers directly when giving instructions.
Unfortunately, there were also several concerning deficiencies in many of these materials. With respect to helping the public understand relevant information, just over half (58%) of the materials made the purpose of the material completely evident. Clearly stating the purpose in the title or introductory text tells a reader whether they are the intended audience and the essence of what they should learn by reading it.
We also observed that many materials failed to use plain language in their word choices. Plain language refers to writing that a reader can understand the first time they see it (Sunstein, 2011). The PEMAT includes two items to assess word choice: one to determine if medical jargon was used unnecessarily or without being defined and another to determine the degree to which common, everyday terms were used (e.g., "used" rather than "utilized"). A majority (57%) failed to meet one or both of these standards.
Fewer than half (39%) of the materials included visual images when such an image would have helped readers understand the information. User guidance from the PEMAT-P directs scorers to reflect a negative (disagree) score when the scorer can identify at least one image that, in included, would improve the likelihood of the reader understanding it. For example, we noted many references to maintaining a 6-foot space between persons. In our experience, many adults struggle to derive value from mathematical concepts such as measurements; thus, this is a concept that would likely be better understood with the addition of a visual image.
Although almost all materials included at least one action a reader could take, fewer than two-thirds (61%) included manageable, explicit steps to act on the instructions. However, for several of those that did include an explicit step, the step was simply to click a link to a different website or document.

PEMAT-P item
Materials that met the standard number (percent)

Explanations and examples Understandability (Strengths)
The material does not expect the user to perform calculations (n = 28) 28 (100%) None of the materials required the reader to add, subtract, multiply or divide, or perform any other mathematical operation.
Numbers appearing in the material are clear and easy to understand (n = 27) 25 (92%) Material should only use numbers when needed and when they are used they should be clear and easy to understand.
• Wash your hands with soap and water for at least 20 seconds • 15 days to slow the spread • Avoid social gatherings in groups of more than 10 people.
The material presents information in a logical sequence (n = 28) 21 (75%) Information in these materials were presented in a logical order with the most important information first and similar information grouped together.

Understandabilty (Weaknesses)
The material uses visual aids whenever they could make content more easily understood (e.g., illustration of healthy portion size) (n = 28) 11 (39%) Many materials would benefit from the use of a picture to show the reader: • How far apart six feet is (a commonly recommended distance to keep between themselves and others) • Samples of appropriate cleaning products and their labeling The material uses common, everyday language (n = 28)

Actionability (Strengths)
The material clearly identifies at least one action the user can take (n = 28) 26 (93%) Specific actions for the reader to take to keep from getting the virus were included in most materials. Examples included washing hands and staying home when ill.

Actionability (Weaknesses)
The material breaks down any action into manageable, explicit steps (n = 28) 17 (61%) Examples of actions recommended but not broken down into manageable steps and thus not fully actionable: • Clean and disinfect surfaces (without steps on how to do so) • Seek medical care (with no guidance as to whether to call primary care, urgent care, specialty care, or emergency care) The material addresses the user directly when describing actions (n = 28) 18 (64%) Example of failing to say "you" or begin a directive with an action verb: • The Department of Public Health recommends that people who have returned from traveling outside the state for business or vacation voluntarily self-isolate for 14 days following their return and monitor for fever and other symptoms

Public Health Impact
During public health emergencies, the public needs easy access to health information that is clear, meaningful, and actionable. Our results are similar to those of previous assessments of health information (Davis et al., 1990;Stossel et al., 2012;Haller et al., 2019;Prince et al., 2019) and suggest that even during a high-stakes public health threat, deficiencies in the quality of consumer health information are common. This is especially concerning given that people in high-risk categories for limited health literacy (e.g., older adults, people with chronic health conditions, and minorities) are also the people at high risk of experiencing the worst effects of COVID-19. These results also have implications for public health ethics. Like other health professions, public health is guided by a set ethical principles (Public Health Leadership Society, 2002;Thomas et al., 2002). These include equity, transparency, and trust. A common concern among public health ethicists is that standard medical models and practices disenfranchise certain populations, including the elderly (Shepherd, 2019), sex and gender minorities (Littlejohn et al., 2019), and racial and ethnic minorities (Thomas, 2019). Consider, for example, advance directives. While the aim of advance directives is to empower autonomy, many living will forms are "blunt" instruments that lack nuanced options for care. And forms that do attempt to provide more options "become increasingly legalistic, lengthy, and difficult to understand" (Shepherd, 2019, p. 186). People who struggle with health literacy are disproportionately disadvantaged by these obstacles.
Further, many such obstacles have led to suspicion and avoidance of well-ness and preventive care among these populations and are often cited as contributing to health inequities (Casagrande et al., 2007;Bauer et al., 2014;Weisz and Quinn, 2018). For example, LaVeist et al. (2009) found that mistrust of health services among African Americans led to significantly more underutilization of health services than their white counterparts.
Recognizing these obstacles places an additional responsibility on public health officials to reduce barriers to care and to rebuild trust within these communities. This means that a deficiency in the readability and actionability of public-facing health information is also a deficiency in responsibility to public health ethics and a missed opportunity to promote health equity. This responsibility is heightened in public health crises, like COVID-19, where populations already at risk for disenfranchisement are also at higher risk of the worst effects of the virus. However, one way to reduce barriers to care is to improve access to accurate health information by addressing the health literacy needs of populations at higher risk of harm from public health emergencies.

Limitations
Our study was limited by several factors. As our intent was to quickly assess and report on a sample of consumer health information to highlight opportunities to improve communication as the pandemic progresses, we selected a brief period in time and a small number of materials to include in our assessment; thus, the materials chosen may not fully reflect the quality of consumer health materials on this topic.
We gathered our data during the week of March 23, 2020, just weeks after the first case of COVID-19 was reported in the US. Given the lack of expert consensus on many of the issues related to COVID-19 at that time, it is possible that some writers of public health information were cautious about how much information and advice to include in the informational materials.
Further, given the disagreement among experts over certain kinds of advisable actions (e.g., whether the general public should wear masks and whether gatherings of more than two people are safe 14,15,16 ) it is not surprising that actionability among the materials was limited and that explicit advice may have been omitted intentionally.
Further still, as the pandemic progressed quickly, we acknowledge that for those working in all areas of the COVID-19 response, time was of the essence, and there was likely little time to apply standard editing processes or other routine approaches to optimize clarity.
Although we took steps to mitigate filtering biases in our searches (using multiple computers and searching in "incognito" mode), we acknowledge that our searches could have been influenced by our geographic location and the past search histories of users who work in health-related fields.
Finally, while the validated tools we used to assess these materials are considered the best available, they are not without limitations. Readability is not an exact science, and there is some interprofessional disagreement over how precisely to determine whether a word or phrase is understandable to the "average" person. Although we followed the PEMAT-P user instructions for scoring materials, some subjectivity remained. In those cases, we established a group consensus on how to score items. Lastly, a favorable score on the PEMAT-P does not guarantee that the material is of high quality.

Future Studies
Our findings are similar to many other reviews of consumer health information, which raises questions about how health communicators perceive the readability, understandability, and actionability of their own work. As there is currently no research on these perceptions, future studies could collect evidence about those perceptions and then compare them with the results of validated assessments of their work. When perceptions and formal assessments differ, interventions could advocate for the use of validated assessment tools and provide training for health communicators.
Investigators could also determine the degree to which efforts to ensure readable and actionable materials were employed, along with related barriers. This information could help communications teams plan ahead for providing clear health information during future events that require timely outreach to the public.
Further, while COVID-19 is genuinely novel, it is only one of many infectious diseases that health professionals address regularly, including tuberculosis, hepatitis, and Ebola. These illnesses warrant the time and effort required to develop messaging templates that are readable, understandable, and actionable. These templates would then allow for quickly 14 Quartz Media, Inc. (2020). Every expert opinion you've heard about wearing masks is right. Available online at: https://qz.com/1826717/do-masks-protectagainst-coronavirus/ (accessed June 7, 2020). 15 The Atlantic. (2020). Masks Are a Tool, Not a Symbol. Lives will be lost if Americans allow the culture war to determine whether they cover their face in public. Available online at: https://www.theatlantic.com/ideas/archive/2020/05/ masks-are-tool-not-symbol/611134/ (accessed June 7, 2020). 16 Global Policy Journal (2020). Why Experts Disagree on How to Manage COVID-19: Four Problem Conceptions, Not One. Available online at: https://www. globalpolicyjournal.com/blog/07/04/2020/why-experts-disagree-how-managecovid-19-four-problem-conceptions-not-one/ (accessed June 7, 2020). disseminating information about new public health threats. Our findings suggest that those creating such templates should pay special attention to: • Stating clearly the purpose of the material • Reviewing the information for plain language • Replacing or defining and explaining any medical terms • Using visual aids to make the material easier to understand or the actions easier to take • Giving explicit, manageable steps for any actions readers are asked to take

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

AUTHOR CONTRIBUTIONS
KL introduced the topic as a study of interest. KL and AC drafted study design with input from JW. KL and JW each drafted a section of the manuscript, reviewed it in its entirety, and approved the final copy. AC drafted remaining sections, edited all sections, and provided oversight to the assessment and writing process. All authors participated in the use of the Patient Education Materials Assessment Tool to formally assess online materials.

FUNDING
Authors nor their institution received any funding in direct support of the work described nor of the preparation of the brief research report, outside of salaries paid to each author by the institution.