An Examination of the Clarity of Government Health Websites Using the Centers for Disease Control and Prevention's Clear Communication Index

Introduction

Navigating online public health information can be difficult for individuals, especially those with low health literacy skills. Individuals with low levels of health literacy are likely to lose interest and stop processing complex health information after seeing multiple lengthy paragraphs. Poorly designed public health websites combined with low health literacy skills can be a major barrier to disseminating relevant health information to public audiences. Health literacy goes beyond basic reading and writing skills for most patients. Limited health skills are one of the strongest indicators of poor health outcomes for patients (U.S. Department of Education, National Center for Education Statistics, 2014). Websites can come with their own complicated technical vocabulary and jargon that can further complicate many individuals' ability to fully comprehend medical information presented to them online (Kreps, 2017). After years of education and training, health care professionals are in the small minority of people with high-level health literacy skills. The entire health care system relies on the assumption that the public can understand complex written information. This is not an accurate assumption since patients often do not fully comprehend complex medical information provided to them (Kreps, 2018).

Health information websites are important sources of health education for the public. The public can choose reputable, relevant health material from these online health information systems and, in turn, the systems should provide information to the public in ways that help increase their understanding of relevant health topics. It is one thing to supply health information online, yet another to communicate the information in ways that can build technical familiarity, learning skills and communication skills to empower the public to manage and improve their health (Butterworth et al., 2010). Designing health information systems to communicate in ways that can enhance consumer understanding is imperative to make an impact on the public's health (Kreps, 2015, 2018).

Since government health websites use various design formats to publish health information for the general public it is important that the information published provides information support, ease of use, a reduction in misunderstandings and is perceived as useful to the consumer (Kreps, 2015). Therefore, the purpose of this study is to evaluate and compare differently formatted government health websites using an established measurement index designed by the CDC in 2014 (Centers for Disease Control Prevention, 2014). The research goals in this study are to assess and help to improve the clarity of online communication products by determining how well the websites score for clarity and identify ways to improve the websites.

One of the more difficult tasks in developing public health websites is to describe all of the information about the medical issue in such a way that it is relevant and understandable to the public (Alpert et al., 2017). According to the United States Department of Health and Human Services (HHS) (U.S. Department of Health and Human Service, 2008), only 12% of U.S. adults had proficient health literacy skills (Table 1) and a third of U.S. adults (77 million people) would have difficulties with common health tasks, such as following directions on a prescription drug label or adhering to a childhood immunization schedule using a standard chart. Information presented from health professionals is one of the most important sources of information on health topics for all health literacy levels (U.S. Department of Health and Human Service, 2008). Simply designating a reading grade level for print material is not an effective standard for assuring that web-based health information will be understandable to most consumers. Therefore, online sources of health information must be designed to be understood and should be periodically tested to make sure they are effective at communicating health information to intended audiences (U.S. Department of Health and Human Service, 2008).

TABLE 1

Table 1. Descriptions of health literacy.

This study is not about challenging the credibility of the United States Department of Health and Human Services' (HHS) information. It is about evaluating message design to determine if relevant health information is presented clearly on government health websites. This study attempts to evaluate the effectiveness of the communicative design of these websites as important public health education tools (Kreps, 2014b). The competent communication design of major health websites is an important part of disseminating relevant health information to the public and informing health decision making (Kreps, 2015).

The HHS administers more than 100 health research and promotion programs across its operating divisions (U.S. Department of Health and Human Services, 2018b). These programs are designed to protect the health of all Americans and provide essential human services, especially for those who are least able to help themselves (U.S. Department of Health and Human Services, 2018b). The Office of the Assistant Secretary for Health (OASH) oversees 12 core public health offices (U.S. Department of Health and Human Services, 2018b). The following offices under the Assistant Secretary for Health are referred to as cornerstones for the delivery of public health services (Table 2) (U.S. Department of Health and Human Services, 2016). These offices have digital interventions that have the capabilities to offer computer-mediated forms of health information and support that could bring about meaningful changes in one's health-related outcome (U.S. Department of Health and Human Services, 2018a). The degree to which the computer-mediated information and support is effective depends on the design and implementation of web-based health information technology, as well as the resources and development that goes into building their respective websites (U.S. Department of Health and Human Services, 2018a). Ten out of 12 of these offices have websites that are available online, and managed through a Digital Council to ensure quality, efficiency, and user focused communications and operations (U.S. Department of Health and Human Services, 2018a). HHS's Digital Communications includes a Plain Writing Checklist (Table 3) for building and managing its websites (U.S. Department of Health and Human Service, 2015). The HHS's checklist is listed under the following five categories: Audience, Structure, Writing, Punctuation and Linking and Reviewing (U.S. Department of Health and Human Service, 2015).

TABLE 2

Table 2. HHS's health offices and agencies, abbreviations, and date accessed.

TABLE 3

Table 3. Building and managing website criteria.

The Plain Writing Act of 2010 requires all United States government agencies to write and present information using clear communication that the public understands (U.S. Department of Health and Human Services, 2011). The official guidelines for the Plain Writing Act of 2010 assist federal agencies in writing clearly so users can find what they need, understand what they find and use what they find to meet their needs (U.S. Department of Health and Human Services, 2011). It applies to all federal agencies including the HHS which encompasses CDC, NIH, FDA, AHRQ, HRSA, and the CMS (U.S. Department of Health and Human Services, 2010).

In an effort to improve health literacy and informed decision-making the CDC developed the Clear Communication Index (CCI) in 2014 as research-based criteria to develop and assess public communication products (Centers for Disease Control Prevention, 2014). This content analysis study evaluated the 10 HHS core public health office websites under the OASH using the CDC's CCI to measure the clarity of the content of these websites and to determine if they are designed and organized in ways that are easily understandable (score high on the CCI) to the general public. In addition to the 10 HHS core public health office websites, six HHS agencies which partner with the NVPO (CDC, NIH, FDA, AHRQ, HRSA and CMS) were also evaluated using the CCI. This was done as an indicator of the effectiveness of the digital communication methodology between HHS's Plain Writing checklist which structure the core public health office websites, and the HHS agencies which use alternative guidelines. Findings will help determine how effective these important government websites are in disseminating health information in public audiences, as well as best practices for creating public health websites content suitable for patients of all literacy levels.

There are a variety of quality scales to evaluate health websites. Daraz et al. (2019) accessed health information through the Internet to evaluate for reliability and accuracy using the NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies. Although they accessed various online providers, their results found that by affiliation government websites scored higher than academic and other media sources (Daraz et al., 2019). Gill et al. (2012) assessed the readability, suitability, and usability of health care literature associated with traumatic brain injury published by the CDC using the Flesch-Kincaid Grade Level, Flesch Reading Ease, Gunning Fog, Simple Measure of Gobbledygook, and Suitability Assessment of Materials (SAM) indices. Sak et al. (2016) performed a content analysis of pro- and anti-vaccination websites using frequency counts, cross tabulations, Pearson's chi-square and other inferential indicators on 1093 webpages from three search engines. Baur and Prue (2014) tested the validity and reliability of the CCI using surveys to evaluate information clarity and audience comprehension of actual CDC material along with the same material redesigned to address the Index question and items. Alpert et al. (2017) evaluated portal pages of MyPreventiveCare using the CDC's CCI. To date, there are few comparative content analysis studies of government health website information as well as studies utilizing CDC's CCI. This study appears to be the first content analysis study to analyze the U. S. government's health websites using the CDC's well-established CCI.

Literature Review

Meaning-Centered Communication

There is a growing need to find a framework that establishes a connection between health communication design and health education. Allen et al. (2017) build on the framework of the practice of health communication (perceived messages), health literacy (shared initiatives) and health education (interactive and perpetual) professionals. Allen et al. (2017) introduced the formation of the new Action Collaborative on Communication and Education for Health under the auspices of the Roundtable on Health Literacy of the National Academy of Sciences, Engineering and Medicine which promotes an integrated approach to health literacy, health education, and health communication to improve health and well-being. All three disciplines rely on evidence-based material and recognize that efforts to improve health and eliminate disparities requires professionals to understand audiences, share knowledge, cross cultures and develop strategies to engage the vulnerable (Allen et al., 2017). The framework introduced by Allen et al. (2017) revolves around the philosophy of meaning-centered communication, as well as rhetorical communication. Meaning-centered communication refers to the intentional, purposeful awareness of the perceived context of both the sender of the message and the receiver (Allen et al., 2017). Meaning-centered views communication as the outcome of a process and recognizes that people have different personalities, cultures and experiences and operate using the idea of perception (Allen et al., 2017). A critical part of meaning-centered health communication involves designing health messages in ways that are easy for audiences to understand. It is imperative that federal government health websites follow this advice in designing clear and easy to understand health messages about important health information.

Health Literacy

The 2003 National Assessment of Adult Literacy found that approximately 50% of the U.S. population has inadequate literacy suggesting that a large proportion of the US public is likely to have problems with health literacy (Graham and Brookey, 2008). Healthcare institutions and public health care professionals have an ethical responsibility toward the patients they serve, especially vulnerable populations affected by low health literacy skills. There are many determinants that affect health literacy, including communication skills, knowledge, and/or beliefs about health topics, cultural, socioeconomic status and functional literacy (Parvanta et al., 2018). These factors interact at all levels, ranging from individual factors to the surrounding community, and to policies that shape the healthcare system. According to studies, patients remember only 50% or less of what healthcare specialists teach them (Ong et al., 1995). People with limited reading ability will have difficulty navigating online health information and those with limited writing skills will be less likely to fill out forms correctly (Parvanta et al., 2018). Incorporating plain and clear language is important since one out of five Americans read at a fifth-grade level or below and most health-related information is written at the tenth-grade level (Point-of-Care-Partners, 2012). Patients with low health literacy are more likely to use passive methods of communication, do not participate in the shared decision-making process, and as a result face difficulty in their interactions with their healthcare providers (Reisi et al., 2014). Effective communication is the foundation for increasing low health literacy skills (Adams and Corrigan, 2003). Establishing effective communication through government public health websites increases confidence in the public which in turn increases the health system's efficacy and patient satisfaction with healthcare (Adams and Corrigan, 2003).

Language

Health is a culturally constructed phenomenon with language and culture providing a context for understanding health information. In terms of health literacy, knowledge is what a person knows, and culture describes what a person believes or values. The 2010 census of the U.S. population found that 60.6 million people, or 21% of the U.S. population 5 years or older, spoke a language other than English at home (Parvanta et al., 2018). Some individuals may be perfectly literate in their native language, and possibly have a high degree of health literacy, but have limited ability to communicate in English (Parvanta et al., 2018). HHS's OMH developed a standard set of language access services for healthcare providers and medical facilities per Title VI of the Civil Rights Act of 1964 called limited English proficiency (LEP) (U.S. Department of Health and Human Services, 2013). It is very likely that individuals with low levels of English language proficiency will have difficulties understanding information presented in English on health websites.

In 2010, the HHS's ODPHP proposed a National Action Plan to Improve Health Literacy (NAP) (Parvanta et al., 2018). NAP offers strategies and goals that healthcare professionals and organizations can use to transform jargon-filled, dense, complex and lengthy health information into clearer, more easily understood messages (Parvanta et al., 2018). Healthy people 2020, a federal government prevention agenda for building a healthier nation, considers health literacy one of the main priorities for improving the quality of health services (Office of Disease Prevention Health Promotion, 2018). Supporting a shared decision-making process between patients and providers about health and behaviors can be shaped by the communication, the information and the healthcare provider (Office of Disease Prevention Health Promotion, 2018).

Working together, professionals with training in the disciplines of health communication, health literacy and health education can help to design evidence-based web-based health information materials to improve health and eliminate disparities to share knowledge; across cultures and develop strategies to engage the vulnerable (Allen et al., 2017).

R1, Are HHS websites designed to provide consumers with information support relevant to their health? The Social Support Theory was selected to underpin research question 1.

R2, Are HHS websites designed to demonstrate communication competence in providing easy to understand information to consumers? The Relational Health Communication Competence Model was selected to support research question 2.

R3, Are HHS websites designed to reduce consumer misunderstanding about health issues by communicating clear health information? Weick's Model of Organizing was selected to support research question 3.

R4, Are HHS websites designed to demonstrate perceived usefulness in providing ease of use in the design of the home page using CDC's CCI? The Technology Acceptance Model was selected to support research question 4.

R5, Are HHS websites designed to demonstrate perceived usefulness in providing ease of use in the design of the home page using HHS's Plain Writing Checklist? The Technology Acceptance Model was selected to support research question 5.

Social Support Theory

Social Support Theory is drawn from the proposition that instrumental, informational, and emotional support reduces stress and boosts self-esteem through self-awareness that a person receives from support from credible resource (Vaux, 1988). Health is a personal and emotional issue and serious health challenges can be emotionally and physically draining (Kim et al., 2015). Public health websites are considered traditional health information websites which spread static information and focus on health issues and informational content (Kreps, 2015). Web-based health information that forms a support system which enables better health displays the importance of social support for better conditions and well-being. This study attempts to show how effectively these websites communicate relevant information to support the health needs of the consumer.

Relational Health Communication Competence Model

The Relational Health Communication Competence (RHCC) model was introduced in 1988 by the health communication scholar Gary Kreps (Kreps, 2014a). The model has been widely used to describe and predict the influences of the consumer-provider communication on important health outcomes (Kreps, 2014a). The RHCC model is represented by a wagon wheel with the hub representing the consumer and the spokes representing the health care providers (Kreps, 1988). The model suggests that the level of communication competence exhibited by those delivering care is directly related to their abilities to establish cooperative health care relationships, share relevant health information, make informed health care decisions and coordinate activities to achieve the desired health outcome (Kreps, 2014a). If the competence decreases it tips the wheel toward pathological communication, which results in a lack of social support, dissatisfaction, information barriers and a lack of cooperation (Kreps, 1988). Increasing competence results in the therapeutic communication of social support, satisfaction, information exchange and cooperation (Kreps, 1988). Although the RHCC does not refer to the Internet as an influence on an individual's important health outcomes, the level of communication competence exhibited by the appropriate design computer-mediated health interventions for different audiences are directly related to the ability to share relevant health information so that the individual accessing the information can make informed health care decisions, thus allowing the person to make informed health decisions and coordinate health promoting activities to achieve desired health outcomes. For example, a specific computer-mediated intervention type called Internet-based psychological intervention (IPI) refers to psychotherapeutic treatment delivered via the Internet in a series of sessions that emulate face-to-face psychotherapy and delivered via the Internet through websites and online programs (Shim et al., 2017). If users of these IPI treatment interventions cannot understand the information provided, these systems will not be very effective in helping users achieve their mental health goals. Similarly, advances in digital information and communication technology through public health websites offers a means of improving access to health care information to consumers but can only be effective if the consumers can understand and use the health information provided.

Another strategy to apply to computer-mediated health information is clear and relevant information so that the consumer can understand the message. Alpert et al. (2017) examined patient portals and found that many health information websites use overly complex language and do not include explanations and examples that could assist the consumer in achieving a clear understanding of the health information provided. Consumers also want specific, relevant health information (Alpert et al., 2017). The consumer would be better served if health information websites target the users' literacy levels by using language that is easy to understand and apply using visual examples (Kreps and Neuhauser, 2010). Illustrations are also useful when consumers possess limited health literacy levels for increasing understanding and recall of health information (Meppelink and Bol, 2015).

Weick's Model of Organizing

Weick's model of organizing is used in health communication and to examine health care and health promotion decision making (Kreps, 2009; Wilson, 2013). In “An Introduction to Organizing,” Weick uses baseball umpires to illustrate the important role people play in creating their own environment. The umpire who says, “They ain't nothin' till I calls them” (Weick, 1969, p. 5), demonstrates that even though organizations, as well as individuals, are preoccupied with objectivity and concreteness they are actually permeated with subjectivity and guesses. In addition, the informational inputs which organizations need to address are often uncertain and equivocal with many sets of outcomes that might occur (Weick, 1969). Organizing the range of possibilities reduces the number that may occur and maintains a workable level of certainty as organizations tend to transform equivocal information into unequivocal information that they are familiar with (Weick, 1969). Nowhere is this high level of equivocality as problematic as in addressing complex and uncertain health problems! Health care consumers (and many health care providers too) often need help to make sense of complex health problems to make accurate diagnoses and provide appropriated treatments for complex health problems. If health information is not provided in ways that consumers (and providers) can easily understand, the information will not be very useful in helping to increase understanding and guide effective responses to health problems.

One of the strengths of Weick's model of organizing is that it focuses on the process of communication instead of the role of individual actors, making it more holistically based and an ideal model for this study. Weick's model deals with the complexities of communication information organization instead of trying to understand people within a group or organization and also focuses on big picture organizational problems, which is different than most other approaches. Weick's model proves to be advantageous in the health care and health promotion setting due to the double interact of professionals in an interdependent hierarchy. Due to the importance of reduction of equivocality, Weick's model is extremely useful in strategic communication. Weick's model suggests that health information communication systems, such as websites, need to be designed to provide users with clear and relevant information to assist in guiding health decision making.

Technology Acceptance Model

Davis et al. (1989) proposed the Technology Acceptance Model (TAM) to explain and predict behavioral intentions of users with respect to evaluating digital interventions' “perceived usefulness” and “perceived ease of use.” There are six perspectives in the model's architecture: “perceived usefulness,” “perceived ease of use,” “attitude,” “behavioral intention,” “actual use,” and “external variables” (Davis et al., 1989). This study is adopting the TAM's “perceived usefulness” and “perceived ease of use” perspectives of this model to investigate whether consumers will adopt the websites to help them achieve their health information goals if they do not find them to be easy to understand and particularly useful.

Statement of Method

This study employs a content analysis of health information websites using the CDC's CCI to evaluate the presentation of information on these websites. Content analysis, as a methodology, is used widely to examine the characteristics of messages to understand patterns of messages, usually for audience effects (Thayer et al., 2007). It is a research method that is verifiable by observation or experience rather than theory or pure logic (Thayer et al., 2007). Content analysis is used in the realm of health behavior because of the assumption that messages and content will influence an audience's beliefs, attitudes, intentions, and behavior (Kreps, 2011).

Instrumentation

The CDC developed the CCI to comply with the Plain Writing Act of 2010 and to achieve goals set forth by its own National Action Plan to Improve Health Literacy (NAP) (Centers for Disease Control Prevention, 2015). NAP is based on two core principles: that all people have the right to health information that allows them to make informed decisions and that health services should be delivered in ways that are easy to understand and improve health, longevity and one's quality of life (U.S. Department of Health and Human Services, 2011). CDC's CCI's development, reliability and validity testing was constructed by the CDC's Office of the Associate Director for Communication (Centers for Disease Control Prevention, 2013). Interestingly, the U.S. Department of Health and Human Service (2015) builds and manages its websites using a checklist (Table 3) designed by its information technology department, Digital Communications Division (DCD), which is similar to the CDC's CCI but unlike the CCI, HHS's checklist is not designed as a reliable and valid measurement tool.

CDC's Clear Communication Index

This study used a content analysis method, the CDC's CCI research tool to evaluate the understandability levels of 10 office and six agency's health websites, under HHS's OASH, designed for patients to have access to information about health interventions. The objective of this content analysis study was to measure the content of the HHS's 10 office websites and six agencies in terms of the clear communication of the health information. Findings will help determine best practices for creating website content suitable for all health literacy levels, identify modifications to improve the index and build on previous literature in regard to the importance of using plain language in public health websites. The CCI is constructed of four main categories used for assessment: Core, Behavioral Recommendations, Numbers, and Risk (Baur and Prue, 2014). The Index consists of 20 items, separated into these four sections that build on and expand plain language techniques described in the Federal Plain Language Guidelines which can be found in Table 6 (Centers for Disease Control Prevention, 2015). Each question receives a score of 1 or 0, depending on whether the criteria was or was not present (Centers for Disease Control Prevention, 2014). The highest score that can be attained is 20/20 (Centers for Disease Control Prevention, 2014). The total score is converted to a scale of 100 where a score of 90% or higher is considered excellent and a score of 89% or below indicated that the material may need to be revised to make it clearer for the priority audience (Centers for Disease Control Prevention, 2014). Baur and Prue (2014) tested the validity and reliability of the CCI and found that when materials were revised based on the CCI, they were more easily understood. Alpert et al. (2017) demonstrated that applying the CDC's CCI to patient portals can be used to identify opportunities for better patient communication and engagement. Alpert's et al. (2017) findings indicated that specific changes to improve patient portals included simpler language, more specific examples, and clearer numerical explanations.

Analytical Process

A total of 48 website pages, one home page and two additional pertinent pages, from each website were selected from the 10 online offices and six agencies under the OASH. For example, NVPO's website has a total of 36 pages with some pages that contain information pertaining to the members, meetings, minutes, reports and policy and, for this study, are not considered to be pertinent pages. Home page is defined as the main page a visitor navigates to the website from a web search engine. Pertinent pages refer to informational pages on specific vaccines, diseases, health, etc., that the public would view and does not include the organizational structured materials. In addition to a total score, pages last updated dates were documented. After accessing NVPO's home page the “Featured Priorities” icon was accessed and the first two priorities (adult immunization and HPV vaccination) were chosen as pertinent pages. Pertinent pages were chosen for the rest of the offices and agencies in a similar manner and listed in Table 4. Each page was evaluated and scored using the CDC's CCI.

TABLE 4

Table 4. Pertinent pages.

Two coders discussed the method they would use in perceiving content as it related to the scoring index. To test inter-coder reliability, a subsample of three additional pertinent pages were coded independently by both coders (Wimmer and Dominick, 2013). The two coders then discussed the differences in their scores and determined the reason for any differences. ReCal2 (Reliability Calculator for 2 coders) is an online utility that computes intercoder/interrater reliability coefficients for nominal data, used for labeling variables without any quantitative value, coded by two coders (Freelon, 2010). ReCal2 calculates four of the most popular reliability coefficients for nominal data including: percent agreement, Scott's Pi, Cohen's Kappa, and Krippendorff's Alpha (Freelon, 2010). This study chose to use Krippendorff's Alpha as opposed to Cohen's Kappa because Krippendorff's Alpha has been documented to provide stable estimates in the case of possible missing data which is foreseeable due to the way the CCI score sheet is structured (Zapf et al., 2016). Krippendorff was also a well-known content analysis expert and notes that its use dates back to the inquisitorial pursuits by the Church in the 17th century (Thayer et al., 2007).

Intercoder reliability was tested using ReCal2 for two coders (Freelon, 2010). Krippendorff's alpha for Variable 1 (Core) was 0.803, Variable 2 (Behavioral Recommendation) was 0.78, Variable 3 (Numbers) was 0.78, and Variable 4 (Risk) was 0.79 (Table 5). The Krippendorff's alpha for total scores was 0.843. Each part of the CCI (e.g., CORE) includes multiple question and intercoder reliability was assessed for each question/variable (Table 6) and Krippendorff's alpha for total score for each question was 0.91.

TABLE 5

Table 5. Total scores = 0.843 (4 variables).

TABLE 6

Table 6. Total scores = 0.91 (20 variables).

According to Keyton (2006), a reliability coefficient of 0.70 or above is adequate for establishing intercoder reliability. The subset of 48 website pages were split evenly between the two coders and individually scored.

Results

Scores from the CDC's CCI are interpreted by total score. Total scores above 90 are interpreted as excellent and have addressed most items that make materials easier to understand and use. Forms that score below 90 are recommended to refer to the User's Guide to revise and improve the material. HHS's Plain Writing Checklist, developed in 2015, addresses hyperlinking under its Punctuation and Linking section while CDC's CCI's questions do not. For the purpose of this study, hyperlinks were considered on certain questions. Most of CCI's questions consist of multiple variables within a variable (question) but question five addresses multiple variables leaving it vulnerable to misinterpretation and possibly outdated with respect to digital communication. Question five asks if the material includes one or more calls to action for the primary audience which refers to a specific behavioral recommendation, having a prompt, a request for more information, a request to share information or a broad call for program or policy change and should possibly be addressed in multiple question to avoid confusion and increase clarity. Additionally, identifying a behavioral recommendation is relevant to Part B and a prompt or request for more information and/or a policy change may not be, depending on the circumstances. This content analysis study considered a link as both a prompt to get more information and a request to share information and validates a links use in question five. Hyperlinks were not identified as appropriate for questions one though four since they were specific to the main message located at the top of the main page. Since hyperlinks are prompts, by definition, questions six through 20 also validate their use, unless there is no main message whereas question six is scored as a zero. If there is not a main message question six is scored as a zero because it requires an active voice in both the main message and the call for action. Additionally, other than the OSG, who utilized multiple links to NIH, the rest of HHS's offices used internal links. This study accessed only internal links and excluded external links. The results show that only two offices scored below a 90%, OSG scored 88% and OWH scored 88%. A comparison of HHS's core public office websites and HHS's agencies average scores can be found in Graph 1.

GRAPH 1

Graph 1. Average scores.

When answering questions one through four it is important to note that if the material contains more than one message there is no obvious main message and CCI's user guide instructs the scorer to answer “no.” If the answer is “no” for question one then, according to CCI's user guide, the answers to questions two through four are also “no.” OWH and OSG, the only two to score below a 90% scored low on their home page due to the fact that there was no main message and instead there were multiple messages, each with their own merit, and resulted in an immediate four-point deduction. Five out of six of the agencies lost points on their home pages because they had multiple messages, which is not necessarily a bad thing, yet they all managed to score a 90% or above because all of their pertinent pages scored 100%. CMS had a main message that was relevant to their mission of Medicare, Medicaid and Health Insurance Exchanges and was the only agency to score a 100% on all three pages.

With the exception of OMH, home pages were either up to date (current) or were not displayed and classified as a no date. While only two HHS's offices had current updated home pages, four out of six agencies had current updated home pages. Additionally, the HHS offices had pertinent pages that were last updated in 2016, 2017, and 2018, while the agencies did not, they were either not dated or dated from 2019 or current daily updates. Eight out of the 10 HHS offices did not have a date on their home page, however, the information on their home pages, for the most part, had one main message and it was usually information on the specific function of that office. The NIH is the only agency that did not date their home page. The pages last updated list is included in Table 7.

TABLE 7

Table 7. Pages last updated.

Research question one addresses if the HHS websites are designed to provide consumers with information support relevant to their health. The results provide support that these websites do provide relevant health information to consumers. With only two offices scoring below 90% for total score, mainly due to multiple messages on their home page instead of one main message, is evidence that the HHS websites are designed to provide consumers with information support relevant to their health (Graph 2). The agencies are especially useful because they are up to date and all of their pertinent pages attained perfect scores on the CDC's CCI. The information found on these web pages communicate instrumental, informational, and emotional support which reduces stress and provides the consumer with a sense of encouragement.

GRAPH 2

Graph 2. Provide support relevant to health.

The second research question asks if the HHS websites are designed to demonstrate communication competence in providing easy to understand information to consumers. The results provide support that these websites present easy to understand health information. Five of the six agencies home pages lost points because they contained multiple pertinent messages for the consumer on health subjects yet maintained a passing score. When the pertinent pages are scored independently of the home page all of the scores are above 90% with four of the 10 offices and all six agencies scoring 100%, Graph 3, which indicates that the level of communication competence exhibited in these websites is directly related to their ability to establish a cooperative health care relationship, share relevant information to make informed health care decisions and have the ability to help coordinate activities to achieve a desired health outcome (Kreps, 2014a).

GRAPH 3

Graph 3. Demonstrate communication competence.

The third research question addresses if the HHS websites are designed to reduce consumer misunderstanding about health issues by communicating clear health information. The results provide support that these websites are designed to reduce consumer understanding by providing clear health information. Pertinent pages, as well as some home pages, incorporated message systems that targeted many different literacy levels such as understandable language, although it is difficult to show statistically due to the main message and question 6 previously mentioned (Graph 4). Illustrations were used as well as an active voice on the majority of the pages. Vivid examples were present such as visuals, videos and translations. Where there was a behavioral recommendation, its importance was explained, and specific direction were included about how to perform the behavior. Numbers were explained simply and without the need to conduct mathematical calculations. Risks and benefits were explained of a recommended behavior and probability information was explained using text or visuals. The websites focused on the process of communication for all levels of health literacy instead of tailoring it to a specific audience. Using these strategies helps health consumers make sense of the equivocal information. These websites reduce equivocality with reliable and accurate interpretations which advocate behavioral recommendations for the consumer to improve their health and well-being.

GRAPH 4

Graph 4. Provide clear health information.

The fourth and fifth research questions addresses if the HHS websites are designed to demonstrate perceived usefulness in providing ease of use in the design of the home page using CDC's CCI and HHS's Plain Writing Checklist (Table 3). The results provide support that these websites do provide perceived ease of use for consumers. HHS's Plain Writing Checklist is not as specific as CDC's CCI and home pages are difficult to compare relative to how they are formatted, whether or not they contain a main message, or multiple messages, and one's interpretation of an active voice (Table 8). The purpose of comparing CCI's home page to HHS's is to illustrate how interpretation can influence a score. Whereas the HHS's Plain Writing Checklist is simply written using one variable for each question, CDC's CCI is written in a much more rigid, complex way (as demonstrated in Table 6). When the two pertinent pages were averaged using CDC's CCI the results were significant, and an indicator that the HHS websites, in relation to pertinent information pages, are designed to demonstrate perceived usefulness in providing ease of use (Graph 3).

TABLE 8

Table 8. Home page scores.

While most consumer who seek online health information use either a desktop or a laptop to access information the number of users accessing online health information on a mobile device, such as a smartphone or a tablet, has dramatically increased (Youngblood, 2018). When the acronyms for the offices and agencies were searched in google on an iPhone there where only three offices that did not appear as the first choice on the screen: OAH (5th), OPA (18th) and OSG (50th +). This is also an indication that these offices and agencies are well searched since they appear immediately. All of the offices and agencies' sites were mobile ready and did not reformat or need an app to access it which confirms its ease of use pertaining to TAM.

Discussion And Implications

This study suggests that generally the HHS websites that were analyzed are well-designed and communicate information in meaningful and effective ways. Analysis of the websites support the desired goals embedded in each of the five research questions examined in this study. The data suggest that all the websites that were analyzed provide relevant health information, easy to understand health information, clear health information, and should be easy for consumers to use. However, there are some concerns suggested by the research in the design of the way information is presented on the home pages of several of these websites.

In order to score high on the CDC's CCI, a page needs to have one main message, preferably at the top of the page, and/or a call for action. Additionally, the main message and call for action need to have an active voice. Although this may be a very sound way of delivering a message and a call for action, most effective home pages of HHS's offices and agencies have multiple messages which speak in both an active and passive voice. A home page is not supportive of having a call for action messages and if they are present most are spoken in a passive voice. Call for action messages located on pertinent pages are more prone to use an active voice in their call compared to a home page.

A good home page design should act as a virtual receptionist. The home page serves as a first impression for a viewer and receives the most traffic out of all the pages. The home page is designed to make an impression and if the first impression is not good then the consumer will probably move to an alternative site that gives them more options. The home page should attract and captivate, educate, and encourage consumers to visit other pages on the website. Home pages that contain multiple health related messages and subjects offer a means of improving the accessibility to deliver health care information to the consumer and provide easy navigation that increases competence which results in the therapeutic communication of social support, satisfaction, information exchange and cooperation.

Updating information on pertinent pages is extremely important. Websites that have current dates are much more believable and credible to the consumer. For example, NVPO's web page on adult immunizations has not been updated since February 5, 2016 and may contain irrelevant information while leaving out new information (Table 7). With the current concern of the measles epidemic in unvaccinated children it should be a top priority of NVPO to update the adult immunization web page with warnings, suggestions, and a call for action to the adult population. The CDC's website page for measles has an abundance of information and is currently updated. Not updating pertinent pages makes the office and/or agency look uncertain and lacking expertise. It is worth noting that if one accesses the CDC's CCI user manual it has not been updated since May 15, 2015, almost 4 years ago.

Despite not updating pages with pertinent information, formatting, and design, the pertinent page statistics indicate that the HHS websites are designed to provide consumers with tangible aids and information that is useful for self-evaluation relevant to their health. The pertinent pages scored much higher than the home pages did which suggests an alternate scoring system should have greater detail for the esthetics of a site, the use of color, the size and font of the text, and images and icons. Heron et al. (2017) found that individuals with low health literacy and digital literacy levels indicated that the esthetics of the site related to the size of the text, type of graphics, colors, icons and pictures, how interactive the website was, ease of physically accessing, the information provided and its trustworthiness were factors that most facilitated their use. Overcomplicated text that did not allow interactions with the content did not engage those with lower levels of digital literacy and digital literacy levels (Heron et al., 2017). While trusting the source of the information was thought to be relevant, it was not a priority for increasing website accessibility (Heron et al., 2017). Gonzalez et al. (2018) found that consumers who used technology for health-related purposes had a direct relationship between positivity and prior experiences with online searches and an indirect relationship to online use concerns like privacy and confidentiality. Confidentiality is the basis for trust and can be defined as the promise or duty to protect informational privacy (Neitzke, 2007). All of the HHS's offices and agencies are some of the most trusted government websites that exist today. Although the health consumers would like to trust their source of information it is not necessarily trustworthiness that draws them to their site. Youngblood (2018) examined the accessibility and mobile readiness of 25 of the top health information website homepages. NIH was ranked number one while seven institutes at NIH were also in the top 25. The CDC was also recognized as number six according to Alexa's Health Category listing (March 3, 2017) (Youngblood, 2018). Most people with low health literacy should be looking at the HHS's webpages because they are extremely readable, reliable, and trustworthy.

The properties and characteristics of the HHS's webpages are complex adaptive systems that include elements interacting dynamically, non-linear interactions, openness, and with a constant flow of energy. These websites, like complex adaptive systems, do not have a hierarchy of command (main message). They are constantly reorganizing (updating) themselves to fit into the environment better. Requisite variety suggests the greater the variety within the system the more likely it can create new possibilities and co-evolve.

The HHS websites show people interacting, thinking and smiling. They are simplistic in nature which draws on the consumers curiosity, confidence and comfort level. A photo of eggs in an egg crate is a prompt to educate someone on the risk of Salmonella infection from eggs and spraying a hand with an aerosol to prevent bug bites that may cause disease. Research is indicated with odd shaped and textured organisms that appear unique, interesting and to some possibly cute or scary. They have information for dog owners about genome editing, vitamin D toxicity, and heart disease. They all contain very valuable tools for the public. They have videos and interactive features and hopefully one day they can add chatrooms and other interactive devices so that people can get faster information instead of emailing or calling for information since most people probably do not follow up with emails or calls.

HHS's Plain Writing Checklist under the Structure category states, “Organized content by questions and answers when possible?” This would be an excellent presentation strategy for these websites to incorporate, yet most of them do not. The FDA uses this approach on occasion for some of their pages, for example on their webpage examining “E-liquid or cough syrup?” they placed a photo of both below the question and then below the photo they answer the question with “warning letters issued for E-liquids that look like cough syrups.” However, the vast majority of the websites do not organize information provided by question and do not provide answers to important health questions. Instead, these websites stick primarily to just presenting the known facts.

An effective health website also needs to be esthetically pleasing and useful in the sense that it is instrumental, informational and useful for self-examination. It needs to demonstrate communication competence by providing easy to understand information to the consumer that is clear and relevant and reduces users' equivocality about the health issues covered to assist consumers in making relevant health decisions. Equally important is that effective webpages, such as these HHS offices and agencies sites, need to be designed to demonstrate perceived usefulness in providing ease of use to all audiences.

This study is unique in that to date there are no other identifiable studies specific to the analysis of U.S. government health web sites. Castro-Sánchez et al. (2015) studied the readability of specific Ebola information on international public health agency websites which included the CDC using seven other scales rather than the CCI. Menchaca et al. (2017) reviewed patient-level information about the Zika virus and perinatal risks on state department websites in the U.S. and its territories. They found that 96.4% of the 56 state/territory health department websites contained patient information about Zika virus (Menchaca et al., 2017). Manganello and Fishbein (2009) found that less than half of published content analysis studies mention theory in an explicit way making this study strong since it relied on four theories of logic.

Limitations

A major limitation of this study is that it depended entirely on content analysis of the information provided on the websites to evaluate the websites and did not test consumer responses to the websites. It is important to test consumer's experiences with and evaluations of using these websites to determine how well the information provided on these sites meet the needs and expectations of users. It would be especially relevant to compare responses to the websites by different groups of consumers based on demographic factors such as age, gender, cultural and ethnic backgrounds, educational levels, educational attainment, health literacy levels, and health conditions. Consumer response data can help to identify any group of consumers who may have problems understanding and using the information on these websites and suggest strategies for redesigning the websites to meet these consumers' health information needs and communication competencies.

This study also examined a limited amount of sample websites and only examined English language websites. There are many consumers in the US who speak primary languages other than English. All of HHS's agencies are equipped with translations, however, only three of the 10 offices have the translation capability as an option, yet they all have translation of HHS's Nondiscrimination Notice on their website. All the HHS offices do have the ability to resize text for those with limited vision. Additionally, the more interactive functions within a website should enhance its appeal and performance. This study was not designed to evaluate the currency and accuracy of the health information provided on these websites.

Direction for Future Studies

Future research could examine user's reactions to health websites using surveys, interviews, and usability studies to assess consumers' understanding of the information provided, the relevance of the information to them, and their ease of use of the website. Question one of the Index score sheet asks if the material contains one main message statement. Future studies could modify this question to specify that this does not pertain to home pages which are designed to have multiple messages. Assessing the outcome of government health websites using the Suitability Assessment of Materials instrument (SAM) and comparing the results to the CDC's CCI would be interesting since much of the current low literacy health research incorporates SAM as a methodology in identifying difficulties with the clarity of health information materials (including graphics, layout, and typography features) (Doak et al., 1985). Research could also further and test contentions from the theories applied in this study and examine health literacy skills framework using additional or alternative theories such as Adult Learning Theory and Cognitive Load Theory.

Applications

These findings have clear implications that the HHS websites are well-designed and communicate information in meaningful and effective way. The data suggests that HHS's websites provide relevant health information (supporting Social Support Theory), demonstrate communication competence (supporting Relational Health Communication Competence Model), are designed to reduce misunderstanding (supporting Weick's Model of Organizing), and demonstrate perceived usefulness in providing ease of use, by design, for consumers use (supporting the Technology Acceptance Model). Additional research could further clarify contentions suggested by these theories.

However, the HHS's website pages could be improved and should be reviewed and updated with relevant current information. The six HHS's agencies have well-structured home pages; however, they have multiple messages. The general design of a home page should be structured around its appearance and the impression it makes on the consumer. A good home page should attract, captivate, educate, and encourage (ACEE) the consumer to seek out more information. Additional technological interventions would be beneficial and could be considered as a call for action. Chatrooms with experts, online support groups, where website users can communicate with other consumers who have similar health concerns as well as experts in the field, and toll free hot-lines, in real time and without robots, are all encouraging, supportive devices designed for the consumer to seek immediate advice and assistance.

The CDC should revisit their index and address the use of technology, this should include hyperlinks, since it is currently being used as an evaluation method for webpages. The index also needs to address question one which states, “Does the material contain one main message?” One main message is not relevant to a website's home page since a home page is the focus of multiple important messages. According to CDC's CCI, if the answer to question one is “no” then question two through four are also marked as “no” as well as question six because without a main message there can be no active voice. In general, the CCI has too many variables within variables which leads to interpretation errors. Although some of HHS's offices had one main message on their home page all the agencies had multiple messages on their home page, including the CDC. The CCI should consider that for many health website home pages, including their own, having only one main message would be a disadvantage and not serve the users' needs and expectations. Additionally, different methodologies could be used to enhance this content analysis study such as the use of surveys to consumers about health information websites, especially for examining consumer reactions to the websites relevant to adoption and uses of these health information sources.

Finally, government health websites should not take the place of a face to face conversation with a health care expert. Generally, most consumers pursue their Internet options prior to engaging in a face to face conversation. This study is important for the consumer seeking health information in that it indicates that the government health websites support relevant health information that use easy to understand health terminology, including numbers, communicate clear health information and demonstrate perceived usefulness in providing ease of use. Collectively, these characteristics are essential in identifying websites that are user-friendly and equipped to meet the consumers' needs and expectations. As always, it is very important that consumers speak with their health care provider about any health questions they may have and ask their provider to help them better understand information they may be confused about that they read on these and other websites.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Author Contributions

DG contributed to the design and implementation of the research and to the analysis of the results. DG and GK contributed to the writing of the manuscript.

Funding

We would like to acknowledge and thank George Mason University's Open Access Publishing Fund/University Libraries for their support in funding this article.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

Adams, K. M., and Corrigan, J. (2003). Priority Areas for National Action Transforming Health Care Quality. Washington, DC: National Academy of Sciences.

Google Scholar

Allen, M., Auld, E., Logan, R., Montes, J., and Rosen, S. (2017). “Improving collaboration amonghealth communication, health education, and health literacy,” in NAM Perspectives. Discussion Paper, National Academy of Medicine (Washington, DC). Available online at: https://nam.edu/improving-collaboration-among-health-communication-health-education-health-literacy. doi: 10.31478/201707c

CrossRef Full Text | Google Scholar

Alpert, J., Desens, L., Krist, A., Aycock, R., and Kreps, G. (2017). Measuring health literacy levels of a patient portal using the CDC's Clear Communication Index. Health Promot. Pract. 18, 140–149. doi: 10.1177/1524839916643703

PubMed Abstract | CrossRef Full Text | Google Scholar

Baur, C., and Prue, C. (2014). The CDC Clear Communication Index is a new evidence-based tool to prepare and review health information. Health Promot. Pract. 15, 629–637. doi: 10.1177/1524839914538969

PubMed Abstract | CrossRef Full Text | Google Scholar

Butterworth, K., Allam, O., and Gray, A. (2010). “The pedagogical requirements of patient E-health education to increase health literacy,” in 2010 IEEE 23rd International Symposium on Computer-Based Medical Systems (CBMS), October 2010 (Perth, WA: IEEE). doi: 10.1109/CBMS.2010.6042632

CrossRef Full Text | Google Scholar

Castro-Sánchez, E., Spanoudakis, E., and Holmes, A. H. (2015). Readability of ebola information on websites of public health agencies, United States, United Kingdom, Canada, Australia, and Europe. Emerg. Infect. Dis. 21, 1217–1219. doi: 10.3201/eid2107.141829

PubMed Abstract | CrossRef Full Text | Google Scholar

Centers for Disease Control Prevention (2013). Available online at: https://www.cdc.gov/healthcommunication/pdf/clearcommunicationindex/fillableformmay2013.pdf (Retrieved January 20, 2019).

Google Scholar

Centers for Disease Control Prevention (2014). Available online at: https://www.cdc.gov/ccindex/pdf/full-index-score-sheet.pdf (Retrieved November 15, 2018).

Google Scholar

Centers for Disease Control Prevention (2015). Available online at: https://www.cdc.gov/ccindex/tool/index.html (Retrieved November 15, 2018).

Google Scholar

Daraz, L., Morrow, A. S., Ponce, O. J., Beuschel, B., Farah, M. H., Katabi, A., et al. (2019). Can patients trust online health information? A metanarrative systemic review addressing the quality of health information on the Internet. J. Gen. Int. Med. 34, 1884–1891. doi: 10.1007/s11606-019-05109-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Davis, F. D., Bagozzi, R. P., and Warshaw, P. R. (1989). User acceptance of computer technology: a comparison of two theoretical models. Manag. Sci. 35, 982–1003. doi: 10.1287/mnsc.35.8.982

CrossRef Full Text | Google Scholar

Doak, C., Doak, L., and Root, J. (1985). Teaching Patients with Low Literacy Skills. Philadelphia, PA: Lippincott (Retrieved April 25). Available online at: https://cdn1.sph.harvard.edu/wp-~content/uploads/sites/135/2012/09/doakchap1-4.pdf

Google Scholar

Freelon, D. G. (2010). ReCal: intercoder reliability calculation as a web service. Int. J. Int. Sci. 5, 20–33.

Google Scholar

Gill, P. S., Gill, T. S., Kamath, A., and Whisnant, B. (2012). Readability assessment of concussion and traumatic brain injury publications by centers for disease control and prevention. Int. J. Gen. Med. 5, 923–933. doi: 10.2147/IJGM.S37110

PubMed Abstract | CrossRef Full Text | Google Scholar

Gonzalez, E., Sinjini, M., and Turel, O. (2018). Motivational impacts on intent to use health-related social media. J. Comput. Inform. Syst. 29, 1–10. doi: 10.1080/08874417.2017.1406829

CrossRef Full Text | Google Scholar

Graham, S., and Brookey, J. (2008). Do patients understand? Permanente J., 12, 67–69. doi: 10.7812/TPP/07-144

PubMed Abstract | CrossRef Full Text | Google Scholar

Heron, E., Winchcombe, M., Cobbledick, J., and Adams, J. (2017). Working with service users to design an accessible health website for people with arthritis and lower levels of digital literacy. Rheumatology 56, 132–133. doi: 10.1093/rheumatology/kex062.133

CrossRef Full Text | Google Scholar

Keyton, J. (2006). Communication research: Asking Questions, Finding Answers. New York, NY: McGraw-Hill.

Google Scholar

Kim, W., Kreps, G. L., and Shin, C. N. (2015). The role of social support and social networks in health information-seeking behavior among Korean Americans: a qualitative study. Int. J. Equity Health 14:40. doi: 10.1186/s12939-015-0169-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Kreps, G. L. (1988). Relational communication in health care. South. Speech Commun. J. 53, 344–359. doi: 10.1080/10417948809372736

CrossRef Full Text | Google Scholar

Kreps, G. L. (2009). Applying Weick's model of organizing to health care and health promotion: highlighting the central role of health communication. Patient Educ. Counsel. 74, 347–355. doi: 10.1016/j.pec.2008.12.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Kreps, G. L. (2011). Methodological diversity and integration in health communication inquiry. Patient Educ. Counsel. 82, 285–291. doi: 10.1016/j.pec.2011.01.020

PubMed Abstract | CrossRef Full Text | Google Scholar

Kreps, G. L. (2014a). “Relational health communication competence model,” in Encyclopedia of Health Communication, Volume III, ed T. L. Thompson (Los Angeles, CA: Sage Publications), 1160–1161.

Google Scholar

Kreps, G. L. (2014b). Evaluating health communication programs to enhance health care and health promotion. J. Health Commun. 19, 1449–1459. doi: 10.1080/10810730.2014.954080

PubMed Abstract | CrossRef Full Text | Google Scholar

Kreps, G. L. (2015). “Communication technology and health: the advent of health applications,” in Communication and Technology, eds L. Cantoni and J. A. Danowski (Berlin: De Gruyter Mouton), 485–496. doi: 10.1515/9783110271355-028

CrossRef Full Text | Google Scholar

Kreps, G. L. (2017). Online information and communication systems to enhance health outcomes through communication convergence. Hum. Commun. Res. 43, 518–530. doi: 10.1111/hcre.12117

CrossRef Full Text | Google Scholar

Kreps, G. L. (2018). Promoting patient comprehension of relevant health information. Israel J. Health Policy Res. 7, 1–3. doi: 10.1186/s13584-018-0250-z

PubMed Abstract | CrossRef Full Text | Google Scholar

Kreps, G. L., and Neuhauser, L. (2010). E-Health and health promotion. J. Comput. Mediat. Commun. 15, 527–529. doi: 10.1111/j.1083-6101.2010.01526.x

CrossRef Full Text | Google Scholar

Manganello, J., and Fishbein, M. (2009). “Using theory to inform content analysis,” in Media Message and Public Health: A Decisions Approach to Content Analysis, eds A. B. Jordan and D. Kunkel (New York, NY: Routledge, 3–14.

Google Scholar

Menchaca, R., Page-Ramsey, S., Rodriguez, E., Gray, J., Ireland, K., and Ramsey, P. (2017). Patient information about Zika virus and perinatal risks on state health department websites in the United States. Obstetr. Gynecol. 129:97S. doi: 10.1097/01.AOG.0000514956.12264.6b

CrossRef Full Text | Google Scholar

Meppelink, C. S., and Bol, N. (2015). Exploring the role of health literacy on attention to and recall of text-illustrated health information: an eye-tracking study. Comput. Hum. Behav. 48, 87–93. doi: 10.1016/j.chb.2015.01.027

CrossRef Full Text | Google Scholar

Neitzke, G. (2007). Confidentiality, secrecy, and privacy in ethics consultation. HEC Forum 19, 293–302. doi: 10.1007/s10730-007-9049-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Office of Disease Prevention and Health Promotion (2018). Health Communication and Health Information Technology. Available online at: https://www.healthypeople.gov/2020/topics-objectives/topic/health-communication-and-health-information-technology (Retrieved November 26, 2018).

Google Scholar

Ong, L. M., de Haes, J. C., Hoos, A. M., and Lammes, F. B. (1995). Doctor-patient communication: a review of the literature. Soc. Sci. Med. 40, 903–918. doi: 10.1016/0277-9536(94)00155-M

PubMed Abstract | CrossRef Full Text | Google Scholar

Parvanta, C., Nelson, D. E., and Harner, R. (2018). Public Health Communication Critical Tools and Strategies. Burlington, MA: Jones and Bartlett Learning.

Google Scholar

Point-of-Care-Partners (2012). Six Areas of Opportunity Under the Affordable Care Act. Available online at: https://www.pocp.com/PDF/health-plans-hit-opportunities.pdf (accessed November 20, 2018)

Google Scholar

Reisi, M., Javadzade, S. B., Heydarabadi, A. B., Mostafavi, F., Tavassoli, E., and Sharifirad, G. (2014). The relationship between functional health literacy and health promoting behaviors among older adults. J. Educ. Health Promot. 3, 119–124. doi: 10.4103/2277-9531.145925

PubMed Abstract | CrossRef Full Text | Google Scholar

Sak, G., Diviani, N., Allam, A., and Schulz, P. J. (2016). Comparing the quality of pro- and anti-vaccination online information: a content analysis of vaccination-related webpages. BMC Public Health 16:38. doi: 10.1186/s12889-016-2722-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Shim, M., Mahaffey, B., Bleidistel, M., and Gonzalez, A. (2017). A scoping review of human- support factors in the context of Internet-based psychological interventions (IPIs) for depression and anxiety disorders. Clin. Psychol. Rev. 57, 129–140. doi: 10.1016/j.cpr.2017.09.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Thayer, A., Evans, M., McBride, A., Queen, M., and Spyridakis, J. (2007). Content analysis as a best practice in technical communication research. J. Tech. Writ. Commun. 37, 267–279. doi: 10.2190/TW.37.3.c

CrossRef Full Text | Google Scholar

U.S. Department of Education, National Center for Education Statistics. (2014). Literacy, Numeracy, and Problem Solving in Technology-Rich Environments Among U.S. Adults: Results from the Program for International Assessment of Adult Competencies, Washington, DC: NCES.

Google Scholar

U.S. Department of Health and Human Service (2008). America's Health Literacy: Why We Need Accessible Health Information. Available online at: https://health.gov/communication/literacy/issuebrief/ (Retrieved November 30, 2018).

Google Scholar

U.S. Department of Health and Human Service (2015). Plain Writing Checklist. Available online at: https://www.hhs.gov/web/building-and-managing-websites/managing-websites/plain-writing-checklist/index.html (Retrieved January 23, 2018).

Google Scholar

U.S. Department of Health and Human Services (2010). National Action Plan to Improve Health Literacy. Washington, DC: U.S. Department of Health and Human Services.

Google Scholar

U.S. Department of Health and Human Services (2011). Federal Plain Language Guidelines. Available online at: https://www.plainlanguage.gov/guidelines/ (Retrieved January 20, 2019).

Google Scholar

U.S. Department of Health and Human Services (2013). Office of Minority Health. Healthcare Language Service Implementation Guide. Available online at: https://hclsig.thinkculturalhealth.hhs.gov/ (Retrieved November 23, 2018).

Google Scholar

U.S. Department of Health and Human Services (2016). About the Office of the Assistant Secretary for Health (OASH). Available online at: https://www.hhs.gov/ash/about-ash/index.html (Retrieved January 23, 2018).

Google Scholar

U.S. Department of Health and Human Services (2018a). About Us (DCD). Available online at: https://www.hhs.gov/web/about-us/index.html (Retrieved January 23, 2018).

PubMed Abstract | Google Scholar

U.S. Department of Health and Human Services (2018b). Programs & Services. Available online at: https://www.hhs.gov/about/index.html (Retrieved January 23, 2018)

Google Scholar

Vaux, A. (1988). Social Support: Theory, Research, and Intervention. New York, NY: Praeger.

Google Scholar

Weick, K. (1969). The Social Psychology of Organizing. Reading, MA: Addison-Wesley.

Google Scholar

Wilson, D. (2013). Registered Nurses' Collective Safety Organizing Behaviours: The Association With Perceptions of Patient Safety Culture. (Order No. 3459087). ProQuest Central; ProQuest Dissertations and Theses Global. (873557012). Available online at: https://search-proquest-com.mutex.gmu.edu/docview/873557012?accountid=14541 (Retrieved February 13, 2019).

Google Scholar

Wimmer, R., and Dominick, J. (2013). Mass Media Research. Boston, MA: Cengage Learning.

Google Scholar

Youngblood, N. (2018). Digital inclusiveness of health information websites. Univ. Access Inf. Soc. 1–12. doi: 10.1007/s10209-018-0629-1

CrossRef Full Text | Google Scholar

Zapf, A., Castell, S., Morawietz, L., and Karch, A. (2016). Measuring inter-rater reliability for nominal data - which coefficients and confidence intervals are appropriate? BMC Med. Res. Methodol. 16, 1–10. doi: 10.1186/s12874-016-0200-9

PubMed Abstract | CrossRef Full Text | Google Scholar