Assessing the Quality and Reliability of COVID-19 Information on Patient Organization Websites

Introduction

Accessing relevant public health information online can be challenging, particularly for those with health literacy deficits. Research indicates that lower levels of health literacy are associated with poorer health outcomes and higher health-related costs (Poureslami et al., 2017). Health literacy and the ability to navigate public health information has never been more critical than during the spread of SARS-CoV-2, known also as COVID-19. Since the outbreak of the coronavirus, the Internet has exploded with information and misinformation about the symptoms of COVID-19, how it is transmitted, the effectiveness of face masks and social distancing, and vaccine safety. Unfortunately, poorly designed public health websites in combination with lower health literacy can create major challenges in distributing accurate, relevant health information to the public (Gagen and Kreps, 2019). In addition to barriers created by complex technical language and disorganized content and messaging, COVID-19 has introduced challenges associated with misinformation and a rapidly changing ecosystem of health knowledge and guidance. Many people are not able to comprehend complex health information and the introduction of rapidly changing, technical information about COVID-19 is likely to intensify health literacy challenges. Community engagement and understanding of public health protocols are critical to stopping the spread of the novel coronavirus outbreak. Even more so, it is important that individuals understand their options when it comes to accessing health care during this time of physical isolation and quarantine.

Patients with living with chronic illness often turn to patient organization websites for information specifically related to their disease or disorder. Communication efforts around COVID-19 have only made these websites more valuable to vulnerable patient populations. Vulnerable populations have largely been left out of the conversation. Though efforts have been made by major health reference websites like Centers for Disease Control and Prevention (CDC) to provide information to vulnerable groups, most of the information being distributed via mass channels is not likely to provide specific guidance to those living with chronic illness and conditions. This has likely led many patients to turn to their patient advocacy organization website for information and guidance relevant to their illness throughout the pandemic. Even outside of a pandemic, these websites serve as important sources of health information about specific diseases, treatment options, research efforts and clinical trial recruitment and progress as well as resources for caretakers, physicians, and family members of patients with that illness (Rose, 2013). The COVID-19 pandemic has dramatically shifted the way that many patients are able to interact with the health systems and has altered what steps they may need to take to maintain their care, exacerbating distress, fear, and existing issues of access (Vanderpool et al., 2020). Though patients with chronic illness or other specialized disease and disorders may interface with health information more often, this does not necessarily translate to higher levels of health literacy or indicate that they are better equipped to navigate these websites. Additionally, these websites do provide information about and related to specific diseases, but that does not guarantee that individuals on the other side of the screen are comprehending what they are reading. Safeguarding patient comprehension of health information through effective health information systems is essential to the ensuing patient health and public health (Kreps, 2018).

Since each of these patient advocacy organizations will have their own unique website that uses a variety of design formats to publish information and guidance regarding COVID-19, it is important that the website provides information, is easy to navigate, and reduces misunderstanding. This is increasingly important in the ecosystem of misinformation that has been fostered during the pandemic. Therefore, the purpose of this study is to evaluate and compare the quality and reliability of COVID-19 information on major patient organization websites using an established measurement index designed by the CDC in 2014 (CDC, 2014). The research goals in this study are to assess and help to improve the clarity of online communication of information on COVID-19 by patient organizations by determining how well the websites score for clarity and identify ways to improve the websites. This study appears to be the first comparative content analysis study to analyze patient advocacy organization websites using the CDC’s well-established CCI.

Literature Review

Health Literacy

Health literacy is the degree to which people can acquire and understand health information to make informed health decisions (Nielsen-Bohlman, 2004). The National Assessment of Adult Literacy (NAAL) conducted by the U.S. Department of Education, National Center for Education Statistics in 2003, found that most participants have less than adequate levels of health literacy (Cutilli and Bennett, 2009). Health literacy levels can differ widely across race, age, education, and socio-economic status. For example, language, education, income, age, and health status are well documented risk factors of limited health literacy (Sun et al., 2013). Literacy, culture, communication skills, and existing beliefs/knowledge about health topics can also influence health literacy. Furthermore, these factors interact on every level (individual level, community level, and policy level) of health care and public health (Gagen and Kreps, 2019). Individuals with higher health literacy skills are generally able to manage their health more effectively than those who have lower skills (Spring, 2020). Patients with lower levels of health literacy are less likely to be engaged in the health care decision-making process, which complicates patient-provider communications and impedes a patient’s ability to participate in their own care. Self-care is a critical element of health care and is reliant on a patient’s comprehension to ensure effective self-management skills. Gaps in knowledge or understanding can disrupt self-management and negatively impact adherence to treatment plans (Poureslami et al., 2016).

Additionally, health literacy is associated with an individual’s ability to navigate digital information systems like the Internet. Low health literacy is negatively related to the ability understand and evaluate online health information (Neter and Brainin, 2019). This has certainly been the case with information seeking related to COVID-19. In contrast to health literacy required for the prevention and management of chronic diseases, pandemics require individuals to adapt and develop pandemic-related health literacy quickly (Seng et al., 2020). For those with chronic illness or disease, these layers of required health literacy compound, making managing care that much more difficult. For example, in a study conducted evaluating COVID-19 awareness and attitudes among chronic disease patients in the United States, one-third of participants were unable to identify COVID-19 symptoms and nearly a quarter of participants did not think they were likely to contract the virus (Wolf et al., 2020). Additionally, the volume and accessibility of information available online has become a double-edged sword because of the rapid transmission of misinformation. Furthermore, the evolving nature of scientific knowledge about COVID-19, related treatments, and the vaccine has led to health communication challenges and widespread confusion because of reversals in recommendations. Effective communication of information is vital for increasing health literacy and improving health outcomes. With many patients living with chronic illnesses in desperate need of information about COVID-19 and steps they can take to manage their care, certifying that patient advocacy organization websites are effectively communicating about COVID-19 and the vaccine will play an important role in improving outcomes for these patient groups.

Social Support Theory

Social support has been linked to overall physical health (Uchino, 2009). Social Support Theory is centered around the idea that instrumental, informational, and emotional support improves an individual’s ability to cope with stress and enhances self-esteem (Feeney and Collins, 2014). Supportive interactions outlined by this theory protect patients from stress and improve adherence to treatment plans. Public health information websites, including patient organization websites, are traditional forms of health information in which static information that focuses on informational, health content is shared (Kreps, 2015). Additionally, informational support, a form of social support, involves the transmission of information including health topics (McMullan, 2006). This form of social support aids in an individual’s sense of self-esteem by providing credible, informational content about a health topic of relevance (Yan and Tan, 2014). Health information available online that forms a support system through the exchange of information is a testament to the importance of social and informational support in delivering better health outcomes. The more health information a patient can access, the better they are able to comprehend their condition, equipping them to take action to improve their own health (McMullan, 2006). Though patient organization websites cannot offer instrumental support, they can provide emotional and informational support during a pandemic. The social support theory suggests that improving that the delivery of COVID-19 information on patient organization websites can improve health outcomes for consumers.

Relational Health Communication Competence Model

The Relational Health Communication Competence Model (RHCCM) was designed to address interpersonal relationships within health care settings. Represented as a wagon wheel with the hub representing the patient and the spokes representing health care provider, the model theorizes that increased communication competence, defined as a combination of interaction, empathy, relationship, behavioral flexibility, and social ease, leads to “therapeutic communication, social support, satisfaction, information exchange, and cooperation,” while decreased competence leads to “pathological communication, lack of social support, dissatisfaction, information barriers, and lack of cooperation” (Kreps, 2014). There is no reference to the Internet or computer-mediated health interventions in the RHCCM. However, the communication competence afforded by informational websites is related to the communication of relevant health information that supports active decision-making and improved health outcomes. Digital information and communication technology through public health websites, like those maintained by patient advocacy organizations, offer opportunities for improved access to health care information (Gagen and Kreps, 2019). However, these technologies cannot be effective if patients are unable to access and comprehend the information being provided.

To that end, it is important to ensure that these technologies, including websites examined in this study, offer clear, relevant information. Most health information websites present information using technical terminology that those unfamiliar with medical literature, which is most of the population, may not understand. Additionally, health websites generally require a high school reading level or higher, which is too high for much of the population to understand (Benigeri and Pluye, 2003). Further, complex, authoritative language can not only be difficult to understand, but can be perceived as intimidating and off-putting. There is room for improvement in the way health and medical information is organized so that it is easier to find, relevant, and ready to be used, even for people with a low level of health literacy (Benigeri and Pluye, 2003). Health information websites should accommodate lower literacy levels by using simple language and applying visual examples to ensure widespread comprehension and improve health outcomes (Kreps and Neuhauser, 2010).

Weick’s Model of Organizing

Weick’s Model of Organizing is used in health communication to describe the process of organizing interconnected communication processes to promote problem solving, adaptation, and growth (Kreps, 2009). The broad nature of this model allows for the dissection of complex health promotion and communications situations and interventions. Weick’s model demonstrates the importance of communication in enabling health care consumers and providers to make sense of the complexities of modern health care, select the best response strategies, and retain relevant information to guide future responses (Kreps, 2009). Critically, this model reduces the number of possibilities that may occur, the uncertainty in any given situation, to a practical level as organizations tend to transform ambiguous information into explicit information (Weick, 1969).

This need to clarify equivocal information and translate it into information that can be more widely understood is desperately needed in addressing complex health problems. This has certainly been emphasized by the volume of complex, technical language and information related to the pandemic. Health care consumers and providers often need help to make sense of complex health problems, to make accurate diagnoses, and to provide appropriate treatments (Gagen and Kreps, 2019). Information that is not presented in a comprehensible manner will not be useful in guiding effective responses to health problems. The model’s focus on clarity and reduction of equivocality makes it useful tool in the development and evaluation of strategic health communications like patient organization websites. Further, this model would suggest that health information communication systems, such as websites, need to be designed to provide patients/consumers with clear and relevant information to assist in guiding health decision making.

R1. Are patient organization websites designed to provide consumers with information about COVID-19 relevant to their health? (related to Social Support Theory).

R2. Are patient organization websites designed to demonstrate communication competence in providing easy to understand information about COVID-19 to consumers? (related to the Relational Health Communication Competence Model).

R3. Are patient organization websites designed to reduce consumer misunderstanding about COVID-19 by communicating clear health information? (related to Weick’s Model of Organizing).

Statement of Method

This study employs a content analysis of patient organization websites using the Clear Communication Index (CCI) to evaluate quality and reliability of COVID-19 information on these websites.

Instrumentation

As a compliance measure of the Plain Writing Act of 2010, the Clear Communication index was developed by the CDC to identify key communication characteristics that enhance clarity and aid understanding of public messages and materials and provide a research-based tool to develop and assess communications for CDC’s audiences (CDC, 2014). The CDC’s CCI development, reliability and validity testing was constructed by the CDC’s Office of the Associate Director for Communication (Gagen and Kreps, 2019).

CDC’s Clear Communication Index

This study used a content analysis method with the CDC’s CCI tool to evaluate the clarity of 15 major patient organization websites designed for patients to have access to information about COVID-19. The objective of this content analysis was to measure the content of patient organizations in terms of the clarity of online communication of information on COVID-19. Results should inform recommendations for improvements to the clarity of online communication of information and build on an existing literature base of the importance of clear communication in health content presented via websites. The CCI research tool is separated into four major categories used for assessment of key aspects of clear communication: Core, Behavioral Recommendations, Numbers, and Risk (CDC, 2014). The index contains 20 items across these four sections that build on and expand plain language techniques described in the Federal Plain Language Guidelines, which are official guidelines set by the Plain Writing Act of 2010 and are required by law to be followed in the development of public communication products (CDC, 2014). Each question is code with a numerical score of zero or one, corresponding to whether the criteria for that question was met or not met (CDC, 2014). The highest point score that can be attained is 20/20, though, depending on the content elements present, certain questions are not required to be answered resulting in a lower maximum point value attainable. This total score achieved is converted to a scale of 100 wherein a score of 90 percent or higher is considered passing or excellent and a score of 89 percent or below indicates that the communication materials will likely require some revisions to improve clarity and understandability for the intended audience (CDC, 2014). The CCI has been tested for reliability and validity as a research tool and results indicate that the CCI performed as intended and made it more likely that audiences could understand the material presented (Baur and Prue, 2014).

Analytical Process

A total of 44 website pages, one home page and two additional pertinent pages related to COVID-19, from each website were chosen from a selection of 15 major patient organization websites. Websites that were written in the English language and provided information on prevention or treatment of COVID-19 intended for the general public were considered eligible. Any websites intended for professionals or specific isolated populations, such as individuals from a specific state, were excluded, as well as websites with only video content, marketing content, daily caseload update, or news dashboard pages with no health information. These websites and webpages were accessed and coded during the week of March 28, 2021–April 3, 2021. Due to the ever-changing nature of website content as well as the rapid changes in information around the COVID-19 virus, vaccines, and treatments, it was important to limit the time that the websites were accessed. Many of these websites had numerous other webpages that contained additional information pertaining to the members, events, reports, policy, and health information not related directly to COVID-19, and, for this study, were not considered to be pertinent pages. Home page was defined as the main page a visitor navigates to the website from a web search engine. Pertinent pages referred to informational pages on COVID-19 including spread, symptoms, vaccines, response, etc., that the public would view and does not include other organizational materials. Each of the organizations were also evaluated based on the inclusion for relevant themes and topics related to COVID-19 listed in Table 1. The relevant themes and topics were determined based on those covered by the CDC and state health department websites. In addition to a total score and thematic inclusion, pages last updated dates were also documented. After accessing each organization’s homepage, the COVID-19 sections were accessed, and the first two relevant pages were chosen as pertinent pages. Pertinent pages were chosen for the rest of the organizations in a similar manner and listed in Table 2. Notably, each website offered different types of informational pages, so content did vary slightly across pertinent pages. Each page was evaluated and scored using the CDC’s CCI.

TABLE 1

TABLE 1. Themes covered by patient organization websites.

TABLE 2

TABLE 2. Pertinent pages.

Two coders discussed the method they would use in perceiving content as it related to the scoring index. To test inter-coder reliability, a subsample of three additional pertinent pages were coded independently by both coders (Lacy et al., 2015). The two coders then discussed the differences in their scores and determined the reason for any differences. Intercoder reliability was tested using ReCal2 for two coders (Freelon, 2010). Krippendorff’s alpha for Variable 1 (Core) was 0.768, Variable 2 (Behavioral Recommendation) was 1, Variable 3 (Numbers) was 0.78, and Variable 4 (Risk) was 0.78 (Table 3). According to Keyton et al. (2004), a reliability coefficient of 0.70 or above is adequate for establishing intercoder reliability.

TABLE 3

TABLE 3. Total scores.

Results

Scores from the CDC’s CCI are interpreted by total score. Scores from the Clear Communication Index are interpreted by total score with 90 being considered passing. Scores less than 90 require adjustments to make the material easier to understand and use. Guidance for these improvements can be accessed in the CDC’s CCI User Guide available online. The CCI’s questions, interpreted in this study as variables, often contain multiple elements or variables as well. Gagen and Kreps (2019) point out that question five addresses multiple variables (Whether or not the material has one or more calls to action, has a prompt, a request for more information, a request to share information or a broad call for program or policy change) leaving it vulnerable to misinterpretation. If the material does not contain a main message, question six is scored as a zero as it refers to an active voice being applied to both the main message and the call for action. If no behavioral recommendations were made, questions 12–14 were skipped. This section was skipped for 29 of the 45 webpages. Similarly, if the webpage did not include one or more numbers related to the topic, questions 15–17 were skipped. Many of the webpages reviewed did not contain numbers, so this section was skipped for 34 of the 45 webpages. Lastly, material that does not present information about risk, including numbers, will skip questions 18–20.27 webpages skipped this section of scoring. The results show that none of the organizations scored above a 90 percent, HA, AS, and ACS scored the highest with scores of 72 percent, 76 percent, and 70 percent, respectively. A comparison of patient organization webpages including average scores can be found in Figure 1. Notably, if the material contains more than one message statement, the CCI requires that the scorer answer “no” for questions one through four. 14 of the 15 patient organization homepages reviewed did not have one main message, but featured multiple messages resulting in an automatic loss of four points. This significantly impacted the scores of most of the homepages and lowered the average total score across the organization websites. HA was the only organization with a single main message relevant to their mission.

FIGURE 1

FIGURE 1. Average scores.

Page updates for all the homepages analyzed were not displayed on the website and classified as a no date. Three of the organizations did not have a date listed for all three webpages reviewed and five of the 15 had at least one pertinent page that had not been updated since 2020. Eight of the 15 websites had at least one pertinent page dated March 2021 or newer, with ACS, JDRF, CFF having the most currently updated information available. Webpages with page updates dated March 2021 or newer were considered to contain up to date information. With the rapidly changing nature of information surround the COVID-19 pandemic including treatments, vaccines, public health procedures, and other critical information, the lack of update material denoted on these webpages was disconcerting. The pages last updated list is included in Table 4.

TABLE 4

TABLE 4. Pages last updated.

Research question one addresses if patient organization websites are designed to provide consumers with information about COVID-19 relevant to their health. The results indicate that these websites do not necessarily provide relevant health information to consumers. With all fifteen websites scoring below 90 for the total score, there is evidence that more work needs to be done to communicate information about COVID-19 to these patient populations. These websites are not adequately designed to provide consumers with information about COVID-19 relevant to their health (Figure 2). Additionally, at least half of the pertinent pages are not up to date and, even when the scores for the pertinent pages are calculated separately, none of the pages achieved a passing score of 90.

FIGURE 2

FIGURE 2. Provide COVID-19 information relevant to health.

The second research question asks if the patient organization websites are designed to demonstrate communication competence in providing easy to understand information about COVID-19 to consumers. The results provide support that these websites do not consistently present easy to understand health information. 14 of the 15 organizations’ home pages lost points because they contained multiple main messages. Even when the pertinent pages are scored independently of the home page, all the scores are still below the 90 percent passing score (Figure 3), which indicates that the level of communication competence exhibited in these websites is not sufficient to share relevant COVID-19, facilitate informed health care decision making around the pandemic, and coordinate activities such as vaccination and social distancing that are so critical to curbing the spread of the pandemic and protecting individual health outcomes.

FIGURE 3

FIGURE 3. Demonstrate communication competence.

The third research question addresses if the patient organization websites are designed to reduce consumer misunderstanding about COVID-19 by communicating clear health information. The results provide support that these websites are not adequately designed to reduce consumer understanding by providing clear health information. Homepages did not always clearly indicate where COVID-19 information could be found. Many of the pages did not use illustrations or visual examples such as video or charts to explain the content. Though websites with only video content were excluded from the study, only six pertinent web pages featured video content and only four web pages included graphic elements such as photos of desired behavior like handwashing or infographics. Despite lower than ideal scores, most of the pertinent pages incorporated active voice and messaging that would accommodate lower levels of health literacy. Most of the websites did focus on delivering information at a level accessible for even those with more limited health literacy, but it is important to acknowledge that much of the content was specifically tailored to the individual patient groups. For example, the American Diabetes Association’s COVID-19 webpages were primarily focused on issues associated with COVID-19 and diabetes like insulin maintenance and other diabetes care instructions. Though the information presented was disease or patient community specific, the languages used would be accessible to most levels of health literacy. Additionally, where there was a behavioral recommendation, the importance of the recommendation was explained, and guidance was usually included about how to perform the behavior. Numbers, the few times they were used, were explained simply and without the need to conduct mathematical calculations. Despite many of the webpages recommending specific behaviors, only the benefits of the behaviors were mentioned. Surprisingly, risks associated with COVID-19, the vaccine, and related treatments were largely not covered by the material available on these websites. However, it is important to point out that many of the behavioral recommendations urged consumers to wear a mask or social distance. These behaviors do not have widely agreed upon risk, so including explanation of risk in these circumstances would be challenging. The specific nature of COVID-19 and the pandemic response made the application of the CCI not fully cohesive in the way that some of the questions are posed.

Given the diversity of information about COVID-19 and the relevancy of certain elements of the pandemic to patient groups, a thematic analysis was conducted to better understand what major COVID-19 topic were covered by these organizations on their websites (Table 1). Many of the organizations featured content that includes information about key elements of COVID-19 symptoms, disease mitigation (hand washing, mask wearing, social distancing, etc.), vaccines, and disease-specific guidelines. Information about COVID-19 testing, contract tracing, and what to do if a person may have contracted COVID-19 was not as commonly included on these websites with only seven websites including information on testing and steps to take if you think you are sick and only one website, AA, including information about contact tracing. Though no information about these topics was presented inaccurately, the amount of information on each topic, if present, varied. As the COVID-19 pandemic has progressed the focus has shifted from tracing the disease, which is accomplished through testing and contact tracing, to disease mitigation through vaccination and other well-known public health measures. This shift may explain why fewer websites included this type of information. Notably, all 15 websites included information and specific guidance for the patient populations represented by the organizations. This inclusion of more targeted guidance in critically important as agencies like the CDC have not produced informational content that specifically address the needs, concerns, and risks of many patient populations. Though, the websites did not meet a passing score based on the CDC’s CCI, the inclusion of this disease-specific information does communicate instrumental information for these groups and provides emotional support which reduces stress and provides patients/consumers with a self-efficacy in ensuring disease-specific guidance.

Discussion

This study suggests that generally the patient organization websites that were analyzed are not adequately designed to communicate information about COVID-19 in meaningful and effective ways. Analysis of the websites indicates that they do not support the desired goals embedded in each of the three research questions examined in this study. The data suggests that all the websites analyzed should take additional steps to enhance the communication of COVID-19 health information to their patient groups. Though these websites did include some information related to COVID-19, they did not always include relevant, updated health information about COVID-19. Additionally, the websites did not facilitate straightforward navigation to COVID-19 related content and did not successfully provide consistently clear health information, making it challenging for consumers to use these resources. However, these results might suggest that there are issues associated with the way that information is being scored using the CCI, particularly when evaluating website homepages. As mentioned previously, with the CCI, a significant number of points are given to pages that have one main message and a call to action, both of which should be in an active voice and should be visible without scrolling. When present, these characteristics surely deliver an effective message to the targeted audience. However, most homepages of patient advocacy organizations, and many other relevant health organizations and agencies as well, included multiple messages and calls to active which are presented in both an active and passive voice. The data indicates that pertinent pages were more likely to have one main message given in an active voice with multiple calls to action compared to the homepages resulting in higher scores on-average for these pertinent pages. Homepages should be designed to make and impressions on the consumer by providing an overview of the content and information the website can offer. The way in which the information is presented should encourage consumers to visit other pages on the website and, in the case of this study, should encourage consumers to pursue webpages related to COVID-19 information. In general, multiple messages offered on a homepage can improve accessibility and provide navigation that increases feelings of social support (Gagen and Kreps, 2019). For consumers looking for information about COVID-19 health information and information about COVID-19 as it relates to their disease or illness, multiple messages can distract from the intended goal of acquiring guidance.

The rapidly evolving ecosystem of information around COVID-19 further emphasizes the importance of updating information on pertinent pages. Websites that have easily identifiable update dates are much more credible to the consumer At least five of the webpages on COVID-19 containing frequently asked questions, testing, treatment, and vaccine information has not been updated since some time in 2020. For example, SBA’s webpage on frequently asked questions on spina bifida and COVID-19 has not been updated since april 17, 2020, and most likely contains outdated, possibly irrelevant information while leaving out critically important new information. In March 2020, the World Health Organization (WHO) declared the COVID-19 outbreak a pandemic. Since then, important information about the virus itself as well as appropriate prevention measures, protocols meaning health care, and information about vaccine development and distribution have been announced. Given the risk of COVID-19 to health, especially for those who may have underlying conditions or medical needs, it is important to update web pages with risks, suggestions, vaccine information and a call to action to engage in mitigation measures and take one of the available vaccines. In general, not updating pertinent pages makes the organizations look uninformed or lacking in expertise. The critical importance of access to updated information during the pandemic may give the appearance that these organizations do not have the capacity or knowledge to update the pages.

Despite the inclusion of clearly communicated, relevant information about disease specific guidance for COVID-19, the pertinent page statistics indicate that the patient organization websites are not designed to provide consumers with support and information that is useful for health decision-making during the pandemic. Chapman et al. (2017) suggest that websites that lack straightforward text phrasing and formatting, interactivity, ease of navigation, and readability could potentially be inaccessible to those with low health and digital literacy levels. Additionally, complex, or bulky paragraphs of text an overwhelm consumers, particularly those with lower levels of health literacy, and impact their ability to engage with the content and their willingness to carry out recommended behavioral directives (ODPHP, 2015). All the websites evaluated in this study had long paragraphs densely packed with information. Though bulleted lists and chunks were created, efforts to further break up or diversify the way in which the information is presented are needed to improve understandability of the websites.

Patient advocacy organizations are trusted by those patient communities to provide informational resources, patient and caregiver support, and advocacy for research and policies that support the development of treatment or cures. The websites of these organizations serve as a hub for patients, caregivers, and families to stay informed on any relevant information related to research, treatment, advocacy, and care. COVID-19 has emphasized this as vulnerable populations are at greater risk for developing serious, life-threatening illness and their medical needs are often more specialized than the general population. It is important to recognize that the elements and characteristics of these websites are constantly changing and adapting to include new elements and information. Though this adaptability well serves the shifting policy and research priorities of these groups, it can make finding content difficult. However, it is worth noting that the capacity for variety being offered through the adaptability of these patient organization websites does present the opportunity for these websites to evolve to better meet consumer needs.

The patient organization websites lack visual elements of any kind. Predominantly consisting of chunks of text, these websites fail to deliver aesthetic stimulation that might process the information and make connections between the content and actionable steps. Furthermore, there is limited use of photos of people performing desired behaviors on these websites. Visual information, is helpful in encouraging individuals to engage with self-management approaches to care (Chapman et al., 2017). Due to the quarantine and isolation, COVID-19 required many people to manage their own care and participate in mitigating illness without close contact with their health care providers. These websites miss the opportunity to use visual information as a tool to facilitate information exchange, which may lead to poorer outcomes for the individuals reliant on these websites. Aesthetically pleasing websites, which not only create a more enjoyable experience for the consumer, can facilitate increased understanding. Websites like these that share health information on topics like COVID-19 should demonstrate communication competence by providing information to the consumer that is clear and relevant and reduces ambiguity about the health issues covered to promote the making of relevant health decisions (Gagen and Kreps, 2019).

Limitations

This study was reliant on content analysis of the information on COVID-19 provided on select patient organization websites and did not evaluate consumer experiences or responses associated with utilization of the website. As previously mentioned, the CDC’s CCI is not a particularly flexible tool for evaluation and does not leave much room for incorporating user expectations. It would be useful to compare user experience based on demographic factors such as age, gender, cultural and ethnic backgrounds, educational levels, health literacy levels, and health conditions. This information can identify those groups that might have issues comprehending the information and can inform solutions for how elements of the websites might be redesigned with these limitations in mind.

This study also examined a limited selection of patient advocacy organization websites and only examined English language websites. There are many consumers that do not consider English as their primary language and roughly half (Seven of 15 websites) of the evaluated websites do not have translation options. Of those that do have translation available, these sites were not reviewed for accuracy of translation or any of the criteria outlined by the CCI. Many of these websites cater to targeted populations that might experience visual or audio impairments. However, only two websites included accessibility options.

The websites in this study are being evaluated based on COVID-19 health information. This limits the pages and types of content and information that were being evaluated, so the result of this study cannot be generalized to the entire website or the effectiveness of information communication on other topics on these websites. Additionally, early in the pandemic, individuals, organizations, and government agencies were directed to the CDC’s website for the most relevant, current information about COVID-19. This reliance on the CDC may have led some organizations to underdeveloped resources for their specific patient populations by assuming that their audience would also be searching for information on the CDC website. User information and information seeking behavior was not measured in this study, so no definitive conclusions to that end can be made.

Direction for Future Studies

Future research should expand the number of websites being evaluated as well as the number of pages being reviewed on each website. There are hundreds of patient organization websites that have yet to be reviewed. Additionally, there are numerous sub-categories of topics and theme under the umbrella of COVID-19. The accuracy of the information on each of these topics should be assessed. Additionally, there is not currently any literature that could be found that evaluates the clarity and reliability of general health information presented on these patient organization websites. Conducting a more generalized study would be beneficial in understanding communication competence of these types of websites on a broader scale. Additionally, future studies analyzing the communication of COVID-19 information across the social media channels of these patient organizations would be beneficial in developing a holistic understanding of the communication of COVID-19 by these organizations. A comparative study of patient organization websites to well-trusted government websites that presented health information would also be useful in providing context. To that end, a future evaluation of how various government bodies communicate in different cultures and across various jurisdiction compared with response and success in disease containment of COVID-19 would also be valuable. Future studies could also examine consumer reactions and opinions of patient organization websites using surveys, interviews, and usability studies to assess consumers’ understanding of the information provided, the relevance of the information to them, and their ease of use of the website. Furthermore, it was clear during data collection that some websites were more visually appealing and easier to navigate than others. Future studies should assess the effects of formatting on readability for these websites.

Broader Implications

These findings have clear implications that the patient organization websites assessed websites do not adequately communicate information in meaningful and effective way as determined by the CDC’s CCI. The data indicates that these websites do not sufficiently provide relevant COVID-19 health information (supporting Social Support Theory), demonstrate communication competence (supporting the Relational Health Communication Competence Model), and do not support the reduction of misunderstanding (supporting Weick’s Model of Organizing). These websites should and can be improved in several ways to support enhanced communication of COVID-19 health information.

Nearly all the patient organization homepages have multiple messages, with many not prominently featuring COVID-19 resources or information. Though the homepages of the reviewed websites are aesthetically pleasing, for the most part, improvement could be made to the pages structure to make it easier to navigate. COVID-19 health information was the primary objective of this study, but it is important to note that all the homepages made efforts to encourage the consumer to seek out more information by offering a call to action, even if this was unrelated to COVID-19. In addition to challenges with navigation, the websites did not have opportunities for consumers to ask questions or interact with the content. Incorporating supportive technologies would be useful additions in aiding consumers. COVID-19 has highlighted the need for interactivity on health organization websites. Information being shared about the novel coronavirus may not answer a consumer’s questions about the virus nor does it allow for consumers to pose specific questions related to their individual health circumstances.

Patients are most trusting of their health care provider when it comes to obtaining health information in general as well as information about COVID-19, but they are likely to take to Google before any face-to-face conversation with their doctor. With the limitations placed on accessing health care during the pandemic, health websites have become increasingly important resources for patients as they try to navigate their health. As mentioned, locating COVID-19 information varied in difficulty. The websites should consider including more prominent menu placements for these resources, especially as COVID-19 guidance is continuing to evolve. Along those lines, it seems that many of these websites are treating COVID-19 information and resources as temporary fixtures of the websites, publishing content in the form of blogs and articles rather than permanent pages. There is still uncertainty surrounding how long the COVID-19 virus will persist following the pandemic and it is possible that information about COVID-19 guidance, treatments, and vaccines will still be relevant in the coming years. For this reason, creating a permanent location on these websites for resources related to COVID-19 would aid in accessibility and create the space for and incentivize improvements and updates to be made to the existing content.

To that end, there was no use of visual communication, with one exception. More visuals should be included to make the information more engaging and to communicate the information in a non-text manner which can facilitate better comprehension of the information. The inclusion of informative images and relevant videos to support text can improve accessibility and comprehension. Many of these webpages did not include discussion of all aspects of COVID-19 and the pandemic. Some websites have information about testing and vaccines, but no discussion of symptoms or guidance in the case of illness. In many cases there was no presentation of the risks associated with COVID-19 or any connection between the information and the recommended behaviors suggested. It would be useful on pages with COVID-19 health information to provide a holistic picture of the entire COVID-19 health ecosystem for those that may not have prior context. These organizations should work to develop targeted information for their patient groups rather than relying on other websites, like the CDC, as consumers are likely seeking disease-specific guidance from the webpages. This will also allow the organization to clarify unclear information on external websites and present the most relevant information to their audience in a comprehensible manner.

This study is important for the consumers that turn to patient organization websites for COVID-19 health information both generally and as it pertains to specific health needs in that it indicates that there are major deficits in the communication that makes it difficult for consumers to understand information about COVID-19 and aid in health decision-making. Improving the quality of health-related websites is critical to national efforts to promote health literacy and shared decision making. In times of public health emergency like COVID-19, it is imperative that trusted organizations and representatives of vulnerable groups like those with preexisting or chronic disease make efforts to present health information in a way that promotes improved access to health services and informed health decision making.

Data Availability Statement

The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.

Author Contributions

EB contributed to the design and implementation of the research and to the analysis of the results. EB contributed to the writing of the manuscript.

Funding

I would like to acknowledge and thank George Mason University’s Open Access Publishing Fund/University Libraries for their support in funding this article.

Conflict of Interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

I would like to thank and acknowledge GK for his review of the manuscript before submission.

References

Baur, C., and Prue, C. (2014). The CDC Clear Communication Index Is a New Evidence-Based Tool to Prepare and Review Health Information. Health Promot. Pract. 15 (5), 629–637. doi:10.1177/1524839914538969

PubMed Abstract | CrossRef Full Text | Google Scholar

Benigeri, M., and Pluye, P. (2003). Shortcomings of Health Information on the Internet. Health Promot. Int. 18 (4), 381–386. doi:10.1093/heapro/dag409

PubMed Abstract | CrossRef Full Text | Google Scholar

Centers for Disease Control and Prevention (CDC) (2014). CDC Clear Communication Index Score Sheet. Atlanta, GA: Centers for Disease Control and Prevention (CDC).

Chapman, L., Brooks, C., Lawson, J., Russell, C., and Adams, J. (2017). Accessibility of Online Self-Management Support Websites for People with Osteoarthritis: A Text Content Analysis. Chronic Illness 15 (1), 27–40. doi:10.1177/1742395317746471

PubMed Abstract | CrossRef Full Text | Google Scholar

Cutilli, C. C., and Bennett, I. M. (2009). Understanding the Health Literacy of America. Orthopaedic Nurs. 28 (1), 27–32. doi:10.1097/01.nor.0000345852.22122.d6

PubMed Abstract | CrossRef Full Text | Google Scholar

Feeney, B. C., and Collins, N. L. (2014). A New Look at Social Support. Pers Soc. Psychol. Rev. 19 (2), 113–147. doi:10.1177/1088868314544222

PubMed Abstract | CrossRef Full Text | Google Scholar

Freelon, D. G. (2010). ReCal: Intercoder Reliability Calculation as a Web Service. Int. J. Internet Sci. 5 (1), 20–33.

Google Scholar

Gagen, D. M., and Kreps, G. L. (2019). An Examination of the Clarity of Government Health Websites Using the Centers for Disease Control and Prevention's Clear Communication Index. Front. Commun. 4. doi:10.3389/fcomm.2019.00060

CrossRef Full Text | Google Scholar

Keyton, J., King, T., Mabachi, N. M., Manning, J., Leonard, L. L., and Schill, D. (2004). Content Analysis Procedure Book. Lawrence, KS: University of Kansas.

Kreps, G. L. (2015). “26. Communication Technology and Health: The Advent of Ehealth Applications,” in Communication and Technology. Editors L. Cantoni, and J. A. Danowski (Berlin, Munchen, Boston: De Gruyter Mouton), 485–496. doi:10.1515/9783110271355-028

CrossRef Full Text | Google Scholar

Kreps, G. L. (2009). Applying Weick's Model of Organizing to Health Care and Health Promotion: Highlighting the central Role of Health Communication. Patient Edu. Couns. 74 (3), 347–355. doi:10.1016/j.pec.2008.12.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Kreps, G. L. (2014). Evaluating Health Communication Programs to Enhance Health Care and Health Promotion. J. Health Commun. 19 (12), 1449–1459. doi:10.1080/10810730.2014.954080

PubMed Abstract | CrossRef Full Text | Google Scholar

Kreps, G. L., and Neuhauser, L. (2010). New Directions in eHealth Communication: Opportunities and Challenges. Patient Edu. Couns. 78 (3), 329–336. doi:10.1016/j.pec.2010.01.013

PubMed Abstract | CrossRef Full Text | Google Scholar

Kreps, G. L. (2018). Promoting Patient Comprehension of Relevant Health Information. Isr. J. Health Pol. Res 7 (1), 56. doi:10.1186/s13584-018-0250-z

PubMed Abstract | CrossRef Full Text | Google Scholar

Lacy, S., Watson, B. R., Riffe, D., and Lovejoy, J. (2015). Issues and Best Practices in Content Analysis. Journalism Mass Commun. Q. 92 (4), 791–811. doi:10.1177/1077699015607338

CrossRef Full Text | Google Scholar

McMullan, M. (2006). Patients Using the Internet to Obtain Health Information: How This Affects the Patient-Health Professional Relationship. Patient Edu. Couns. 63 (1-2), 24–28. doi:10.1016/j.pec.2005.10.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Neter, E., and Brainin, E. (2019). Association between Health Literacy, eHealth Literacy, and Health Outcomes Among Patients with Long-Term Conditions. Eur. Psychol. 24 (1), 68–81. doi:10.1027/1016-9040/a000350

CrossRef Full Text | Google Scholar

Nielsen-Bohlman, L. (2004). Health Literacy a Prescription to End Confusion. Washington, D.C.: National Academies Press.

Poureslami, I., Nimmon, L., Rootman, I., and Fitzgerald, M. J. (2016). Health Literacy and Chronic Disease Management: Drawing from Expert Knowledge to Set an Agenda. Health Promot. Int. 32 (4), daw003. doi:10.1093/heapro/daw003

PubMed Abstract | CrossRef Full Text | Google Scholar

Rose, S. L. (2013). Patient Advocacy Organizations: Institutional Conflicts of Interest, Trust, and Trustworthiness. J. L. Med. Ethics 41 (3), 680–687. doi:10.1111/jlme.12078

CrossRef Full Text | Google Scholar

Seng, J. J. B., Yeam, C. T., Huang, C. W., Tan, N. C., and Low, L. L. (2020). Pandemic Related Health Literacy - A Systematic Review of Literature in COVID-19, SARS and MERS Pandemics. MedRxiv. doi:10.1101/2020.05.07.20094227

CrossRef Full Text

Spring, H. (2020). Health Literacy and COVID‐19. Health Info Libr. J. 37 (3), 171–172. doi:10.1111/hir.12322

PubMed Abstract | CrossRef Full Text | Google Scholar

Sun, X., Shi, Y., Zeng, Q., Wang, Y., Du, W., Wei, N., et al. (2013). Determinants of Health Literacy and Health Behavior Regarding Infectious Respiratory Diseases: A Pathway Model. BMC Public Health 13 (1), 261. doi:10.1186/1471-2458-13-261

PubMed Abstract | CrossRef Full Text | Google Scholar

Uchino, B. N. (2009). Understanding the Links between Social Support and Physical Health: A Life-Span Perspective with Emphasis on the Separability of Perceived and Received Support. Perspect. Psychol. Sci. 4 (3), 236–255. doi:10.1111/j.1745-6924.2009.01122.x

PubMed Abstract | CrossRef Full Text | Google Scholar

U.S. Department of Health and Human Services, Office of Disease Prevention and Health Promotion (ODPHP) (2015). Health Literacy Online: A Guide to Simplifying the User Experience. Retrieved from https://health.gov/healthliteracyonline/.

Google Scholar

Vanderpool, R. C., Huang, G. C., Mollica, M., Gutierrez, A. I., and Maynard, C. D. (2020). Cancer Information-Seeking in an Age of COVID-19: Findings from the National Cancer Institute's Cancer Information Service. Health Commun. 36 (1), 89–97. doi:10.1080/10410236.2020.1847449

PubMed Abstract | CrossRef Full Text | Google Scholar

Weick, K. E. (1969). Social Psychology in an Era of Social Change. Am. Psychol. 24 (11), 990–998. doi:10.1037/h0028881

CrossRef Full Text | Google Scholar

Wolf, M. S., Serper, M., Opsasnick, L., O'Conor, R. M., Curtis, L., Benavente, J. Y., et al. (2020). Awareness, Attitudes, and Actions Related to COVID-19 Among Adults with Chronic Conditions at the Onset of the U.S. Outbreak. Ann. Intern. Med. 173 (2), 100–109. doi:10.7326/m20-1239

PubMed Abstract | CrossRef Full Text | Google Scholar

Yan, L., and Tan, Y. (2014). Feeling Blue? Go Online: An Empirical Study of Social Support Among Patients. Inf. Syst. Res. 25 (4), 690–709. doi:10.1287/isre.2014.0538

CrossRef Full Text | Google Scholar