Evaluation of online videos to engage viewers and support decision-making for COVID-19 vaccination: how narratives and race/ethnicity enhance viewer experiences

Background Vaccine hesitancy has hampered the control of COVID-19 and other vaccine-preventable diseases. Methods We conducted a national internet-based, quasi-experimental study to evaluate COVID-19 vaccine informational videos. Participants received an informational animated video paired with the randomized assignment of (1) a credible source (differing race/ethnicity) and (2) sequencing of a personal narrative before or after the video addressing their primary vaccine concern. We examined viewing time and asked video evaluation questions to those who viewed the full video. Results Among 14,235 participants, 2,422 (17.0%) viewed the full video. Those who viewed a personal story first (concern video second) were 10 times more likely to view the full video (p < 0.01). Respondent–provider race/ethnicity congruence was associated with increased odds of viewing the full video (aOR: 1.89, p < 0.01). Most viewers rated the informational video(s) to be helpful, easy to understand, trustworthy, and likely to impact others' vaccine decisions, with differences by demographics and also vaccine intentions and concerns. Conclusion Using peer-delivered, personal narrative, and/or racially congruent credible sources to introduce and deliver vaccine safety information may improve the openness of vaccine message recipients to messages and engagement.


Introduction
Vaccine hesitancy and its role in vaccine uptake and also the subsequent control of vaccine-preventable diseases have become a major focus of research and practice (1).Since the convening of WHO's Strategic Advisory Working Group (SAGE) on vaccine hesitancy (2012) and the group's published definition of vaccine hesitancy in 2015 (2), researchers and practitioners in the vaccine community have continued to propose frameworks for measuring (3) and testing strategies to address vaccine hesitancy (3)(4)(5)(6)(7).In light of rising attention and efforts to address vaccine hesitancy over the past decade, vaccine hesitancy was formally recognized-prior to the start of the COVID-19 pandemic-as one of the top 10 threats to global health and security (8,9).
The acceptance or refusal of COVID-19 vaccination-despite the widespread availability of vaccines-has been hampered by beliefs that COVID-19 does not present a serious health risk and a variety of concerns related to vaccine effectiveness and safety (10,11).Along with sociodemographic factors (e.g., sex, education, race/ethnicity, and age), political affiliation, trust in public health authorities, and receiving the influenza vaccine in the prior year have been identified factors of COVID-19 vaccination (10)(11)(12)(13).Despite a plethora of publications on vaccine hesitancy and the piqued interest of both experts and the public, to the best of our knowledge, there are few examples of rigorously tested vaccine communication strategies and interventions that have increased vaccine acceptance (14-16).
Experts agree that addressing vaccine hesitancy is contextspecific, requiring tailored interventions that include a range of vaccine communication strategies (7).Patient-provider race/ethnicity concordance-defined as the occurrence of matching patient race/ethnicity and provider race/ethnicity-has been associated with an increased likelihood of care-seeking and continued care-seeking behaviors (17), as well as better patient-provider communication (18).Moreover, in science communication, listeners have been found to delay or not develop counter-arguments when listening to peer/personal narratives (19).
Our study was conducted as a part of the CDC-funded COVID-19 Vaccines Information Equity and Demand Creation (COVIED) program (20)(21)(22), a body of work designed to increase COVID-19 vaccination through the use of evidence-based, contextspecific/tailored messaging.Based on our previous study on tailored vaccine education using racial/ethnic and gender diverse clinicians as credible sources and animation as a vehicle for conveying vaccine information (23), we conducted an internetbased, quasi-experimental study to evaluate the performance of 11 animated informational vaccine animation videos to address common vaccine attitudes and beliefs.We aimed to (1) evaluate the effect of using peer/personal narrative introductions (24) and (2) examine the role of race/ethnic congruence between the survey participant and a credible source (25) on the viewer engagement and their subsequent evaluation of the animated vaccine video intervention, randomizing on both video characteristics.We hypothesized that (1) introducing COVID-19 vaccine information with a personal narrative and (2) race/ethnic congruence between the survey participant and a credible source would be associated with an improvement in survey participant engagement, including an increase in the time spent viewing video content and positive ratings of video content.Additionally, we explored whether a credible source or participant's race/ethnicity was independently associated with viewer engagement and their evaluation of the intervention.

Study design
Using real-time interactive worldwide intelligence's (RIWI) patented Random Domain Intercept Technology (RDIT) (20)(21)(22), we implemented a national-level quasi-experimental design to evaluate 11 animated vaccine information videos with three variations of each based on different credible sources (i.e., clinical providers differentiating by race/ethnicity) who introduced and concluded each of the videos.A personal story video-narrated by an average peer of survey respondents, i.e., not a clinical providerwas created to precede or succeed eight (of the 11) videos that provided information on a common vaccine concern.

Development of intervention
We developed content tailored to each of the following relevant sub-populations of COVID-19 vaccine decision-makers: primary caregivers of children (i.e., <18 years of age) who have concerns about COVID-19 vaccines for their children, primary caregivers who do not have concerns about COVID-19 vaccines, non-caregivers who have concerns about COVID-19 vaccines for themselves, and non-caregivers who do not have concerns about COVID-19 vaccines.Formative research for our study was conducted using analyses of RIWI RDIT-derived data from two other national-level rapid response surveys designed to ascertain the public's COVID-19 vaccination attitudes and beliefs (21,22), rapid formative ethnography to elicit insights from subpopulations on the root causes and other related influencing factors of reluctance (and the intention) to receive a SARS-CoV-2 vaccine, and continuous message development and testing.Based on a similar process described elsewhere (23), we collaborated with a scriptwriter to develop evidence-driven video content grounded in the insights gained for each sub-population during our formative phase.We developed 11 animated vaccine informational videos that were refined through an iterative process between the scriptwriter and scientists to ensure content appropriateness, messaging true to current scientific knowledge, and application of defined behavioral theories (23).
All videos included an introduction and concluding message by a clinical provider as well as an animated informational video.Introduction and concluding recordings were performed by three clinicians representing different racial and ethnic backgrounds (Black, White, and Hispanic).The use of a personal story to introduce (or conclude) eight informational videos on COVID-19 vaccine concerns was developed based on the theory of change (26) that by establishing empathy and credibility and briefly addressing specific concerns followed by conveying disease risk and vaccine effectiveness, attitudes toward COVID-19 vaccination would become more positive.The effect is hypothesized to be greater if the message is introduced by a strong and personalized recommendation from a clinical provider (27).

Data collection
We implemented the quasi-experiment from 06 December 2021 to 01 January 2022.We collected participant sociodemographic characteristics and determined video intervention eligibility based on answers to five questions regarding (1) caregiver status, (2) COVID-19 vaccination status, (3) race and ethnicity, (4) presence of COVID-19 vaccination concerns, and (5) intention to get the COVID-19 vaccine among the unvaccinated respondents.Based on answers to these questions, participants were assigned to appropriate message pathways (Supplementary Figure 1).Supplementary Table 1 presents all 11 possible response-specific videos.

Video assignment
Child caregivers were stratified by those with any COVID-19 vaccine concern and those without a concern.All caregivers without any concern were assigned to view a video discussing benefits for the child (Child Benefit video).Caregivers who had a concern about the COVID-19 vaccine and infertility were randomly (3:1) assigned to view a concern video addressing their concern about infertility or the child benefit video.The remaining concerned caregivers were assigned to the child benefit video.
Among non-caregivers, any unvaccinated participant without COVID-19 vaccine concerns was assigned to a video discussing benefits for adults (adult benefit video).Vaccinated non-caregivers with a previous COVID-19 vaccine concern as well as any unvaccinated non-caregivers with a COVID-19 vaccine concern were asked a multiple-choice question-"what are/were your main concerns about the COVID-19 vaccine?"-and could select all that applied.Based on anticipated sample size limits, any participant indicating concern about vaccine ingredients (specifically, fetal cell lines) was automatically assigned to the concern video addressing this specific concern.Participants selecting only one concern were assigned to the associated concerned video.Participants who selected more than one concern (not about fetal cell lines) were randomly assigned to a concern video addressing one of the selected concerns.Participants who selected "other" concerns were assigned to view the adult benefit video.In order to test our theory of change using personal narrative to introduce vaccine risk and safety messages, one out of every eight participants assigned to view a concern video was randomly selected to view a personal story video before viewing the concern video (i.e., seven out of every eight viewed the personal story after).
Unvaccinated non-caregivers without concern about the COVID-19 vaccine were assigned to view the adult benefit video.Clinical provider race/ethnicity was randomly assigned (with equal probability) for all videos.Vaccinated non-caregivers without previous COVID-19 vaccine concerns were asked to participate in a brief survey about COVID-19 vaccine boosters.

Outcomes of interest
We used a continuous measure of the length of time (in seconds) each respondent spent viewing their assigned video and a standardized measure of the proportion of the video viewed based on the total length of each video.The total length included the introduction, informational animation, and concluding message.We then created a dichotomous outcome variable identifying those who fully viewed their assigned video and those who did not.For respondents viewing a concern video, the total length of viewing time included both the concern video and the personal story video.
Participants who completed viewing their assigned video were asked to provide their level of agreement or disagreement using a five-point Likert scale (strongly agree, agree, neither agree nor disagree, disagree, and strongly disagree) with three statements to evaluate their assigned video: (1) The video was helpful in making a vaccine decision.
(2) I trusted the information in the video.
(3) The video was easy to understand.
Viewers were asked to answer a fourth evaluation question ("how would this video influence others to get vaccinated?") using a five-point Likert scale (much more likely, somewhat more likely, no impact, somewhat less likely, and much less likely).We opted for a neutral phrasing of the question to avoid any appearance of favor toward or against vaccination.Likert scales were dichotomized for analysis by combining "strongly agree" and "agree" or "much more likely" and "somewhat more likely" to form agree vs. disagree and likely vs. unlikely categories, respectively.

Exposures of interest
The main exposure of interest was a categorical variable classifying respondents according to their assigned video type.For those assigned to any of the concern videos, separate categories were used to indicate whether the personal story video or concern video was viewed first.Sociodemographic characteristics of respondent age, sex, race and ethnicity, and COVID-19 vaccination status (vaccinated vs. unvaccinated), as well as provider's race and ethnicity, were measured.We created a dichotomous variable to indicate racial/ethnic congruence between the provider and the respondent.Respondent location was categorized according to nine regions using Centers for Disease Control and Prevention (CDC) definitions (28).

Statistical analysis
For descriptive analyses, we compared distributions using chisquare and Fisher's exact tests for proportions, Wilcoxon ranksum for non-parametric data, and Student's t-test for normally distributed data.No covariates had missingness of data exceeding 5%.Analyses were performed using the two-sided significance level (0.05).All analyses were conducted using Stata 16.1 (StataCorp, College Station, TX).
We used multivariable logistic regression to model the log odds of each of our dichotomous outcomes of interest: (1) viewing the assigned video in its entirety vs. not, and the respondent agreed (or disagreed) the following of the video; (2) is easy to understand; (3) is helpful for making vaccination decisions; (4) information is trustworthy; and (5) will influence others to get vaccinated.Table 1 shows video and respondent characteristics included in a backward stepwise selection process using 0.2 as the level of significance.We reviewed both the statistical significance of a fixed term used to control for survey date and interaction terms between provider-respondent racial congruence and provider race (and secondly, respondent race), and the results of a likelihood ratio test comparing extended and nested models that included (and subsequently excluded) a survey data fixed term.We reviewed the residual plots as well as Hosmer-Lemeshow chi-square estimates to evaluate the model goodness of fit.We used a robust variance estimator to adjust for clustering on respondent location (region).

Study population characteristics-descriptive analysis
Among 117,750 individuals who initially reached for participating, 75,616 (64.2%) completed the first five qualifying questions required for video assignment and 14,235 (18.8%) started to view the assigned video, allowing us to evaluate the length of time spent viewing the assigned video (e.g., the proportion of viewers who completed viewing their assigned video).Among those who started, 2,422 (17.0%) completed viewing the full video and answered at least one video and content quality or potential video utility evaluation question.

Evaluation of viewing completion
In Table 3, the odds of completing the view of the assigned video are adjusted for the assigned video, provider race and ethnicity, and respondent race and ethnicity, sex, age, and COVID-19 vaccination status (vaccinated vs. unvaccinated).When compared to those who viewed the assigned concern video first (personal story second), viewers of the personal story first (concern video second) were ∼10 times (p < 0.01) more likely to watch the complete informational animation.Viewers of the personal story first (concern video second) were also ∼4.5 times (p < 0.01) more likely to watch the complete informational animation compared to viewers of the infertility concern video, as well as 20 times (p < 0.01) and nine times (p < 0.01) more likely to watch than the viewers of the child benefit and adult benefit videos, respectively.Unadjusted estimates of viewing time measures are provided in Supplementary Tables 2, 3.
Those viewing introduction/concluding material presented by the Black or Hispanic physician had lower odds of fully viewing the assigned video (58%, p < 0.01; 82%, p < 0.01) than those viewing videos presented by the White physician (Table 3).There was title difference between the adjusted and crude odds ra8os for respondent sex, age, race and ethnicity, and COVID-19 vaccina8on status except that viewers who self-iden8fied as American Indian (AI), Alaskan Na8ve (AN), or Other had lower odds of fully viewing the assigned video (27%, p < 0.01) compared to their White counterparts as did Asian viewers (40% lower, p < 0.01).Those reporting a multi-racial and/or ethnic identity had greater odds of completely viewing the assigned video (aOR 1.30, 95% CI 0.99, 1.71) though the When adjusting for racial congruence between the provider and the viewer/respondent (Table 4), the odds of fully viewing the assigned video were lower for all videos compared to personal story video first (concern video second) viewers (all p-values < 0.01).Racial congruence was associated with increased odds of fully viewing a video (aOR: 1.89, p < 0.01).

Evaluation of video content (post-viewing)
Overall, the odds of evaluating the video positively (easier to understand, helpful for making vaccination decisions, providing trusted information, and influencing others to get vaccinated) were greater among those who viewed the personal story first (concern video second) though the adjusted odds ratio was statistically significant only when asking about the video's influence on others to get vaccinated (1.6 times greater odds, p < 0.01; Table 3).Independently, the provider's race was not significantly associated with a positive evaluation of the videos, whereas respondent race/ethnicity was significantly associated with positive evaluation, and mainly Black, Asian, and Hispanic/LatinX have greater odds (aOR: 1.4-2.1 where p < 0.01) of evaluating the videos positively [and American Indian or Alaskan Native (AI/AN)] or others have a lower odds (≤46% lower odds where p < 0.01) than their White counterparts (Table 3).Racial congruence was associated with increased odds of evaluating the video as influencing others to get vaccinated (Table 4; aOR: 1.14, p = 0.03).When adjusting for racial congruence, female viewers had greater odds (1.3 greater odds compared to males, all p ≤ 0.03) of positively evaluating their assigned videos whereas older adults (75+ years of age; ≤53% lower odds where p ≤ 0.02) and the unvaccinated ones had lower odds (≤84% lower odds where p < 0.01; Table 4).

Sub-analyses-potential confounding and e ect modifying e ects
There was no evidence of a confounding effect of survey time on the primary relationship of interest (p = 0.97).Moreover, we found no evidence that provider race (or respondent race) acts as an effect modifier on the relationship between provider-respondent racial congruence and our outcomes of interest (p > 0.40 and p > 0.30 for provider and respondent race, respectively).

Discussion
Using peer-delivered, personal narratives to introduce vaccine safety information may increase the likelihood that viewers will engage with informational vaccine videos.Despite fewer clear benefits in the likelihood of videos receiving positive evaluations, personal story videos had a consistently improved effect on the likelihood that viewers thought their respective videos would influence others to get vaccinated.Emotional engagement-an important part of communication strategies developed to engage the public for fostering vaccine confidencehas been a central part of health behavior change research and practice (23, 29, 30).Emotional engagement and transparent communication likely serve as important tools for messaging and vaccination program administration, particularly during periods of heightened collective and diverse emotions among the public such as the COVID-19 pandemic (30).Moreover, the use of personal narratives may foster learning environments of openness (open-mindedness) (31) for vaccine messages designed to build general confidence and understanding of evidencebased medicine (32).In light of these findings and previous research that has highlighted the polarization of social media content between positive and negative-toned content on vaccines, especially vaccine-hesitant topics (33), our findings on the influence of peer-delivered, personal narrative to introduce vaccine safety information may be an important area of further research.Our findings suggest the importance of racial congruence between the patient and the provider in vaccine safety communication and also further support the importance of identifying sub-population attitudes (e.g., by race and ethnicity) and tailoring messages.Race and ethnicity have been identified as factors associated with COVID-19 (and other) vaccine uptake and vaccine hesitancy (25,(34)(35)(36).In general, healthcare providers who communicate effectively with patients are known influencers of vaccine uptake (25,37,38).However, some evidence supports that persons self-identifying as non-Hispanic White are more likely to receive a healthcare provider's recommendation than racial and ethnic minorities (37).Studies have shown that patients who are racially and/or ethnically concordant with their provider report greater satisfaction, levels of trust, and perceived quality of care (17).Although concordance has been found to affect patients' clinical encounter experiences and relates to better patient-physician communication, there is no general consensus on the positive effect of racial/ethnic patient-provider concordance on patient outcomes (nor specifically on effective vaccine communication) (18).Our study indicates support for this theory.
We found that while unvaccinated respondents were more likely to fully view their assigned video compared to vaccinated respondents, they were also less likely to give positive feedback on video content and usefulness.The majority of unvaccinated respondents (62%) agreed or strongly agreed that the video content was easy to understand, but greater proportions of unvaccinated respondents disagreed or strongly disagreed that the videos were helpful for making vaccine decisions or influencing others to get vaccinated or that the information was trusted than agreed or strongly agreed (69, 68, and 74%, respectively).Even before the COVID-19 pandemic, vaccine hesitancy and refusal were identified as public health concerns (4, 8), with emotion-driven vaccine beliefs spreading across (and even flourish) during the pandemic (30,39).Regardless of their interest to view the content of our informational videos, deliberate efforts to engage viewers in a positive manner may rather activate emotions that decrease the likelihood of positive feedback (30).Resistance to vaccination is complex, and positive vaccine messages may have unintended and undesirable consequences (25,40).Rigorous approaches to both measuring latent vaccination attitudes and beliefs and testing interventions for their effect on vaccination behavior (i.e., uptake) are needed and must take the psychology behind health decision-making into consideration (23, 25, 41, 42).

Limitations
Selection bias may have been introduced due to the opt-in, unincentivized design of the study.We explored the distribution of sociodemographic characteristics (Supplementary Table 4) among those who were lost to follow-up (i.e., early drop-out and did not get assigned to a video or did not start viewing the assigned video), noting that the proportion of these individuals are more vaccinated (77.3 vs. 55.0%,p < 0.01) and have fewer reported COVID-19 vaccine concerns than those described in our study sample (26.3 vs. 52.7%,p < 0.01).This may strengthen our findings that suggest the usefulness of our vaccine communication videos among specific sub-populations of the American public, noting that the distribution of intentions to complete the COVID-19 vaccine series or booster were similar between those lost to follow-up and those in our study (Supplementary Table 4).We note that our sample size may have limited our ability to identify statistically significant differences in the odds of viewing the entire assigned video among the multi-racial and/or ethnic subgroup compared to their racial/ethnic counterparts.The focus of this study was on the evaluation of the whole package of information and associations with demographic characteristics and vaccine intentions and concerns.Thus, we were not able to determine if vaccination rates improved in persons who viewed full videos compared to others.Further research is needed to evaluate this question.

Public health implications
Our study findings further support the importance of tailoring vaccine communication strategies to sub-population vaccine attitudes by delivering vaccine messages through trusted, race/ethnicity-congruent providers or other trusted health authorities.Introducing vaccine safety information with peerdelivered, personal narratives may improve the openness of vaccine message recipients to vaccine messages and engagement.Further research is needed to evaluate the effect of vaccine safety informational video packages on vaccine uptake.Additionally,   further formative work is needed to explore message engagement among sub-populations that maintain fewer positive views of vaccine safety information.

TABLE Demographic ,
survey, and COVID-vaccination characteristics among non-caregivers, stratified by completely viewing the assigned video or not.

TABLE Demographic ,
survey, and COVID-vaccination characteristics among caregivers, stratified by completely viewing the assigned video or not.
TABLE Association between fully viewing the assigned video or positive opinions of the viewed informational video and video and viewer sociodemographic characteristics.
* Statistically significant point estimates (p < 0.05) are bolded.a Concern videos include: Benefits of vaccination for pregnancy, COVID-19 is not that serious, Concerned about common side effects, Concerned about vaccine ingredients, Concerned about fetal cell line, Concerned about general safety (of COVID-19 vaccines), Vaccines were developed too fast, and Serious side effects (see Supplementary Table 1).The total viewing time-regardless of viewing order-includes both Concern and Personal Story video lengths added together.b Video for those Concerned about infertility (Supplementary Table 1 TABLE Association between fully viewing the assigned video or positive opinions of the viewed informational video and video and viewer sociodemographic characteristics that include provider-respondent racial congruence.Benefits of vaccination for pregnancy, COVID-19 is not that serious, Concerned about common side effects, Concerned about vaccine ingredients, Concerned about fetal cell line, Concerned about general safety (of COVID-19 vaccines), Vaccines were developed too fast, and Serious side effects (see Supplementary Table 1).The total viewing time-regardless of viewing order-includes both Concern and Personal Story video lengths added together.Video about the Benefits of vaccination for adults.The bold values are used where the associated p-value indicates statistical significance <0.05.
). c Video about the Benefits of vaccination for children.d Video about the Benefits of vaccination for adults.a Concern videos include: b Video for those Concerned about infertility (Supplementary Table 1).c Video about the Benefits of vaccination for children.d