Moving Toward Inclusion: Participant Responses to the Inclusive SciComm Symposium

This study shares key findings from evaluation research for Inclusive SciComm: A Symposium on Advancing Inclusive Public Engagement with Science. The symposium, organized by the University of Rhode Island's Metcalf Institute for Marine & Environmental Reporting with support from partner organizations, took place on September 28 and 29, 2018 at the University of Rhode Island. Pre- and post-symposium surveys showed that after attending the symposium, participants reported higher levels of knowledge about and confidence in implementing inclusive approaches to science communication. Participants also exhibited three types of response orientations: emotion, knowledge, and action.


INTRODUCTION
Social inclusion is an emerging area of importance in the field of science communication (see Canfield et al., this issue). The discipline of science communication itself is still growing and the term science communication has been defined in a variety of ways, with little clarity as to how it is differentiated from other associated terms such as public engagement with science, public understanding of science, and even outreach or broader impacts (Burns et al., 2003;Trench and Bucchi, 2010). We define science communication here as "the exchange of information and viewpoints about science to achieve a goal or objective such as fostering greater understanding of science and scientific methods or gaining greater insight into diverse public views and concerns about the science related to a contentious issue" (National Academies of Sciences Engineering and Medicine, 2010, p. 1, 2). We use this definition specifically because it emphasizes a bi-directional relationship that notes understanding and growth on both the part of scientists and the public. This definition of science communication aligns with how others define public engagement. The American Association for the Advancement of Science defines public engagement as "intentional, meaningful interactions that provide opportunities for mutual learning between scientists and members of the public" (American Association for the Advancement of Science, 2019, par 1). While we recognize that these two terms are often separated in the literature, they are also often times conflated, and we see both definitions aligning around goals of mutual learning and information sharing.
• Identify needs and opportunities for more inclusive, intersectional, and asset-based approaches to science communication and public engagement. • Highlight the work of science communication and public engagement practitioners and researchers (from academia, non-profits, public, and private sectors) whose work demonstrates effective inclusive and intersectional approaches for the fields. • Discuss the structural problems that hinder inclusive approaches and how these problems can be addressed. • Inspire new collaborations among attendees and provide practical information that attendees can implement in their work to prioritize inclusion.
The study was designed to evaluate the symposium, its impact on participant knowledge and efficacy, and ask exploratory questions about participant experiences with inclusive science communication. The study specifically aimed to address the following questions: 1) Did attending the symposium increase attendees' knowledge of and confidence in enacting inclusive approaches? 2) What do participants view as the biggest barriers and opportunities for inclusive engagement and science communication? 3) How did participants respond to the symposium experience?

METHODS
This study took place in Fall 2018. Data were collected in two online surveys, one pre-test and one post-test, both administered through Qualtrics. The two surveys asked both closed ended and opened ended questions about attendee perception and experience of the science symposium (complete surveys available as Supplementary Material). Surveys were chosen as the method for data collection because of funding and time constraints. For the upcoming 2019 symposium, researchers have added focus groups to account for the need to include more in-depth qualitative analysis.
One-hundred-fifty registered symposium attendees were invited to participate in this research. The symposium organizers provided an email list of all registered participants, which included the speakers and planning committee. Attendees received notification about the study from the lead organizer of the symposium, and then three initial recruitment emails were sent, each 2 days a part, during the week prior to the symposium. The post-test survey followed the same protocol, with three recruitment emails being sent in the 2 weeks following the symposium.
This pre-symposium survey was designed to take respondents ∼5-10 min to complete and assessed participant expectations for the symposium.
The post-symposium survey was designed to take respondents ∼10-15 min to complete and assessed participant experiences and reflections after symposium attendance.
The pre-test survey return rate was 53% (N = 80). The post-test survey return rate was 36% (N = 54). A total of 45 participants completed both the pre-symposium and post-symposium surveys. Several survey questions for were repeated in both the pre-test and post-test. Some additions were added to the post-symposium survey to gauge attendee perception of specific symposium sessions and events. For the majority of this study, researchers used all post-symposium survey responses. For the measurements of change between preand post-symposium responses, researchers only used matched response data from participants who completed both the pre-and post-tests (N = 45). Each results subsection, below, indicates which specific data were used for specific analyses.

Descriptive Statistics of Survey Respondents
Demographics were collected only for the initial (pretest) survey. Of the 80 participants who responded, 78% reported a female gender, 20% reported male, 1% identified non-binary/third gender, and 1% preferred not to identify.
Of the 80 respondents, most (97%) held higher education degrees. When asked to report their most advanced degree, 46% had doctoral degrees, 11% had partial credit toward a doctorate, 20% had master's degrees, 4% had partial credit toward a master's, and 16% had bachelor's degrees.
Participants represented diverse fields of work. The largest group of participants were in natural science research (24%), followed by the non-profit sector (14%), post-secondary education (13%), informal science education (9%), and social science research (9%). Other represented fields included art, K-12 education, education administration, funding, journalism, government regulatory agencies, government non-regulatory agencies, science communication training, science policy, and graduate studies.
A limitation of this study is that demographics and race/ethnicity were not included due to miscommunication among the conference planning and research team about whether this information was being collected during registration or through the survey instrument. The research team regrets this error and it has been corrected for the 2019 evaluation research, yet still believes the findings reported are useful for knowledge-building purposes in the growing area of inclusive science communication.

Changes in Participant Knowledge and Efficacy
RQ1 asked: Did attending the symposium increase attendees' knowledge of and confidence in enacting inclusive approaches?
The results reported in this section are based on participant data of the 45 survey respondents who completed both the pre-and post-test surveys. Surveys were matched based on a unique ID code assigned through Qualtrics. Results show both a significant positive effect in self-reported participant knowledge and efficacy after the symposium.
Pre-and post-event surveys asked attendees a variety of questions regarding their knowledge and confidence about inclusive science communication on a five-point Likert scale. Knowledge-based questions asked how knowledgeable individuals were at: identifying challenges related to inclusive science communication and public engagement with science (PES); identifying opportunities related to achieving inclusive science communication and PES; implementing strategies for creating more inclusive science communication and PES practices; implementing strategies for creating more inclusive science communication and PES research; implementing strategies for overcoming structural barriers that hinder inclusive approaches; and identifying new ways to become engaged in science communication and PES (see Table 1 for knowledge-based frequencies). All knowledge-based questions were compiled into a composite and a mean score was then calculated. Next, to ensure reliability of the composite, a scale reliability test was run for pre-test and post-test knowledge-based questions. The Cronbach's alpha for the 6 knowledge-based items was 0.61 for the pre-test questions and 0.91 for the post-test questions. A Cronbach alpha of 0.61 is considered low but acceptable for exploratory communication research (Boyle and Schmierbach, 2015) and researchers expect that the pre-test alpha was low because of variation between some participants' experience with certain scale items measured upon entering the conference (i.e., some people came in with more knowledge of certain inclusion-related topics than others).
A repeated-measures analysis of variance (RM-ANOVA) was performed examining change over time in knowledgebased survey questions. There were two time points (Pre-test 1, Post-test 2). The analysis was performed using SPSS 24.0 for Mac.
The main effect of time on knowledge-based questions was significant, F (1, 46) = 104.132, p ≤ 0.001, partial η 2 = 0.69. Therefore, the nature of change included a positive linear effect; Symposium participants reported feeling significantly more knowledgeable about inclusive science communication strategies after the symposium (see Table 2 for knowledge-based mean data).
The efficacy-based questions asked individuals how confident they were regarding the same six actions asked for the knowledge-based questions. All confidence-based questions were compiled into a composite and a mean score was then calculated. Next, to ensure reliability of the composite, a scale reliability test was run for pre-test and post-test confidence-based questions. The Cronbach's alpha for the 6 confident-based items was 0.88 for the pre-test questions and 0.93 for the post-test questions (see Table 3 for confidence-based frequencies).
A repeated-measures analysis of variance (RM-ANOVA) was performed examining change over time in confidence-based survey questions. There were two time points (Pre-test 1, Posttest 2). The analysis was performed using SPSS 24.0 for Mac.
The main effect of time on confidence-based questions was significant, F (1, 44) = 70.129, p ≤ 0.001, partial η 2 = 0.61. The nature of change included a positive linear effect: Symposium participants reported feeling significantly more confident about inclusive science communication strategies after attending the symposium (see Table 4 for confidence-based mean data).

Identified Barriers for Advancing Inclusive Science Communication
RQ2 asked: What do participants view as the biggest barriers and opportunities for inclusive engagement and science communication? In the post-survey, participants were asked what they saw as key barriers for science communication and public engagement to become more inclusive. These questions were intentionally broad, allowing participants to share their own lived experiences. Participant responses were thematically coded using the method of constant comparison (Corbin and Strauss, 2008), which involves researchers looking for (1) prevalent themes from among all, or at least several, of the participant responses, and (2) discrepancies and differences among participant responses. Overall, responses were divided into two categories: barriers caused by presence and barriers caused by absence.
Barriers caused by presence indicate that some occurrence is keeping science communication and public engagement from being more inclusive. The most common respondent examples in this category were existing organizational structures in research and the academy, followed by inherent, unconscious, and implicit biases. Other responses included: white communicators not sharing leadership spaces with non-white communicators; laziness, stubbornness, or resistance toward inclusion efforts which result in fatigue for those doing inclusive science communication and public engagement; siloing of research and information; and geographic, linguistic, financial, cultural, and socioeconomic status factors.
Barriers caused by absence indicate that something is missing, which keeps science communication and public engagement from being more inclusive. The most common respondent examples in this category include lack of funding followed by lack of understanding, knowledge, training, or resources for doing inclusive science communication work. Other responses include: not assessing if inclusion efforts are actually inclusive; inadequate diversity among leadership in science communication efforts; limited opportunities or platforms; minimal networking, collaboration, or sharing of When asked what they saw as key opportunities for making science communication and public engagement more inclusive, participant responses varied widely. Responses most heavily emphasized knowing and understanding diverse audiences and responding to those audiences by moving science out of the academy and into communities. Responses also indicated the importance of creating connections and building trust with the audience by engaging science communicators from marginalized groups. Other themes that appeared but with less prevalence across participant responses included making science education more inclusive, intentionally making space for and elevating diverse voices, changing restrictive institutional structures, measuring and assessing inclusive science communication approaches, creating a network of inclusive science communicators, and improving wider understanding of what inclusive science communication means and needs.

Response Orientations
RQ3 asked: How did participants respond to the symposium experience?
In the post-survey, participants explained how the symposium affected their perceptions of inclusive science communication and public engagement with science. Using the method of constant comparison (Corbin and Strauss, 2008), researchers examined responses to develop local concepts from the participants' experiences. Researchers found that participant responses aligned in three major themes: emotion-oriented, knowledge-oriented, and action-oriented. Some participants expressed one of these response orientations, while others expressed multiple orientations in their post-symposium reflections. Thus, it is difficult to say that a certain number of participants expressed one orientation more than another, but the prevalence with which certain types of concepts arose in responses resulted in the distillation of these three response orientations. For the purpose of this study, written statements were coded to one category based on the overarching or dominant sentiment of the statement. However, we acknowledge that crossover, especially with emotion/affect and knowledge, occurred. For example, the statement, "It pushed me to think differently, but I was also frustrated by some of the attendees who didn't seem to really understand how they are part of the problem, " was coded as knowledge-based because the statement indicates that they thought differently or learned as a result of what was encountered at the event. At the same time, knowing that others lacked an understanding of their role in existing structures impacted the participant's emotional state and caused frustration.

Emotion-Oriented Response
Responses that articulated that the symposium made participants experience different affects during and after attending were coded as emotion-oriented responses. These responses were primarily positive, but some participants identified feeling overwhelmed or disheartened at the current state of inclusivity within science communication and PES. Below are examples of emotionoriented responses: "It made me feel very hopeful and more confident." "It's almost overwhelming to realize how far we have to go in some respects." "It was a powerful validation that what I have been doing is important and there is much to do." "The meeting was very powerful and motivating for me personally. Powerful in the sense that we had truly meaningful dialogue with one another [to] talk about the hard issues surrounding inclusion; some of these conversations were triggering of my personal adversities or those of other attendees. It was those experiences that actually provide a surge of motivation in me to focus on educating myself further and changing my personal practices." "The symposium didn't just help me develop the way I think about inclusive scicomm/PES but made me feel more secure in my role in the movement." These responses suggest that rich engagement, including keynote speakers and group discussions, had meaningful impacts on participants and influenced their feelings during and after symposium participation. This finding suggests that affect may play an important role in symposium participants' perceptions of inclusive science communication and public engagement with science.

Knowledge-Oriented Response
Participants who expressed a knowledge-oriented response explained that the symposium made them think differently or taught them something (Mack et al., 2012;Featherstone, 2014). This theme had the most responses. Below are examples of knowledge-oriented responses: "I learned so much about deep challenges and potential solutions to these challenges." "I feel much more aware of the issues at-hand." "It pushed me to think differently, but I was also frustrated by some of the attendees who didn't seem to really understand how they are part of the problem." "Being welcomed at this symposium made it clear to me that being white doesn't preclude me from being a part of this conversation." ". . . there are many different ways to engage with the public that I had not previously considered." "Access and the barriers on that road of accessibility are so much more treacherous, winding, and uncharted, than I originally thought. I was uncomfortable for most of the symposium and felt out of place. . . this was an enlightening step toward recognizing what needs to happen if things are going to change in science communication and public engagement." Knowledge-oriented responses indicate that participants' experiences were informative in a variety of ways. These included, as illustrated in the examples above, better understandings of specific issues, general awareness, new ways of thinking, and better understanding of one's own experiences as related to inclusive science communication. Ultimately the knowledgeoriented responses indicate that participants left the symposium with new understandings about science communication and their relationship to it.

Action-Oriented Response
Participants who expressed an action-oriented response explained that the symposium gave them tools or motivation to act differently after leaving (Massarani and Merzagora, 2014;Streicher et al., 2014). Below are examples of action-oriented responses: "It encouraged me to think more about specifically asking the needs of my students and working to provide tools that help them accomplish their goals, rather than setting too many concrete goals for a class myself." ". . . I both know and can identify more of the barriers to inclusive science communication, but I also feel like I have more tools and strategies to overcome those barriers." ". . . going forward I will use my connections/privilege to raise up the voices and experiences of minority scientists." This response orientation indicates that the symposium allowed participants to feel empowered to do inclusive science communication. These responses indicate the potential of such a symposium to cause participants to see a need to change their behaviors toward creating more inclusive science communication. Some participants expressed one of these response orientations, while others expressed multiple orientations and some expressed none (i.e., "None" or "It was fabulous") in their post-symposium reflections. Thus, it is difficult to say that a certain number of participants expressed one orientation more than another. But the prevalence of response orientations across all participant reflections shows that knowledge-oriented responses were most prevalent (14 instances), followed by emotion-oriented responses (nine instances) and action-based responses (six instances).
These three responses orientations-emotion, knowledge, and action-indicate the ways in which participants responded to their experience at this symposium. Each provides a distinct way in which respondents reported being affected by the symposium and their perceptions of inclusive science communication and public engagement with science. Responses suggest that this kind of symposium has the potential to provide transformative experiences for participants in multiple ways. In the case of this symposium, respondents described changes in their affect toward, understanding of, and ability to act on inclusive science communication and public engagement with science. Thus, it is important for inclusive science communication symposium organizers to think beyond merely informing attendees and to consider the transformational potential of engaging participants' emotional responses and empowering them with actionable tools. We recommend acknowledging, responding to, and further studying the dynamic and interconnected nature of information, affect, and action in doing inclusive science communication, as evidenced by our sample's responses.
After attending, participants shared the specific activities or networks they would like to develop or participate in to advance inclusive science communication/public engagement with science on a national scale. Answers varied, but creating or joining online networks for inclusive science communication was a popular answer. This included developing searchable databases around inclusion efforts for ideas and to see what does or does not work, hosting a network for best practices and creative solutions to local challenges, establishing an email listserv, and making available more webinars and digital discussions. Other responses addressed support for working and networking with large organizations. This included community organizations, such as YMCA, and larger science organizations. Some responses directly addressed educational efforts (Calabrese Barton and Tan, 2019), including developing guidelines and organizational resource banks for STEM institutions, developing a pedagogy of inclusion group/network, and broadening existing STEM outreach programs (e.g., Ask a Scientist and Skype a Scientist) by including more diverse scientists and schools. A few participants identified interest in support for working with news media and others identified support for storytelling events. One respondent noted a lack of attendees from Midwestern and Southern states at the symposium and suggested that national activities and networks cannot exist until all areas are represented and active. Another noted the emergent theme of an "urgent need for dialogue, " and said that they would welcome more preparation about "facilitating difficult conversations about getting out of the way and lifting up."

STUDY LIMITATIONS
Limitations include practical considerations of administering an online survey, including lack of participant time, survey fatigue, and lack of tangible incentive. Limitations also include the exclusion of race/ethnicity from the survey's demographic questions due to an oversight by researchers, but this has been corrected for the ongoing 2019 study. An additional limitation is that participants were largely homogeneous, with the majority being females with advanced degrees, especially in natural science. Another limitation is the sample size, which decreased from the pre-test to the post-test survey. A final limitation is that of the method (survey) which does not allow for follow-up questioning or clarifications for qualitative responses. These limitations may have influenced results as the respondent pool was inherently reflective of the symposium being held at a university with a largely highly-educated audience. These limitations have been discussed at length in the interpretation of data and researchers do acknowledge that the lack of responses from certain diverse occupations, fields, and organizations.

CONCLUSION
Results from this study indicate that the symposium increased participant knowledge of and confidence in enacting inclusive approaches, reflecting the symposium's objectives to identify needs and opportunities for more inclusive science communication and PES, and to discuss structural problems and how these problems can be addressed. The qualitative data also make clear that this symposium had impacts on attendees. Attendee responses were emotion-oriented, knowledge-oriented, and/or action-oriented, indicating that the event achieved its intended objectives of highlighting varied approaches to science communication, discussing structural problems and solutions, providing practical information for implementation and inspiring new collaborations among attendees. Participants reflected on how they can apply what they heard and learned during the symposium, in various ways, in their own work, again reflecting the event's objectives of addressing structural problems and providing practical information that attendees can implement.
RQ2 asked what participants view as the biggest barriers and opportunities for inclusive engagement and science communication. Participant responses suggested that participants have experienced barriers in inclusive science communication caused by presence and caused by absence. These responses corroborate existing research literature. Regarding barriers caused by presence included existing organizational structures in research and the academy (Chilvers, 2012); siloing of research and information (Falk et al., 2011;Chilvers, 2012;National Science Foundation, 2018); inherent, unconscious, and implicit biases (Christidou, 2011;Taylor, 2014); white communicators not sharing leadership spaces with non-white communicators (Taylor, 2014(Taylor, , 2018; and laziness, stubbornness, or resistance toward inclusion efforts which result in fatigue for those doing inclusive science communication and public engagement (DiAngelo, 2012;Feinstein and Meshoulam, 2014;Bang et al., 2018).
The case is the same for barriers caused by absence: current literature indicates similar examples. These include lack of funding (Mack et al., 2012;Taylor, 2014); lack of understanding, knowledge, training, or resources for doing inclusive science communication work (Dawson, 2014;Feinstein and Meshoulam, 2014); not assessing if inclusion efforts are actually inclusive (Mack et al., 2012;Featherstone, 2014); inadequate diversity among leadership in science communication efforts (Feinstein and Meshoulam, 2014;Pearson and Schuldt, 2014;National Science Foundation, 2018); limited opportunities or platforms; minimal networking, collaboration, or sharing of information (Falk et al., 2011;Chilvers, 2012;Berditchevskaia et al., 2017); and few opportunities for diverse, young scientists to be engaged in science (Calabrese Tan, 2010, 2019).
Responses about opportunities were less cohesive than those about barriers, however the most common responses reflected understanding and connecting with diverse communities, and engaging members of diverse communities as science communicators. These responses indicate a need for further research on understanding the role of diverse voices in science communication as science communicators, community liaisons, and audiences.
Participants demonstrated a strong desire to continue a national conversation about how to increase inclusion in science communication and public engagement with science. Respondents offered various mechanisms for this, suggesting the creation of online networks, an online resource hub, or partnerships with existing institutions in education, research, or community (Davies et al., 2009;Feinstein and Meshoulam, 2014;Hobbs et al., 2019). These findings highlight the need to expand opportunities for online and in-person discussions about how to prioritize and achieve inclusive approaches to science communication (see Canfield et al., this issue;Falk et al., 2011). These networks and events could help participants work through identified barriers and opportunities to inclusive engagement and, importantly, build new collaborations, especially between researchers and practitioners. Participants identified structural barriers or deficiencies as some of the most difficult to address, such as lack of funding for this type of work (Dawson, 2012;Mack et al., 2012). Participants identified innovative strategies for moving science communication out of the academy and into more culturally-contextualized settings, offering ideas of storytelling events and partnering with already-established community groups.
Finally, this study highlights the need for more coordinated efforts for inclusive science communication engagement that spans geography, audience, and scale. While participants identified various areas for development, it was clear that there was a desire for more information-sharing and collaboration across contexts to help practitioners, researchers, and other interested groups learn from each other's successes and failures (Falk et al., 2011;Featherstone, 2014;Treffry-Goatley, 2014).

FUTURE WORK AND USE OF RESULTS
The goal of this evaluation research was to identify what participants wanted from Inclusive SciComm: A Symposium on Advancing Inclusive Public Engagement with Science, what their experiences were at the symposium, and their broader reflections on inclusive science communication after attending the symposium. This study is immediately useful for informing the design of future convenings with similar objectives, helping organizers understand how to be more responsive to participants' needs, expectations, and experiences. Inclusive science communication, defined in its broadest sense, is an area ripe for further study. Convenings like the InclusiveSciComm Symposium can help identify research gaps that, once addressed, could truly expand inclusive practice. This research provides insights for inclusive science communication researchers and practitioners based on the experiences of participants by examining their perceived knowledge and confidence after attending, clarifying their perspectives on barriers and opportunities for inclusive science communication and engagement, and understanding their responses to the symposium experience.

DATA AVAILABILITY STATEMENT
The datasets generated for this study will not be made publicly available as they are protected by IRB.

ETHICS STATEMENT
The studies involving human participants were reviewed and approved by University of Oregon Research Compliance Services. The patients/participants provided their written informed consent to participate in this study.