Sarah B. Cavanah
Kaylan Kemink- 1William Allen White School of Journalism and Mass Communications, The University of Kansas, Lawrence, MA, United States
- 2Ducks Unlimited, Inc., Bismarck, ND, United States
The dissemination of scientific knowledge to the public, policymakers, and practitioners is crucial, particularly in democratic societies. Brokers, intermediaries, and boundary spanners (BIBS) facilitate this transfer through relationship-building and capacity development. Social media, widely adopted globally, provides BIBS with an important platform for engagement. However, current metrics, such as reach, inadequately measure the impact of these efforts on behavior and societal change. A Science Communication in Social Media Theory of Change (SciSM ToC) framework is proposed to align social media goals with appropriate evaluation methods. By integrating theory-based evaluation and strategic communication approaches, this framework provides a systematic process for planning, implementing, and assessing social media campaigns by BIBS. The SciSM ToC delineates inputs, activities, outputs, and outcomes, culminating in societal-level impacts. It incorporates established behavioral change models, such as the COM-B model, to address capacity factors (capability, opportunity, motivation) and emphasizes engagement as a dynamic, multidimensional process. Metrics are proposed to evaluate cognitive, affective, and behavioral changes, integrating practical tools like platform analytics and surveys. The framework allows for iterative evaluation, ensuring alignment with long-term goals and adaptation to diverse audiences. While limitations include its generalizability and exclusion of specific message design guidance, the SciSM ToC provides a flexible, practical tool for BIBS to maximize their social media effectiveness.
1 Introduction
The ability to transfer research evidence and implications from the producers of research to practitioners, policymakers, and the public is a major concern, especially in democratic societies where the electorate must make decisions involving scientific knowledge and understanding (Motta, 2024). This transfer involves individuals and organizations from many science-based disciplines, collectively referred to as brokers, intermediaries, and boundary spanners (BIBS). BIBS are involved in the dissemination of research evidence, building capacity for scientific understanding in the public, and relationship building between researchers and the public (Neal et al., 2022). BIBS can be individual researchers, non-governmental organizations, governmental bodies, university centers, or any individual or organization with an explicit goal of helping move scientific knowledge into the public. BIBS operate in complex social environments with often subtle, gradual, and difficult-to-evaluate goals (Posner and Cvitanovic, 2019), involving knowledge transfer (Smedlund, 2006), but even more so encouraging better use of knowledge in decision-making processes (Van Enst et al., 2017) to lead to positive social change (Maag et al., 2018); facilitating relationships between different stakeholder groups impacted by the creation and use of knowledge (Bornbaum et al., 2015); supporting innovation in processes and technologies (Bornbaum et al., 2015; Feser, 2023); and increasing trust in research and knowledge production (Posner and Cvitanovic, 2019).
The worldwide adoption of social media by NGOs (Nonprofit Tech for Good, 2019), governments (CivicPlus, 2022), and businesses (Balan and Rege, 2017) has helped to facilitate this process to some extent (Wilkinson and Weitkamp, 2013). As of October 2024, 64% of people worldwide were active social media users, a 5.2% growth from the previous 12 months, with the average user spending nearly 2.5 h on social media each day across several different platforms (Kemp, 2024). Given the importance of social media for the average individual, it is unsurprising that most BIBS use social media communication to help achieve their mission-related outcomes (Arts et al., 2015). In a study of 130 global conservation-based non-profits, Kim et al. (2024) found that 97% were using social media to communicate with the public.
Despite increasing use of social media by BIBS, there is little guidance to support effective approaches for the development of content, nor measurement of efficacy. When evaluation of social media efforts is reported by BIBS, it most commonly involves the number of people outside the professional scientific community exposed to content, a metric referred to as reach (e.g., Adetunji and Renoe, 2017; Kim et al., 2024). While reach is a useful metric, it can contribute to a demonstrated bias in science communication referred to as the Deficit Model, in which science communicators can believe that mere exposure to scientific content is enough to achieve their goals with the public (Simis et al., 2016), and thus communication efforts included as “broader impacts” in grant-funded projects often lack measurement details (Nadkarni et al., 2019; Watts et al., 2015). Reach metrics do not provide any information about how social media interventions are working to affect change, why they may/may not be working, or the processes occurring to influence change in the audience. Further, they focus on outcomes rather than impact, which is essentially the comparison of the factual situation (what actually happened with the intervention) with the counterfactual (what would have happened without the intervention) (Lewis, 1973). This comparison is challenging to address for cross-disciplinary entities like BIBS (Belcher et al., 2020; Pressey et al., 2021), and in most cases, statistical and experimental approaches to causal inferences aren’t appropriate or feasible.
In this piece, we present a framework that merges theory-based evaluation methods commonly used in the sciences with evaluation techniques developed in strategic communication to provide a process for BIBS to match their goals with appropriate ways of evaluating progress. We will proceed by explaining the approach behind theory-based evaluation and its congruency with social media planning, followed by an explanation of theoretical approaches to social media engagement relevant to BIBS, and conclude by presenting the Science Communication in Social Media Theory of Change (SciSM ToC) evaluation framework, which details appropriate measures for evaluating BIBS’s social media efforts at particular phases of the engagement process. Our purpose is to provide a sustainable method of evaluation for BIBS when focused on social media.
The model proposed in this paper grew out of the authors’ involvement in BIBS, specifically a student-led social media effort to communicate about duck nesting research in the Prairie Pothole Region of North America (Cavanah et al., 2023), sponsored by several U.S. science agencies, the University of North Dakota, and the nonprofit conservation organization Ducks Unlimited, Inc. Students were trained in both biology and science communication, but the team discovered that despite growth in followers on social media and several “viral” posts, it was difficult to gauge if the effort was meeting its true goals of influencing public awareness about the value of wetland and grassland conservation and balanced ecosystems in ways that could lead to public action.
1.1 Social media campaigns within the field of communication
In the communications industry, social media campaigns are planned using systems, such as one referred to by the acronym ROSTIR (Luttrell and Capizzo, 2020). A campaign begins with research around the issue, then the setting of a measurable objective, followed by the development of strategies and tactics that are expected to meet the objective, which are then implemented. In the last stage, all efforts are re-evaluated and assessed. Objectives are ideally chosen because of their ability to support broader organizational goals (Goldberg and Gustafson, 2023). This approach is a path for strategic communication in the general sense (Plowman and Wilson, 2018). However, it is not suited to the special needs and approaches of BIBS.
Unlike more traditional general social media campaigns, BIBS are often engaged in sustained progress toward large social goals (Maag et al., 2018; Posner and Cvitanovic, 2019), and their campaigns can extend for much longer periods of time, meaning they would benefit from more regular evaluation and maintenance, rather than evaluation at the end of a campaign. BIBS are also likely to be working with a smaller, defined audience that would encompass only part of the population needed to effect change (Maag et al., 2018; Posner and Cvitanovic, 2019). Most BIBS operate with the understanding they are part of an informal consortium attempting to inform and persuade the public to move toward research-backed positions and behaviors (e.g., Bornbaum et al., 2015; Rodway et al., 2021). BIBS also often work with complex information that needs to be tailored based on the existing knowledge, attitudes, and behavior of the audience (e.g., Cavanah et al., 2023; Dawson, 2018; Wolsko et al., 2016). Because of these factors, BIBS’s social media efforts would benefit from more structured measurement within the implementation of the campaign, such as in theory-based evaluation.
1.2 Theory-based evaluation
Theory-based evaluation can be used to address the need for a more holistic, context specific, and adaptive approach to interventions and their impacts (Rice et al., 2020). Theory-based evaluation focuses on both the intervention, its effectiveness, and the causal mechanisms and contextual factors hypothesized to drive changes caused by the intervention (e.g., Chen, 2012; Coryn et al., 2011). While most well-established in public health (Lawless et al., 2017), the concept has become applied more frequently throughout the BIBS disciplines (e.g., Belcher et al., 2017; Rice et al., 2020; Smallhorn-West et al., 2022; Zuercher et al., 2023) and aligns well with BIBS’s need to demonstrate progress toward their missions/goals and assess/evaluate how certain actions connect or drive this progress. We use the theory-based evaluation Theory of Change (ToC) framework (Weiss, 1997) to propose a generic pathway for BIBS to evaluate their use of social media in terms of progress toward mission and organizational goals. Rather than focusing on whether interventions promote improved outcomes, the ToC framework examines the assumptions that are made in the strategies developed to achieve program outcomes. This facilitates a greater understanding of why certain activities do or do not promote desired results and outcomes and provides better feedback regarding what types of revisions might be needed. Theory of change frameworks usually include descriptions and assessments of the context, long-term change expected, the process and sequence of change, and assumptions about how the change is expected to happen (Douthwaite et al., 2020).
2 Theoretical foundations
The SciSM ToC assumes that BIBS’ ultimate goals will relate to behaviors or behavior change, either at the individual or societal level. Following Michie et al.’s (2011) Capability, Opportunity, and Motivation Behavior Model (COM-B), which was developed to synthesize multiple behavior-change theories into a form more suitable for practical interventions, we assume that in order for individual behavior changes to occur a person would need to be physically (e.g., age) and psychologically (e.g., knowledge) capable (C) to use opportunities (O) via motivators (M) that are reflective (e.g., goals) or automatic [e.g., emotion: (Barker et al., 2016): Figure 1; Box 4]. Here, opportunity is defined as all factors outside an individual’s sphere of influence that drive or prompt behavior (Michie et al., 2011). Within the SciSM ToC framework these factors interact to influence behavior change, which has a feedback effect on capacity (Figure 1; Box 4, F1). We anticipated social media interventions would need to target one or several aspects of an individual’s capacity for change before being able to observe the behavior changes listed above, and that interventions should be conceptualized through the lens of engagement.
Figure 1. Science Communication in Social Media Theory of Change (SciSM ToC) components with related goals, metrics, and assumptions.
Engagement is “a dynamic multidimensional relational concept featuring psychological and behavioral attributes of connection, interaction, participation, and involvement, designed to achieve or elicit an outcome at individual, organization, or social levels,” (Johnston, 2018: 53). Further, engagement is simultaneously an individual psychological state and an individual-to-social process, including dimensions such as: collectively held beliefs and behaviors that emerge from interaction (orientation); interactions and connections that precede or stem from engagement (experiences); participation in joint activities; collective action; and intention or readiness to act (Johnston, 2018). In short, BIBS’s social media efforts can spur a process that starts with engagement at the individual level, cultivates and increases that engagement, and contributes to the social-level impacts BIBS seek to achieve. Centering engagement as the propelling force in the SciSM ToC mirrors its status as the pre-eminent concern of content creators (Santos et al., 2022). Engagement has also been centered in science communication. For example, Rowe and Frewer’s (2005) definition of engagement is defined by the directions communicators can take: (1) conveying information; (2) asking for public consultation and feedback; and (3) engaging in dialogic, two-way communication with the public (e.g., Habibi and Salim, 2021; Hargittai et al., 2018). Social media is a medium that allows for all three directions to take place simultaneously. Effective use of digital technology like social media can allow for a fuller embrace of true public engagement around science (Brossard and Scheufele, 2013), something that has been more difficult for scientists and science communicators to achieve, despite their beliefs in its value (Besley et al., 2019; McKinnon and Vox, 2014; Fähnrich et al., 2021). In this way, our conceptualization of engagement is most aligned with the participatory or public engagement views of science communication (Metcalfe et al., 2022; Rowe and Frewer, 2005). Our framework is focused on measurement of ‘audience’ outcomes, but the assumption is that effective science communicators will be using the social media best practices of creating and engaging in dialogue with the audience, respecting and engaging with public views, and creating ways for the public to participate in the creation and spread of scientific knowledge (Fähnrich et al., 2021; Habibi et al., 2014).
As with the theoretical underpinnings of the SciSM ToC, we similarly synthesize evaluative metrics from various related fields. We adopt the approach of social media engagement being delineated into cognitive, affective, and behavioral forms from consumer marketing research, as well as established forms of measurement (Santos et al., 2022). We build upon Habibi and Salim (2021), who, following Rowe and Frewer’s (2005) conception of science communication public engagement, create a hierarchy of social media engagement based on metrics to reflect increasing levels of engagement. We merge this with Johnston’s (2018) hierarchy of engagement to create the final measurement approach. The goal of measurement suggestions in our ToC is to provide maximum efficiency. We defer to quicker, less resource-intensive methods such as dashboard analytics where they are appropriate, and reserve intensive, off-platform methods such as online surveys for stages where more complex data are required to evaluate progress toward the goal.
3 The BIBS science communication in social media theory of change evaluation framework (SciSM ToC)
We conceptualized a ToC for a generic BIBS social media campaign, working backwards from the desired intervention’s impact to the inputs required for the intervention to take place (Fig, 1). The elements typically represented in theory-based evaluation models include inputs, activities, outputs, initial outcomes, intermediate outcomes, and long-term outcomes or impacts. Outputs are often described as the immediate result of actions; outcomes are the anticipated changes. Initial outcomes are changes in knowledge, skills, abilities; intermediate outcomes are classified as behavioral changes (Coryn et al., 2011). If the intermediate steps in the model are well specified, they can serve as indicators of progress (Belcher et al., 2024).
In the SciSM ToC, impacts consist of the group- or societal-level behavioral change most often considered the goal of public engagement around scientific concepts, including legislative and policy changes (Figure 1; Box 5). Outcomes cover the strengthening of individual cognition and affect [i.e., emotions; in the case of social media, emotional reactions (Zajonc, 1984)] toward the BIBS’s goals, as well as changes in individual-level behaviors (Figure 1; Box 4). Outputs cover the part of the process where individuals acquire new information from the social media content and begin to experience changes in attitudes and affect around topics (Figure 1; Box 4). Activities include any undertaking that involves the actual creation and distribution of social media content, and inputs include resources BIBS control that can be used during the entire process (Figure 1; Box 1–2). Our framework allows for measurements during each part of the process that can track progress, test assumptions, diagnose issues, and recalibrate to better position for success (Figure 1). Below we describe components of the ToC, related assumptions, recommendations for goal-setting, and approaches for measuring progress. As with Mayne (2023), we identify two different types of assumptions in our ToC: cause-effect and causal-link assumptions. The former are assumptions that there is a connection between an activity or outcome and a result while the latter are assumptions about what conditions need to be in place for a causal link to occur.
3.1 Impact
In developing our ToC we began by identifying a generic desired impact (Figure 1; Box 5), which we described as an organizational- or societal-level change in behavior included in a BIBS organizational goal. This aggregated level of behavioral change would occur only assuming individual online or offline behaviors are widespread enough to affect larger-scale collective action. We base this assumption on the current body of literature regarding collective action in offline behavior and how social media can support online collective action. Social media has a well-known role in community building, norm formation, and the development of shared realities (Greijdanus et al., 2020). For example, despite fears of ‘slacktivism’ or ‘armchair advocacy’ (Büssing et al., 2019; Kubo et al., 2021), some studies have postulated these types of supporters can help stabilize the flow of financial support and lead to new social norms online (Takashina et al., 2023). Further studies have emphasized the importance of context, both political and social, in the effect social media have on spurring collective action (Wolfsfeld et al., 2013), which underscores the utility of an approach like ToC that focuses on contextualizing connections, outcomes, and assumptions.
Measuring progress toward a desired impact like organizational- or societal-scale behavioral change can be challenging. Goals and associated metrics can cross a boundary between individual and group/societal levels because of the ways individual-level engagement creates social-level engagement. This results in the largest breadth of possible appropriate measures in the SciSM ToC framework and the largest potential range of engagement. For instance, a BIBS goal may be to enact a governmental policy change, in which an initial direct measure of success may be the introduction of a bill and a final measurement being the passage of the bill. For these types of goals, most measures are largely dichotomous direct measures of specific events; they either happen or they do not. While social media can be a contributing factor in these results, it is not likely to be the sole factor. Thus, a purist in counterfactual thinking could argue the impact of said social media campaign was low or non-existent. However, the SciSM ToC framework assumes these impacts are the end-result of a cumulative multi-stage engagement process that can activate through social media content. Mass behavior like voting for initiatives and legislation that is either informed by scientific understanding or supports the continuation or extension of science and research in societal life are also direct measurable impacts. Measurement of voting can occur either at the individual level—through off-platform self-reports on online surveys or platform polling features—or at the aggregate through election results. Many other measures of impacts will also involve behaviors, either at the individual or social level, and are best measured through direct means or through off-platform self-reports, like measures of outputs and outcomes.
3.2 Outputs and outcomes
In terms of affecting and measuring change, the boundary between outputs and outcomes in the SciSM ToC is porous (Figure 1; Box 4). Outputs might best be considered the most immediate deliverables from inputs and activities, laying the cognitive groundwork for eventual behavior change, while outcomes finish this process and capture the lower-effort behaviors that lead to the more large-scale changes BIBS target. BIBS might assess progress in this process through dashboard polling features, or off-platform online surveys, interviews, or focus groups. Individual behavior changes influenced by social media content are often more observable and measurable than large-scale social changes, and in some campaigns, changes in individual-level behavior change may be the goal.
Targeted behaviors may include in-person or virtual attendance at events sponsored by BIBS, such as forums, town hall meetings, or fundraising events. Behavior may also be changes to individual routines, such as increased home recycling, purchasing more environmentally friendly consumer goods, or choosing to conserve home energy use during peak energy periods. Particularly for organizations, targeted behaviors can also include individual-level donations, volunteering for specific events like waterway cleanups, or individual involvement with BIBS boards and committees. The most accurate measurement for behaviors will be direct observation, even if these measures will not include data on the motivations or antecedents behind individuals’ participation, including any connection to or influence from social media content. However, including brief off-platform surveys at the point of the targeted behavior that capture participants’ exposure to creators’ content can help shed light on the connection between the social media campaign and the measured behavior. For behaviors that occur in more private settings, self-reports gathered through off-platform online surveys can capture a wider range of behaviors, as well as associated concepts like attitude and affect. For simple measurement of individual behavior, platform polling features can be a quick measurement tool. Polling functions usually involve a multiple-choice or binary-choice interactive feature and can be used to gauge frequencies of single items in account followers.
Alternatively, BIBS might focus exclusively on posting content that influences individuals’ affect or attitude, with the intent to build toward behaviors through the engagement process. Both approaches would, according to the COM-B behavior change model (Michie et al., 2011), have an influence on the motivational component of capacity. None of these are mutually exclusive, of course. One intervention can quite possibly be developed to address all three latent components of capacity and, ultimately, behavioral change (Mayne, 2017). BIBS may want to target just one of these outputs or a combination. Measurement will vary based upon the output targeted, and benchmarks for campaign success should be established before a campaign begins. Here, we will address measurement for output-related goals in three categories: (1) Understanding (C); (2) Attitude change (M - reflective); and (3) affect (M - automatic).
3.2.1 Understanding
At the foundation of all communication goals with the public is some level of understanding. We propose a hierarchical structure to thinking about these types of goals that starts with basic awareness of a BIBS or issue (Mueller-Herbst et al., 2020); moves up to individual mastery of knowledge, particularly of scientific concepts (Laugksch, 2000); with the highest level being understanding, or the ability to incorporate that knowledge into aspects of an individual’s life or societal functioning (Miller, 1983). Measurement is simpler at the lower levels. For awareness, simple polling functions found on most platforms can measure followers’ perception of individuals, organizations, or scientific concepts. These features may also be sufficient for measuring more concrete and discrete knowledge. After distribution of content, asking followers to indicate their recall knowledge using polling can give a sense of whether content is effective in transmitting knowledge.
For goals that involve more extensive knowledge uptake and contextual understanding, direct measurement on social media is difficult. Instead, using social media to recruit followers to participate in off-platform established forms of data collection is more appropriate. The most efficient will be constructing online surveys to measure participant awareness, knowledge, and basic understanding. However, when issues are difficult to contextualize into individual knowledge, the best measurement may come from less generalizable, but more contextually rich, qualitative methods such as in-depth interviewing and online or in-person focus groups. A useful heuristic is considering whether the data sought are exploring something unknown about how the audience is understanding a particular issue or whether the data is meant to confirm that the audience is understanding an issue in a specific way. For instance, in the case of BIBS using social media to increase community involvement in the clean-up of a local waterway following a chemical spill, exploratory methods such as focus groups or interviews can help the BIBS understand how the audience is thinking about responsibility and efficacy in the cleanup. However, if BIBS in the same situation were more interested in establishing that the audience understands the process and timeframe required for the clean-up project, a confirmatory method like a quantitative survey could diagnose specific areas where the audience is lacking understanding. Both methods for collecting data on understanding would then allow for any needed modifications in the content of the BIBS social media campaign.
3.2.2 Attitude change
A central feature of persuasive communication is to create attitude change in the audience (Vogel and Wanke, 2016). Individual communicators can differ on the cognitive mechanisms behind attitude change, but measurement of that change is relatively consistent (Vogel and Wanke, 2016). One aspect that should be addressed early in the process is that measuring attitude change will require measurements at multiple points in time, ideally before and after content is distributed. Preliminary research into social media audiences of science accounts and science communication in general show that these groups often already have high levels of scientific knowledge, as well as positive attitudes toward science-related issues (Cavanah et al., 2023; Metag, 2020; Metag et al., 2018; Metag and Schäfer, 2018). Understanding existing audience attitudes can allow for a more efficient campaign that can skip laying a foundation of knowledge or further influencing attitudes that are already adequate for the desired behavior changes. For very basic measurement, platform polling features can be employed, but should not be relied upon for comprehensive insights into attitude change in the audience. For instance, if a BIBS is interested in changing perceptions on the value of predators in ecosystem maintenance, polling followers prior to and following a campaign featuring content on the value of predators can give an indication of the success of the campaign in changing attitudes. Tracking changes in the applause rate (Mukesh and Rao, 2017) on posts may also give an indirect measurement of attitude change. The applause rate adds all platform metrics that are collected when individuals interact with content in ways meant to show positivity. These vary by platform, but generally collect a measure of positive sentiment, and include “+1” on Reddit, likes on Facebook, favoriting on Instagram, and other similar platform behaviors. The total approval actions for a post are divided by the total number of followers and multiplied by 100 to calculate the applause rate percentage. An increase in this percentage over the course of a campaign could be an indirect measurement of attitude change among followers. However, understanding more complex or more detailed attitude changes will require off-platform measurement. This would be most efficiently done through online survey instruments, which also would allow for more specific and detailed measurement of different facets of attitudes (Vogel and Wanke, 2016).
3.2.3 Affect
Emotional response is viewed as an important precursor to behavior (Baumeister et al., 2007). For goals that involve evoking emotions in the audience, many of the same measurements used for attitude measurement can also be appropriate. This would include platform polling features and tracking the applause rate. For campaigns using Facebook as a platform, measurement of six specific reactions including positivity (“like” and “love”), humor (“haha”), surprise (“wow”), sadness (“sad”), and anger (“angry”) are currently available. Measuring affect would also be the appropriate place for the use of sentiment analysis (Rambocas and Pacheco, 2018), a digital media industry technique that uses artificial intelligence to analyze comments made on social media connected to certain terms to give an overall measurement of positivity versus negativity. It should be noted that it is possible to conduct sentiment analysis through traditional content analysis using human coders, but the time, effort, and resources involved would make this less efficient than machine coding and analysis. Several social media management systems allow users to access automated sentiment analysis, with more advanced systems allowing for the ability to parse sentiment on the sentence level for more nuanced measurement. Many paid services offer sentiment analysis to clients, and some offer free limited access. Affect can also be measured through traditional off-platform self-report methods, such as online surveys, in-depth interviewing, and focus groups.
3.3 Distribution
Many campaigns run under the assumption they will automatically be contacting the intended audiences with their messaging efforts (Figure 1; Box 3). However, the actual distribution of content to end users is a complicated process that occurs through a ‘black box’ of algorithmic choices made for each individual user (Narayanan, 2023). Content selection algorithms are proprietary and often considered the actual ‘business’ of a social media platform, with the quality of the algorithm dictating the overall financial success of a platform (Wu and Zhu, 2024). Based on a platform’s algorithmic parameters, variables such as user profile settings, past content engagement, and overall engagement with specific content will be used to determine which content a user sees in their feed, all with the goal of maximizing user time on the platform (Narayanan, 2023). Because of this, professional content creators and brands focus on boosting engagement with their content (Hollebeek et al., 2014; Influencer Marketing Hub, 2024), thus driving industry to move away from building large follower counts and toward boosting engagement metrics (Hutchinson, 2024). As of the time of this writing, many social media industry experts have noted a shift toward more artificial intelligence selection of content for users, privileging predicted content engagement over user connections on the platform (Narayanan, 2023).
Reach is a common dashboard industry metric that captures the scope of distribution and is often used as the heuristic metric for overall content success in evaluations of science communication on social media (Mukesh and Rao, 2017). However, the SciSM ToC uses reach only as a diagnostic tool for distribution, as reach does not capture effects in the audience. Reach is typically the number of users who had the content appear in their visible feed, but can also involve more complicated breakdowns, especially if a creator pays for more advanced analytic options or chooses to engage in paid content promotion. On most platforms, creators can boost reach to certain demographic categories by paying a fee to the platform. Related to reach is impressions, a dashboard metric that tracks the overall number of times content was displayed, thus capturing repeated exposure to content for some users. More complex measures of distribution may also be useful for BIBS with larger social media presences. The amplification rate is a dashboard measure of how much the original audience for content is spreading that content to others, through shares, retweets, repins, and other platform-specific features. A slight difference in calculation results in the virality rate, a dashboard metric that substitutes impressions for follower counts in the same formula. Finally, for video-based content, a useful dashboard measure that starts to bridge from measuring exposure to measuring understanding is watch time. Most platforms that allow for video-based content include this measure at the individual-post level. At its most basic, watch time will report the average time users spent watching a video before moving on to other content or leaving the app. More detailed measurement systems, such as on YouTube, will allow for second-by-second reports of which points of the video were most watched or where users were most likely to leave the content.
3.4 Activities
We define activities as any task directly involving the creation and/or distribution of social media content (Figure 1; Box 2). Most metrics of success in this stage of the SciSM ToC framework are not about effectiveness of social media messages but instead measuring the actual production and quality of the content. Content production measurements are fairly easy to conduct, often focused on counting the quantity and timing of posts produced and available through platform dashboards. These may also be subdivided into goals for number and timing of posts designed to influence aspects of the audience, such as awareness and attitudes, or covering specific content areas. This also covers efforts to engage in discussion with the public or to inspire discussion within the public. Measurements for activity will mostly involve elements of content analysis—from the basic counting of content within a particular time period to more complex systems of coding characteristics of that content (See Riffe et al., 2023). For discourse, measuring the comment rate can be a useful metric. This is done by dividing the number of comments generated in a series of posts by the number of posts in the series. Failure to meet these goals should prompt a reassessment of the campaign’s initial activities. However, measuring activities cannot be connected to measurement of actual campaign goals. The existence of content or discussion—even quality content and discussion—is not a guarantee of changes in the audience.
3.5 Inputs
As BIBS can often be smaller operations—or even single researchers—this framework was developed to support minimal inputs to achieve attainable goals. Common-ground inputs we anticipated included the money, time, materials, and equipment needed to execute the social media campaign (Figure 1; Box 1). These are all measured directly.
4 Conclusion
BIBS have taken on a larger and more important communication role as information systems have shifted from legacy media to direct communication with audiences. Social media has increasingly become a site of knowledge transfer, relationship building, and public discussion in the digital media age. Social media allow BIBS a breadth of communication activities to a larger potential audience, but also shifts the responsibility for successful management away from media professionals to personnel who may have had limited exposure to communication theory, measurement, or techniques.
The need for BIBS to succeed in their communication endeavors is high. Individuals in democratic societies must possess increasingly sophisticated understandings of scientific concepts in order to vote in the best interests of themselves and their societies (Motta, 2024). Municipalities, states, and countries will determine in the coming years how they will react to climate change, legislate around artificial intelligence, and fund continuing efforts into foundational research, among other science-related topics. Addressing these concerns will require an electorate and elected officials who have an understanding and appreciation for the scientific process, as well as understanding of a plethora of scientific concepts, many just emerging from research. At the same time, trust in all institutions—including those that produce research and disseminate it to the public—has fallen to all-time lows in several countries (OECD, 2024), including the United States (Saad, 2023). These informational and comprehension needs can be filled by BIBS, but only if BIBS can get the appropriate messages to individuals at the appropriate times through the appropriate media.
We present the SciSM ToC evaluation framework as a method to strategically plan and assess BIBS communication methods on social media. By following the framework, BIBS can set goals for communication efforts to maximize impactand move beyond measuring success in terms of reach or the quantity and timing of posts. Even more so, our framework provides guidance on how and when to monitor metrics of success. The SciSM ToC framework is intended to allow BIBS to be as effective with their social media communication as possible without the need for rigorous scholarly support.
To make the SciSM ToC framework effective in a wide variety of practical situations, we made choices to optimize usability and generalizability that came at the expense of other aspects. The framework is generalized to the broad category of BIBS, which means it may need to be tailored to more specific contexts. We do not address the topic of message design in the framework. Communicating about science and science-related topics has benefitted in recent decades from an increasing focus by science communication scholars (Guenther and Joubert, 2017). Their discoveries about best practices in message design and engagement strategies are too numerous, complex, and context specific to include, but are available in the literature. It is recommended that BIBS seek these resources out when planning campaigns. (See Kidd et al., 2019 for a review). We also do not include specific guidance on steps and instrument design for measurements for the same reasons. Inclusion would require more space and could detract from the goal of the framework to be a practical guide for planning and managing social media communication. However, we encourage BIBS to use methodological resources that specify steps and concerns with the measurement methods. Basic, free online resources are included in Table 1. Finally, the SciSM ToC evaluation framework is focused on general concepts and measurements common in social media platforms, but these systems continue to evolve. Future innovations may require revisions.
Table 1. Recommended free online resources for basics of measurement methods used in SciSM ToC.
Despite these limitations, the SciSM ToC evaluation framework is a workable, practically oriented system designed specifically for the communication needs of BIBS on social media. While we adopt measures and methods often employed by scholars conducting rigorous communication research, the precision and exactitude needed for scholarly work is not required to be successful. Instead, the framework is built to be a way for BIBS to quickly, efficiently, and strategically understand the effects of their communication efforts on social media. It was constructed to be adaptable to specific contexts, as well as for technological and platform shifts. As the first comprehensive system for evaluating the effects of BIBS social media communication, it is also constructed to be supplemented and edited as BIBS experience in strategic communication on social media evolves and expands.
Author contributions
SC: Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Supervision, Writing – original draft, Writing – review & editing. KK: Conceptualization, Formal analysis, Investigation, Methodology, Visualization, Writing – original draft, Writing – review & editing.
Funding
The author(s) declare that no financial support was received for the research and/or publication of this article.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The authors declare that no Gen AI was used in the creation of this manuscript.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Adetunji, O. O., and Renoe, S. D. (2017). Assessing broader impacts. MRS Advances 2, 1681–1686. doi: 10.1557/adv.2017.136
Arts, K., Van Der Wal, R., and Adams, W. M. (2015). Digital technology and the conservation of nature. Ambio 44, 661–673. doi: 10.1007/s13280-015-0705-1
Balan, S., and Rege, J. (2017). Mining for Social Media: usage patterns of small businesses. Business Systems Research J. 8, 43–50. doi: 10.1515/bsrj-2017-0004
Barker, F., Atkins, L., and De Lusignan, S. (2016). Applying the COM-B behaviour model and behaviour change wheel to develop an intervention to improve hearing-aid use in adult auditory rehabilitation. Int. J. Audiol. 55, S90–S98. doi: 10.3109/14992027.2015.1120894
Baumeister, R. F., Vohs, K. D., Nathan DeWall, C., and Liqing, Z. (2007). How emotion shapes behavior: feedback, anticipation, and reflection, rather than direct causation. Personal. Soc. Psychol. Rev. 11, 167–203. doi: 10.1177/1088868307301033
Belcher, B. M., Bonaiuti, E., and Thiele, G. (2024). Applying theory of change in research program planning: lessons from CGIAR. Environ. Sci. Pol. 160:103850. doi: 10.1016/j.envsci.2024.103850
Belcher, B. M., Davel, R., and Claus, R. (2020). A refined method for theory-based evaluation of the societal impacts of research. MethodsX 7:100788. doi: 10.1016/j.mex.2020.100788
Belcher, B., Suryadarma, D., and Halimanjaya, A. (2017). Evaluating policy-relevant research: lessons from a series of theory-based outcomes assessments. Palgrave Commun. 3:17017. doi: 10.1057/palcomms.2017.17
Besley, J., O’Hara, K., and Dudo, A.. (2019). Strategic science communication as planned behavior: understanding scientists’ willingness to choose specific tactics. PLoS One 14:e0224039. doi: 10.1371/journal.pone.0224039
Bornbaum, C. C., Kornas, K., Peirson, L., and Rosella, L. C. (2015). Exploring the function and effectiveness of knowledge brokers as facilitators of knowledge translation in health-related settings: a systematic review and thematic analysis. Implement. Sci. 10:162. doi: 10.1186/s13012-015-0351-9
Brossard, D., and Scheufele, D. A. (2013). Science, new media, and the public. Science 339, 40–41. doi: 10.1126/science.1232329
Büssing, A. G., Thielking, A., and Menzel, S. (2019). Can a like save the planet? Comparing antecedents of and correlations between environmental liking on social media, money donation, and volunteering. Front. Psychol. 10:1989. doi: 10.3389/fpsyg.2019.01989
Cavanah, S., Owens, S., Kemink, K., Riley, C., Kim, S., Lee, J., et al. (2023). Birds of feather flock together: a longitudinal study of a social media outreach effort. Biol. Conserv. 281:109999. doi: 10.1016/j.biocon.2023.109999
Chen, H. T. (2012). “Theory-driven evaluation: conceptual framework, application and advancement” in Evaluation von Programmen Und Projekten Für Eine Demokratische Kultur. eds. R. Strobl, O. Lobermeier, and W. Heitmeyer (Springer Fachmedien Wiesbaden: Wiesbaden), 17–40.
CivicPlus. (2022). State of Social Media 2022: Insights from Public Communicators. CivicPlus. Available online at: https://www.civicplus.com/blog/sma/state-of-social-media-stats-and-recommendations/ (Accessed 31 October 2024).
Coryn, C. L. S., Noakes, L. A., Westine, C. D., and Schröter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. Am. J. Eval. 32, 199–226. doi: 10.1177/1098214010389321
Dawson, E. (2018). Reimagining publics and (non) participation: exploring exclusion from science communication through the experiences of low-income, minority ethnic groups. Public Underst. Sci. 27, 772–786. doi: 10.1177/0963662517750072
Douthwaite, B., Ahmad, F., and Shah, G.-M. (2020). Putting theory of change into use in complex settings. Can. J. Program Eval. 35, 35–52. doi: 10.3138/cjpe.43168
Fähnrich, B., Wilkinson, C., Weitkamp, E., Heintz, L., Ridgway, A., and Milani, E. (2021). Rethinking science communication education and training: towards a competence model for science communication. Front. Commun. 6:795198. doi: 10.3389/fcomm.2021.795198
Feser, D. (2023). Innovation intermediaries revised: a systematic literature review on innovation intermediaries’ role for knowledge sharing. Rev. Manag. Sci. 17, 1827–1862. doi: 10.1007/s11846-022-00593-x
Goldberg, M. H., and Gustafson, A. (2023). A framework for understanding the effects of strategic communication campaigns. Int. J. Strateg. Commun. 17, 1–20. doi: 10.1080/1553118X.2022.2137674
Greijdanus, H., De Matos Fernandes, C. A., Turner-Zwinkels, F., Honari, A., Roos, C. A., Rosenbusch, H., et al. (2020). The psychology of online activism and social movements: relations between online and offline collective action. Curr. Opin. Psychol. 35, 49–54. doi: 10.1016/j.copsyc.2020.03.003
Guenther, L., and Joubert, M. (2017). Science communication as a field of research: identifying trends, challenges and gaps by analysing research papers. JCOM 16:A02+. doi: 10.22323/2.16020202
Habibi, M. R., Laroche, M., and Richard, M.-O. (2014). The roles of brand community and community engagement in building brand trust on social media. Comput. Hum. Behav. 37, 152–161. doi: 10.1016/j.chb.2014.04.016
Habibi, S. A., and Salim, L. (2021). Static vs. dynamic methods of delivery for science communication: a critical analysis of user engagement with science on social media. PLoS One 16:e0248507. doi: 10.1371/journal.pone.0248507
Hargittai, E., Füchslin, T., and Schäfer, M. S. (2018). How do young adults engage with science and research on social media? Some preliminary findings and an agenda for future research. Social Media Society 4:205630511879772. doi: 10.1177/2056305118797720
Hollebeek, L. D., Glynn, M. S., and Brodie, R. J. (2014). Consumer brand engagement in social media: conceptualization, scale development and validation. J. Interact. Mark. 28, 149–165. doi: 10.1016/j.intmar.2013.12.002
Hutchinson, A (2024) Instagram chief says follower counts are not a good Indicator of account value. Social Media Today. Available online at: https://www.socialmediatoday.com/news/instagram-chief-follower-counts-are-not-good-indicator/712090/ (Accessed 25 October 2024).
Influencer Marketing Hub. (2024). The state of the Creator economy | definition, growth & market size. Influencer marketing hub. Available online at: https://influencermarketinghub.com/state-of-the-creator-economy/ (Accessed 29 October 2024).
Johnston, K. A. (2018). “Toward a theory of social engagement” in The handbook of communication engagement. ed. M. Taylor (Medford, Massachusetts, USA: John Wiley & Sons, Incorporated), 53–71.
Kemp, S (2024). Digital 2024 October global Statshot report. 23 October. DataReportal. Available online at: https://datareportal.com/reports/digital-2024-october-global-statshot (Accessed 24 October 2024).
Kidd, L. R., Garrard, G. E., Bekessy, S. A., Mills, M., Camilleri, A. R., Fidler, F., et al. (2019). Messaging matters: a systematic review of the conservation messaging literature. Biol. Conserv. 236, 92–99. doi: 10.1016/j.biocon.2019.05.020
Kim, H., Armsworth, P. R., Masuda, Y. J., and Chang, C. H. (2024). US -based and international environmental nongovernmental organizations use social media, but few have large audiences online. Conservation Science Practice 6:e13037. doi: 10.1111/csp2.13037
Kubo, T., Veríssimo, D., Uryu, S., Mieno, T., and MacMillan, D. (2021). What determines the success and failure of environmental crowdfunding? Ambio 50, 1659–1669. doi: 10.1007/s13280-021-01522-0
Laugksch, R. C. (2000). Scientific literacy: a conceptual overview. Sci. Educ. 84, 71–94. doi: 10.1002/(SICI)1098-237X(200001)84:1<71::AID-SCE6>3.0.CO;2-C
Lawless, A., Baum, F., Delany-Crowe, T., MacDougall, C., Williams, C., McDermott, D., et al. (2017). Developing a framework for a program theory-based approach to evaluating policy processes and outcomes: health in all policies in South Australia. Int. J. Health Policy Manag. 7, 510–521. doi: 10.15171/ijhpm.2017.121
Luttrell, R., and Capizzo, L. (2020). Public relations campaigns: An integrated approach. Thousand Oaks, California, USA: SAGE Publications.
Maag, S., Alexander, T. J., Kase, R., and Hoffmann, S. (2018). Indicators for measuring the contributions of individual knowledge brokers. Environ. Sci. Pol. 89, 1–9. doi: 10.1016/j.envsci.2018.06.002
Mayne, J. (2017). Theory of change analysis: building robust theories of change. Can. J. Prog. Eval. 32, 155–173. doi: 10.3138/cjpe.31122
Mayne, J. (2023). Assumptions in theories of change. Eval. Program Plann. 98:102276. doi: 10.1016/j.evalprogplan.2023.102276
McKinnon, M., and Vox, J. (2014). Engagement as a threshold concept for science education and communication. Int. J. Sci. Educ., Part B 5, 297–318. doi: 10.1080/21548455.2014.986770
Metag, J. (2020). What drives science media use? Predictors of media use for information about science and research in digital information environments. Public Underst. Sci. 29, 561–578. doi: 10.1177/0963662520935062
Metag, J., Maier, M., Füchslin, T., Bromme, L., and Schäfer, M. S. (2018). Between active seekers and non-users: segments of science-related media usage in Switzerland and Germany. Environ. Commun. 12, 1077–1094. doi: 10.1080/17524032.2018.1463924
Metag, J., and Schäfer, M. S. (2018). Audience segments in environmental and science communication: recent findings and future perspectives. Environ. Commun. 12, 995–1004. doi: 10.1080/17524032.2018.1521542
Metcalfe, J., Gascoigne, T., Medvecky, F., and Nepote, A. C. (2022). Participatory science communication for transformation. J. Sci. Commun. 21:e21020501. doi: 10.22323/2.21020501
Michie, S., Van Stralen, M. M., and West, R. (2011). The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement. Sci. 6:42. doi: 10.1186/1748-5908-6-42
Motta, M. (2024). Anti-scientific Americans: The prevalence, origins, and political consequences of anti-intellectualism in the US. New York, NY: Oxford University Press.
Mueller-Herbst, J. M., Xenos, M. A., Scheufele, D. A., and Brossard, D. (2020). Saw it on Facebook: the role of social Media in Facilitating Science Issue Awareness. Social Media Society 6:205630512093041. doi: 10.1177/2056305120930412
Mukesh, M., and Rao, A. (2017). “Social media measurement and monitoring” in Contemporary issues in social media marketing. eds. B. Rishi and S. Bandyopadhyay. 1st ed (New York: Routledge), 184–205.
Nadkarni, N. M., Weber, C. Q., Goldman, S. V., Schatz, D. L., Allen, S., and Menlove, R. (2019). Beyond the deficit model: the ambassador approach to public engagement. Bioscience 69, 305–313. doi: 10.1093/biosci/biz018
Narayanan, A. (2023). “Understanding social media recommendation algorithms” in Knight first amendment institute: Algorithmic amplification and society (New York, NY: Knight First Amendment Institute, Columbia University. Available at: https://knightcolumbia.org/content/understanding-social-media-recommendation-algorithms).
Neal, J. W., Neal, Z. P., and Brutzman, B. (2022). Defining brokers, intermediaries, and boundary spanners: a systematic review. Evidence Policy 18, 7–24. doi: 10.1332/174426420X16083745764324
Nonprofit Tech for Good. (2019). Global NGO technology report 2019. Nonprofit Tech for Good. Available online at: https://www.funraise.org/techreport (Accessed 31 October 2024).
OECD. (2024). OECD survey on drivers of Trust in Public Institutions – 2024 results: Building Trust in a Complex Policy Environment. Paris: OECD Publishing. doi: 10.1787/9a20554b-en
Plowman, K. D., and Wilson, C. (2018). Strategy and tactics in strategic communication: examining their intersection with social media use. Int. J. Strateg. Commun. 12, 125–144. doi: 10.1080/1553118X.2018.1428979
Posner, S. M., and Cvitanovic, C. (2019). Evaluating the impacts of boundary-spanning activities at the interface of environmental science and policy: a review of progress and future research needs. Environ. Sci. Pol. 92, 141–151. doi: 10.1016/j.envsci.2018.11.006
Pressey, R. L., Visconti, P., McKinnon, M. C., Gurney, G. G., Barnes, M. D., Glew, L., et al. (2021). The mismeasure of conservation. Trends Ecol. Evol. 36, 808–821. doi: 10.1016/j.tree.2021.06.008
Rambocas, M., and Pacheco, B. G. (2018). Online sentiment analysis in marketing research: a review. J. Res. Interact. Mark. 12, 146–163. doi: 10.1108/JRIM-05-2017-0030
Rice, W. S., Sowman, M. R., and Bavinck, M. (2020). Using theory of change to improve post-2020 conservation: a proposed framework and recommendations for use. Conservation Science Practice 2:e301. doi: 10.1111/csp2.301
Riffe, D., Lacy, S., Watson, B. R., and Lovejoy, J. (2023). Analyzing media messages: Using quantitative content analysis in research. 5th Edn. New York: Routledge.
Rodway, J., MacGregor, S., Daly, A., Liou, Y. H., Yonezawa, S., and Pollock, M. (2021). A network case of knowledge brokering. J. Professional Capital Community 6, 148–163. doi: 10.1108/JPCC-11-2020-0089
Rowe, G., and Frewer, L. J. (2005). A typology of public engagement mechanisms. Sci. Technol. Hum. Values 30, 251–290. doi: 10.1177/0162243904271724
Saad, L (2023) Historically low faith in U.S. institutions continues. Gallup news. Gallup Available online at: https://news.gallup.com/poll/508169/historically-low-faith-institutions-continues.aspx (Accessed 29 October 2024).
Santos, Z. R., Cheung, C. M. K., Coelho, P. S., and Rita, P. (2022). Consumer engagement in social media brand communities: a literature review. Int. J. Inf. Manag. 63:102457. doi: 10.1016/j.ijinfomgt.2021.102457
Simis, M. J., Madden, H., Cacciatore, M. A., and Yeo, S. K. (2016). The lure of rationality: why does the deficit model persist in science communication? Public Underst. Sci. 25, 400–414. doi: 10.1177/0963662516629749
Smallhorn-West, P., Cohen, P. J., Phillips, M., Jupiter, S. D., Govan, H., and Pressey, R. L. (2022). Linking small-scale fisheries co-management to U.N. Sustainable development goals. Conserv. Biol. 36:e13977. doi: 10.1111/cobi.13977
Smedlund, A. (2006). The roles of intermediaries in a regional knowledge system. J. Intellect. Cap. 7, 204–220. doi: 10.1108/14691930610661863
Takashina, N., Cheung, H., and Miyazawa, M. (2023). Spread the word: sharing information on social media can stabilize conservation funding and improve ecological outcomes. Conserv. Sci. Pract. 5:e12857. doi: 10.1111/csp2.12857
Van Enst, W., Driessen, P., and Runhaar, H. (2017). Working at the boundary: an empirical study into the goals and strategies of knowledge brokers in the field of environmental governance in the Netherlands. Sustain. For. 9:1962. doi: 10.3390/su9111962
Watts, S. M., George, M. D., and Levey, D. J. (2015). Achieving broader impacts in the National Science Foundation, Division of Environmental Biology. Bioscience 65, 397–407. doi: 10.1093/biosci/biv006
Weiss, C. (1997). “Theory-based evaluation: past, present, and future” in Progress and future directions in Evalution: Perspectives on theory, practice, and methods. New directions for evaluation 76, vol. 1997 (San Francisco: Jossey-Bass), 41–55.
Wilkinson, C., and Weitkamp, E. (2013). A case study in serendipity: environmental researchers use of traditional and social media for dissemination. PLoS One 8:e84339. doi: 10.1371/journal.pone.0084339
Wolfsfeld, G., Segev, E., and Sheafer, T. (2013). Social media and the Arab spring: politics comes first. Int. J. Press Politics 18, 115–137. doi: 10.1177/1940161212471716
Wolsko, C., Ariceaga, H., and Seiden, J. (2016). Red, white, and blue enough to be green: effects of moral framing on climate change attitudes and conservation behaviors. J. Exp. Soc. Psychol. 65, 7–19. doi: 10.1016/j.jesp.2016.02.005
Wu, K, and Zhu, J (2024) Exclusive: ByteDance prefers TikTok shutdown in US if legal options fail, sources say. Reuters. Available online at: https://www.reuters.com/technology/bytedance-prefers-tiktok-shutdown-us-if-legal-options-fail-sources-say-2024-04-25/ (Accessed 29 October 2024).
Zajonc, R. B. (1984). On the primacy of affect. Am. Psychol. 39, 117–123. doi: 10.1037/0003-066X.39.2.117
Keywords: audience engagement, brokers intermediaries and boundary spanners, science communication, social media evaluation, strategic communication, theory of change
Citation: Cavanah SB and Kemink K (2025) The science communication in social media theory of change evaluation framework: a system of establishing goals and assessing efficacy for knowledge brokers, intermediaries and boundary spanners. Front. Commun. 10:1534363. doi: 10.3389/fcomm.2025.1534363
Edited by:
Anders Hansen, University of Leicester, United KingdomReviewed by:
Kevin M. Folta, University of Florida, United StatesNicole Kelp, Colorado State University, United States
Copyright © 2025 Cavanah and Kemink. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Sarah B. Cavanah, c2FyYWhjYXZhbmFoQGt1LmVkdQ==
†ORCID: Sarah B. Cavanah, http://orcid.org/0000-0001-7512-4866
Kaylan Kemink, http://orcid.org/0000-0002-1404-0690