Framing Science: How Opioid Research Is Presented in Online News Media

Matthias, Lisa; Fleerackers, Alice; Alperin, Juan Pablo

doi:10.3389/fcomm.2020.00064

ORIGINAL RESEARCH article

Front. Commun., 30 September 2020
Sec. Science and Environmental Communication
Volume 5 - 2020 | https://doi.org/10.3389/fcomm.2020.00064

Framing Science: How Opioid Research Is Presented in Online News Media

Lisa Matthias^1,2^*

Alice Fleerackers^2,3

Juan Pablo Alperin^2,4^*

¹Political Science Department, John F. Kennedy Institute, Freie Universität Berlin, Berlin, Germany
²Scholarly Communications Lab, Simon Fraser University, Vancouver, BC, Canada
³Interdisciplinary Studies, Simon Fraser University, Vancouver, BC, Canada
⁴School of Publishing, Simon Fraser University, Vancouver, BC, Canada

Popular news media play an instrumental role in shaping public perception of issues like the opioid crisis. Using a detailed coding instrument, we analyzed how opioid-related scholarly publications were covered in 149 news stories published by nine major US and Canadian online news outlets at the height of the crisis. We find that a small proportion of available studies receive coverage, mostly within issue-focused rather than science communication news stories. While most studies are framed as established “facts,” stories rarely provide sufficient information for news consumers to critically evaluate the validity of the research. Potential implications for science communication and public perception of the opioid crisis are discussed.

Introduction

“This is not too bad,” Richard Sackler responded when he learned about 59 fatal overdoses his company's drug, OxyContin (Oxycodone, 2019), a prescription opioid, had caused in early 2001. “It could have been far worse” (James, 2019, p. 98).

Fast forward to today, when—after almost two decades of deceptive marketing of opioids and economic and social upheaval (Dasgupta et al., 2018; James, 2019)—North America's overdose problem has, indeed, become “far worse” (James, 2019, p. 98). Since Sackler's original statement, opioid-related overdoses have increased dramatically, contributing to 47,600 deaths in the United States (US) in 2017 alone (14.9 per 100,000 population; NIDA, 2019). The situation is just as dire in Canada, which saw a corresponding 4,100 fatal overdoses that same year (11.2 per 100,000 population; Special Advisory Committee on the Epidemic of Opioid Overdoses, 2019). But while both nations refer to the opioid crisis as a public health emergency (Ending America's Opioid Crisis, 2019; Health Canada, 2019), concrete actions toward harm reduction and treatment options have been limited (Casey, 2016; CFMS, 2018; GAO, 2018; Warren, 2018; Fischer et al., 2019; Frank and Haffajee, 2019). Moreover, public opinion remains divided about important aspects of the opioid crisis, with opinion leaders reporting conflicting views about the severity of the situation, how best to respond to it, and who is responsible (Nanos Research, 2017; American Psychiatric Association, 2018; Angus Reid Institute, 2018; Blendon and Benson, 2018; Manchikanti et al., 2018). Given the strong link between public opinion and policy adoption (Page and Shapiro, 1983; Cook, 2002; Burstein, 2003), understanding these divides, why they persist, and how to manage them have literally become questions of life and death. As an influential guide of public health discourse, the news media are ideally positioned to aid and shape public understanding and perception of the scientific evidence that can contribute to policies designed for addressing the opioid epidemic (Wallack et al., 1993; Maibach and Parrott, 1995; Seale, 2003). As such, this study investigates how American and Canadian online news media covered scholarly publications on opioid-related disorders in 2017 and 2018, at the height of the opioid crisis (NIDA, 2019; Special Advisory Committee on the Epidemic of Opioid Overdoses, 2019).

Literature Review

Previous studies have shown that heightened media attention to an issue can result in a direct rise of public concern—the so-called agenda setting function of the media (McCombs, 2004; Pew Research Center, 2014; Iyengar, 2015). By choosing to focus on certain issues and not others, news journalists play key roles in “setting the agenda” for what publics think about and which topics they deem important (McCombs, 2002; Entman, 2010). But the news media do more than influence whether publics think about certain issues; they also play a key role in shaping how they think about them. By presenting, or framing, news in particular ways—that is, “selecting and highlighting some facets of events or issues, and making connections among them” —journalists can “promote a particular interpretation, evaluation, and/or solution” to the issues they cover (Entman, 1993, p. 52). Conceptualized another way, frames act as “central organizing idea[s]” that journalists use to structure their stories (Gamson and Modigliani, 1989, p. 3), and that audiences can rely on to make sense of the information presented to them. By raising the salience of select ideas, framing activates certain interpretive schemas, or ways of thinking, in audience members' minds, encouraging them to act, think, or feel a particular way (Entman, 2010).

Especially within the context of science communication, the effects of such frames can be profound. Previous studies across a range of science topics have shown that different frames can influence public interest in science issues, perceptions about individuals and diseases (e.g., compassion, stigma), preferences for science-informed policies, as well as personal and political behaviors (Nisbet, 2009; Gallagher and Updegraff, 2012; Gollust et al., 2013; Riles et al., 2015). Some scholars have gone so far as to argue that, in many science debates, power has been determined by one group or individual's ability to define the nature of the problem and to suggest potential solutions—in other words, to frame the issue at hand (Nisbet et al., 2003; Nisbet and Huge, 2006).

The news media are a top source of science news for a majority of Americans (Pew Research Center, 2017), and can thus play a key role in framing scholarly publications for public consumption. Journalists can amplify frames used by scientists, “hype” novel findings, raise doubts concerning the validity of scientific findings or the morality of their implications, or reframe the debate scholarly publications are placed in (Lawrence et al., 2000; Caulfield, 2004; Nerlich and Halliday, 2007; O'Keefe et al., 2015; Stecula and Merkley, 2019). Such framing choices may be particularly important when it comes to health topics, as publics often rely on media coverage for information about diseases, medical screenings and diagnostic tests, illness prevention, and more (Stryker et al., 2008; Kealey and Berkman, 2010; Wilson et al., 2012). Furthermore, the nature and quantity of health news coverage can influence individuals' illness prevention and detection behaviors (Yanovitzky and Blitz, 2000; Chapman et al., 2005; Shen et al., 2015), use of healthcare services (Grilli et al., 2002), and treatment preferences (Chen and Siu, 2001; Shaffer et al., 2018), as well as policy-making processes (Lawrence et al., 2000).

Together, these findings suggest that the news media's framing of scholarly publications related to the opioid epidemic may have important implications for society, both for individuals and policy makers. This had been the case with regards to the news media's framing of the opioid epidemic more broadly, which, much like with previous drug epidemics related to crack cocaine and methamphetamine, saw people with opioid use disorders stigmatized in media reports in ways that are believed to have hindered policy implementation and harm reduction efforts (Lawrence et al., 2000; McGinty et al., 2018, 2019). Yet, contrary to media reporting on prior epidemics, coverage of the opioid epidemic has increasingly shifted toward a public health lens, which positions opioid use and related disorders as treatable health conditions, and away from the more traditional emphasis on law enforcement, which frames opioid misuse in terms of criminal activities (McGinty et al., 2016; Willis and Painter, 2019; Shachar et al., 2020). As both these frames promote two distinct intervention strategies—law enforcement solutions focusing on arrests and prosecutions and public health responses aimed at expanding treatment and harm reduction—these narratives matter. A recent study by Shachar et al. (2020) suggests that these differences in framing stem from the different perceptions of which demographic groups are most affected by the epidemic and the legal status of the substance involved; specifically, opioids, with the exception of heroin, are not illegal drugs but are available via medical prescription, and opioid users are perceived as being predominantly white (see also Netherland and Hansen, 2016).

However, while these studies provide important insights into the news media's framing of the opioid epidemic in general, they do not examine the specific ways in which journalists present scholarly publications related to the prevention, harm reduction, and treatment of opioid-related disorders—a key consideration given that publics often defer to the testimony of scientists and other experts when dealing with unfamiliar issues (Nisbet et al., 2003; Maier et al., 2014; Boutron et al., 2019). Notably, the opioid epidemic is not unique in this sense; while there has been a wealth of research exploring how health issues such as cancer, vaccines, and stem cells are framed in the news media (Nisbet et al., 2003; Nerlich and Halliday, 2007; Barry et al., 2011; Jung Oh et al., 2012; Hove et al., 2015; O'Keefe et al., 2015; Dan and Raupp, 2018), relatively few studies have investigated how scholarly research about those issues is framed (see Zehr, 2000; Antilla, 2005; Ruhrmann et al., 2015; Guenther et al., 2019 for exceptions). The limited research that has suggests that scholarly publications are often framed as being valid and trustworthy in news coverage but can also be framed as uncertain or controversial—especially when they relate to politicized topics (e.g., climate change, molecular medicine). Yet, no previous work has examined the media framing of scholarly publications that relate to the opioid epidemic—an important oversight, given that uncertainty and controversy frames can shift public opinions, perceptions of credibility, and behavioral intentions. For example, research on the portrayal of cancer research found that audiences perceived scientists and journalists as more trustworthy when the research was “hedged” by reporting study limitations (Jensen, 2008). Related research has found that framing scientific findings as controversial can also reduce public beliefs about the certainty of climate change and intentions to aid farmers and workers affected by its impacts (Corbett and Durfee, 2004; Gustafson and Rice, 2019), and undermine perceptions of the safety of routine childhood vaccines (Dixon and Clarke, 2013). Importantly, such framing effects appear to be especially potent when the science is communicated with a lack of context (Corbett and Durfee, 2004).

As such, understanding the ways in which scholarly publications about the opioid epidemic are presented to the public is critical to the larger understanding of how the news media are shaping public opinion on the subject. Importantly, doing so requires considering news media stories beyond dedicated health or science coverage. While such dedicated coverage offers one way in which publics can learn about such publications, news stories about broader topics may also play an important role. A survey by the Pew Research Center (2017) found that nearly half of US adults (49%) are “uninterested science news consumers” who access science news infrequently and usually by chance, and that only 16% of Americans intentionally consume science news a few times a week. Similarly, while specialized science publications are one way in which publics learn about science news, more generalized outlets appear to be a more important source for many readers (ibid.). As such, news stories focused on broader issues, such as policy development, economics, or personal experiences, may be better aligned with news audiences' existing habits and preferences than specialized science journalism coverage. Such issue-focused stories may thus be one effective media strategy “for ‘going broad’ with science-related content,” a recent recommendation for science communicators wishing to “generat[e] attention and interest among non-elite audiences” (Nisbet and Scheufele, 2009, p. 1775). However, to our knowledge, they are not often examined in media framing studies of science and health topics. The current study seeks to address this knowledge gap and extend the currently limited knowledge of how popular news media portray scholarly publications by examining the framing of an important but understudied topic: the prevention, harm reduction, and treatment of opioid-related disorders. To do so, it examines the framing of scholarly publications in both issue-focused and science communication stories, relying on three uncertainty-related frames that have been previously identified in studies of scientific news coverage, each of which is outlined below.

Valid Science Frame

Antilla describes the valid science frame as “not discuss[ing] skepticism of the research” (2005, p. 344). These news stories do not convey any opposing views or doubts toward the research findings or topic, such that the authors of the study in question act as the “primary definers” (ibid.) of the issue at hand. Ruhrmann et al. characterize the valid science frame by “its above-average frequency of depicting scientific certainty” (2015, p. 10). That is, scientific information is communicated using a neutral tone, without controversy, discussion, or evaluation of risks or benefits. Finally, Guenther et al. propose that framing science as valid and credible is characterized by “certain single result(s), sufficient data, (strong) methodological quality, results pointing in the same direction, successfully replicated findings, […and/or] highly experienced researcher(s)” (2019, p. 50). As part of their analysis, Guenther and colleagues also recorded relevant criteria to assess scientific evidence, such as information about the authors, publication venue, study design, sample sizes, or funding sources. The valid science frame was found to be prevalent in news coverage of an array of science topics, from climate change to molecular medicine (Antilla, 2005; Ruhrmann et al., 2015) and was most commonly found in news stories published within dedicated science sections (i.e., as opposed to general interest stories; Guenther et al., 2019). Prevalence rates vary by topic and study, but findings suggest that the valid science frame appears in about 25 to 70 percent of news stories covering science (Antilla, 2005; Ruhrmann et al., 2015; Guenther et al., 2019).

Uncertain Science Frame

The uncertain science frame tends to emphasize “insufficient data or some uncertainties associated with the research” (Zehr, 2000, p. 92). Likewise, Guenther et al. describe the uncertain frame as privileging “preliminary data, knowledge gap[s], [and/or poor] methodological quality” (2019, p. 50). In their study of stem cell media coverage, Nisbet et al. (2003) blend elements of the uncertain and controversial science frames into a single “Uncertain” frame. This frame includes a “focus on scientific uncertainty over efficacy or outcomes of stem cell–related research and applications,” but also incorporates elements that are not specifically related to the framing of the research itself, such as uncertainty surrounding the viability of stem cell lines and disputes about the benefits of using embryo stem cells. In the context of climate change, meanwhile, Stecula and Merkley (2019) operationalized the uncertainty frame as shedding doubt on the veracity of the Intergovernmental Panel on Climate Change (IPCC) climate consensus, by referring, for example, to a lack of understanding about the issue at hand. Importantly, neither Nisbet et al. (2003) nor Stecula and Merkley (2019) investigated the framing of scholarly publications specifically, focusing instead on the framing of a scientific topic more generally. Despite this limitation, aspects of both studies can be used to understand the uncertainty frame in more detail. This frame appears to be slightly less prevalent than the valid science frame, occurring in between 1 to 25 percent of news stories (Nisbet et al., 2003; Antilla, 2005; Stecula and Merkley, 2019).

Controversial Science Frame

Finally, previous research has defined the controversial science frame as one that emphasizes conflicts and disagreements. Zehr posits that this frame often relies on a “practice of interjecting and emphasizing controversy or disagreement among scientists” (2000, p. 90). Antilla (2005) expands on this definition, arguing that controversy can also be mediated by highlighting disagreements between scientists and those outside of academia, such as industry leaders. She remarks that, in the context of climate change, news stories employing a controversial frame often “included rhetoric from climate skeptics with known fossil fuel industry ties” (Antilla, 2005, p. 347) that called the research findings into question. Ruhrmann et al. describe this frame as presenting “conflicting scientific results, which journalists…discuss with above-average frequency” (2015, p. 10). Like Nisbet et al. (2003), Guenther et al. do not explicitly identify a controversial science frame, instead characterizing stories that highlight “contrasting findings of research, contrasting interpretation of [the] same data-set, [or] conflicting viewpoints of researchers” (2019, p. 50) as uncertain. While not every study explicitly reported the prevalence of the controversial science frame, it appears to be present in between ~16 and 27 percent of news stories (Antilla, 2005; Ruhrmann et al., 2015).

Although uncertain and controversial science frames are sometimes treated as one and the same, research suggests they affect audiences differently; specifically, while reading a news story that frames scientific findings as uncertain appears to have minimal negative effects (and may even improve public perceptions about scientists and medical professionals), encountering a controversial science frame has been found to decrease news readers' beliefs in scientific claims and judgments about the credibility of the message source (Jensen et al., 2011; Gustafson and Rice, 2019). As such, we treat the two frames as distinct for the purposes of this study.

Specifically, we build on previous work examining the portrayal of scientific research in the news to explore the following three research questions:

RQ1. How often was research on opioid-related disorders mentioned in online news stories published by popular American and Canadian news media between 2017 and 2018?

RQ2. What contextual information do journalists provide when referencing scholarly publications about opioid-related disorders?

RQ3. How are scholarly publications about opioid-related disorders framed?

Methodology

Sample Selection and Collection

To investigate the framing of scientific research studies about opioid-related disorders in particular, as opposed to the opioid crisis more broadly, we needed to identify news articles (hereinafter “stories”) that mentioned or linked to relevant scholarly publications (hereinafter “studies”; see Figure 1, below). To do this, we first identified relevant scholarly publications as relating to the prevention, harm reduction, or treatment of opioid-related disorders by querying PubMed, a database for biomedical and life sciences research. PubMed indexes publications according to Medical Subject Headings, or “MeSH terms,” which provide a consistent way of cataloging and searching the database when authors use different terms for the same concept or topic (e.g., opioid addiction and heroin addiction). The MeSH term “opioid-related disorders” (U.S. National Library of Medicine), includes abuse, addiction, and dependence on any kind of opioid or opiate (e.g., heroin, morphine, codeine), from which we selected the six subtopics that aligned most closely with prevention, harm reduction, and treatment efforts: diagnosis, drug therapy, prevention and control, rehabilitation, therapy, and epidemiology. These subtopics became the basis of our final query, to which we added the MeSH term “needle-exchange programs” (found under the sub-heading “Preventative Health Services”) due to their importance as a harm reduction strategy that is not covered by any of the other search terms (Fernandes et al., 2017). This search yielded 12,166 relevant research studies.

FIGURE 1

Figure 1. Sample size and conceptualization of studies, references, and stories. In this study, we define “studies” as scholarly publications referenced in news stories; “stories” as news articles that implicitly or explicitly refer to research studies; “references” as those parts of the news story that relate to the research study.

Next, we searched the Altmetric Explorer for news stories mentioning or linking to any of the scientific studies identified by our search (May 20, 2019). The Altmetric Explorer is a tool built for exploring the Altmetric database, which tracks online activity and news mentions of scientific research outputs by identifying direct links to publications (i.e., a hyperlink or a publication identifier, such as a DOI) and by mining news texts for details such as author names, journal titles, and study timeframes (Altmetric.com, 2015). Since the Altmetric Explorer allows for PubMed queries, we were able to use the same search terms as above to find news stories that included research references. We define research references as those parts of a news story that clearly relate to a relevant, identifiable scholarly publication (e.g., a hyperlink to a publication and/or sentences, paragraphs, quotes discussing its results or implications). We limited our query to the most widely used news sources by Canadians and Americans (Alexa Internet, n.d.)—BBC, Breitbart, CBC, CNN, Fox News, Global News, the New York Times, Vice, and the Washington Post—and to news stories published in 2017 and 2018. Following this approach, we identified 212 news stories that included 310 references to 175 unique studies about opioid-related disorders (a story could include more than one reference to research). We collected and read all 212 news stories to verify each research reference manually. During this process, we removed 86 research references (27.7%): 35 duplicates (references identified by Altmetric more than once); 34 references in stories not written in English; 11 references in stories that were no longer available online; and 6 mismatched references (i.e., the reference in the news story was not identical to the one identified by Altmetric). Our final dataset consisted of 149 news stories that included 223 mentions of 164 unique studies.

To understand what kind of studies received media attention, we recorded basic background information about each study in our sample using data provided by Altmetric. Specifically, for each reference, we noted the study's digital object identifier (DOI) as well as the publication venue and year. Based on the DOIs, we also determined whether each study was freely available on the publisher website, in an institutional repository, or not publicly available. This was done using Unpaywall, a database containing the open access status of publications (Piwowar et al., 2018).

For further context, we queried the Media Cloud Explorer, an open-source platform for media analysis (MIT Center for Civic Media and Berkman Klein Center for Internet and Society)¹, to find out how the number of news stories mentioning scholarly publications compared to the overall volume of news coverage on the opioid crisis and opioid-related disorders during the same time frame. We set the search parameters to the same nine news sources and timeframe as before, using the following search terms: “(opioid OR heroin OR morphine) AND (addiction OR abuse OR epidemic OR crisis OR dependence OR overdose OR overdoses).” This search resulted in 9,565 news stories from which we removed 965 duplicate news stories, leaving a total of 8,600 stories.

Content Analysis

Since we were interested in the framing of scholarly publications in particular, we manually extracted only those parts of the news story that directly referred or linked to one of the studies in our sample. Following a directed content analysis approach (Hsieh and Shannon, 2005), we coded these extracts (hereinafter “references”) to identify which frames journalists used most often when covering research on opioid-related disorders. We also coded whether the news stories included any bibliographic information about the study, such as author name(s), affiliation(s), or publication venue as well as any details about the study design, such as sample size, timeframe, or other relevant details, such as the type of data (e.g., face-to-face interviews, nationally representative survey data), sample specifications (e.g., gender, age), geographical location (e.g., Rhode Island), or details about the analytical methods (e.g., cost-benefit analysis). Zooming out to examine each reference in the broader context of the news story, we determined the thematic focus of the news story—that is, whether it primarily reported on research findings or on some broader issue—as well as whether the story had first been published by the news source itself or by some other outlet (i.e., aggregate content). Finally, we noted whether each story included a link to the research study it mentioned.

To perform the frame analysis, we applied a deductive approach that relied on a detailed coding instrument (available in the Supplementary Materials) which was developed by drawing from previous research investigating valid, uncertain, and controversial science news frames (Zehr, 2000; Antilla, 2005; Ruhrmann et al., 2015; Guenther et al., 2019) and refined for the topic at hand. While deductive framing analysis may overlook certain frames that have not been identified in previous research (Matthes and Kohring, 2008), what the method lacks in flexibility it makes up for in terms of reliability, validity, and efficiency; relative to inductive approaches, deductive framing analysis tends to be easier to replicate and validate, can be applied to larger samples, and can more easily detect differences between media outlets (Semetko and Valkenburg, 2000; Van Gorp, 2010).

The three frames presented earlier, and included in the codebook, were operationalized as follows:

Valid Science

We defined the valid science frame as presenting scientific research as sound and trustworthy. Specifically, we proposed that this frame omits any potential scientific uncertainty and controversy regarding the study, such as poor methodological quality or contrasting research findings. We operationalized this frame using two indicators: portrayal of scholarly publications as trusted facts and an emphasis on positive scientific credibility. When journalists present research findings as trusted or established facts, they often do so without further evaluation or discussion (cf. Antilla, 2005; Ruhrmann et al., 2015). At the extreme, this can manifest as a news story that does not directly refer to a study at all, but merely cites it in the form of a hyperlink. In doing so, the findings are represented as established “truths,” rather than as outcomes of a scientific study with limitations requiring deeper consideration. Moreover, journalists can allude to positive scientific credibility by mentioning researcher affiliations or journal titles (cf. Guenther et al., 2019).

Uncertain Science

The uncertain science frame, on the other hand, presents scientific results as tentative or requiring additional research (cf. Zehr, 2000; Nisbet et al., 2003; Guenther et al., 2019). It cautions against viewing the research findings as trusted facts, often by raising doubts about the scientific credibility of the research results or the researchers themselves, emphasizing conflicts of interest (e.g., industry funding) or methodological shortcomings, such as small sample sizes (Zehr, 2000; Guenther et al., 2019; Stecula and Merkley, 2019). We operationalized this frame through two indicators: emphasis on preliminary findings and questioning of scientific credibility.

Controversial Science

Finally, the controversial science frame was conceptualized as injecting or emphasizing controversy and disagreement surrounding the reported study, either from within or beyond the academic sphere. We operationalized this frame with two indicators: contrasting research findings (Zehr, 2000; Ruhrmann et al., 2015; cf. Guenther et al., 2019) or conflicting viewpoints about the topic at hand (cf. Antilla, 2005). Contrasting research findings highlighted conflicts from within the scientific community, such as when research findings departed from those of other studies. Conflicting viewpoints, on the other hand, originated outside of the academy, such as when scientific consensus did not translate into political consensus or when a personal anecdote presented in a news story differed from the research findings.

Using Nvivo 12 Pro, each research reference was coded for the presence of each of the six indicators by one of the authors (LM; see Table 1, below), as per Syed and Nelson (2015). If indicators of multiple frames were present, we determined the dominant frame as follows: controversy superseded uncertainty, and both superseded validity (cf. Antilla, 2005). That is, stories could only be deemed uncertain if there was no indicator of controversy and would be deemed valid only if the research reference did not highlight any uncertainties or controversies. We chose this hierarchy because previous research suggests these frames affect audiences differently. Kortenkamp and Basten (2015) show that news readers rated discredited scientists (e.g., those portrayed as having a conflict of interest) as less credible by readers than non-discredited scientists. Moreover, presenting scientific certainty, or validity, can increase the audience's interest in science, whereas uncertainty does not (Retzbach et al., 2013). While an emphasis on uncertainty seems to have minimal effects, controversy can negatively affect health behaviors, beliefs in scientific consensus and in scientific claims more generally, and judgments about the credibility of the message source (Jensen et al., 2011; Kienhues et al., 2011; Dixon and Clarke, 2013; Gustafson and Rice, 2019).

TABLE 1

Table 1. Frame operationalization.

While coding for the frames of each reference, we also noted whether the news story included any bibliographic information, such as author name(s), affiliation(s), or publication venue, as well as information about the study design or process, such sample size, timeframe, or any other details pertaining to the research (e.g., type of data, methods, sample specifications).

We assessed inter-coder reliability using a second coder (AF) and random subsamples at various stages of the coding process (Bennett et al., 2008, p. 206). For these subsamples, we first itemized all references in our sample (1–226) and then used a random number generator to select references (Urbaniak and Plous, n.d.). After developing the initial codebook, both coders coded a subsample of 15 references; we discussed disagreements and revised the codebook to better reflect nuances between individual codes and to separate them more clearly. We repeated this process once more before coding the full sample. Table 2 presents the results of the final inter-coder reliability test, which consisted of 45 research references (20% of the full sample) that were coded by both coders independently. We measured inter-coder agreement using Krippendorff's Alpha, which allows for any number of coders or variables and corrects for small sample sizes (Krippendorff, 2004). Our results were above the generally accepted threshold of ·7 (Lombard et al., 2002), with most codes exceeding ·85. Agreement was lowest for “positive credibility” and “conflicting viewpoints”; after reviewing the results for these two codes, we realized the discrepancies did not come from actual disagreement between coders but from a lack of clarity in the codebook. The codebook was revised accordingly and used for the rest of the analysis.

TABLE 2

Table 2. Inter-coder reliability.

After determining which frames journalists used to present the individual references, we coded for certain characteristics of the news stories. First, we categorized the news stories in our sample according to their broad thematic focus—science communication or another issue. Science communication stories primarily focus on reporting scientific research whereas issue-focused stories place the primary focus on a larger issue, in which a scholarly publication is referenced, but not the main focus of the story. Prioritizing the thematic focus of the news story over style, we classified “health communication” stories (Schiavo, 2011) accordingly as either science communication or issue-focused and categorized editorial and opinion pieces as issue-focused stories (Bader, 1990). For this part of the analysis, we used only the headline and the first paragraph, as this is where journalists typically present the main topic of a news story (Pöttker, 2003; Burscher et al., 2016). Second, we coded whether the news story was an original story—that is, it had been first published by the news outlet itself—or a republished story that had been previously published by another source, such as a press agency. For this part of the analysis, we checked the beginning and header as well as the end and the footer or the news story, depending on where the news source listed authorship information. Third, we coded whether the news story contained a link to the research study, either within the body of the news story or at the end of it.

Statistical Methods

Throughout the results we present the result of various binary logistic regressions to estimate the relative probability of various outcome variables (e.g., that references are framed as valid or that they are accompanied by bibliographic or study details) in relation to various predictor variables (e.g., that stories are focused on science communication or that they have not been reposted from elsewhere). The calculated models are all in the form (Y = 1) = β₀ + β₁x, where Y is a binary outcome variable coded as 1 if at the studied variable was observed for a reference, and 0 otherwise, and where x is the predictor, coded as 1 if the story where the reference is observed is classified according to the variable in question, and 0 otherwise. Calculations were performed using the Python statsmodels package (Seabold and Perktold, 2010).

Results

How Often Was Research on Opioid-Related Disorders Mentioned in News Stories?

From 2017 to 2018, the nine news sources collectively published 149 news stories that included 223 references to 164 unique studies. Comparing this number to the results of the Media Cloud query for news stories from the same sources that broadly relate to the opioid crisis (n = 8,600), these results indicate that less than two percent stories mentioned or linked to scientific research on the prevention, harm reduction, and treatment of opioid-related disorders.

In relation to the published research on opioid-related disorders, the proportion of studies that get mentioned by the news media is similarly low. Of the 11,028 records that matched our PubMed query for research records corresponding to the prevention, harm reduction, and treatment of opioid-related disorders in January 2019, the majority of scientific studies were not referenced by any of the selected news outlets in 2017 or 2018, and only 164 studies (1.5%) appeared in at least one of news story in our sample. Of these, 129 (78.6%) studies were referenced only once, 22 (13.4%) twice, and 13 (7.9%) three or more times.

The most recently published studies received the most coverage, with more than half of the references published between 2016 and 2018 appearing in at least one story (n = 84, 51.2%). Over a quarter (n = 46, 28%) of the studies were free to read at the publisher's website, and an additional 36 (22%) were freely available in an institutional or subject repository. In total, slightly more than half of the studies (n = 84, 51%) were freely available for journalists and the public to read.

Not every news source in our study referenced scholarly publications to the same degree or in the same way. We examined the number and proportion of references found in science communication (as opposed to issue-focused stories) and original news stories (as opposed to reposted stories), as well as references framed as valid science or with at least one bibliographic or one study design detail (Table 3). However, since these differences cannot be meaningfully studied with this sample, the remaining analysis is performed using the collection of stories and research references as a whole.

TABLE 3

Table 3. Research references by news source.

What Contextual Information Do Journalists Provide When Referencing Scholarly Publications About Opioid Related Disorders?

Most references to studies about opioid-related disorders appeared in the context of a larger news issue (i.e., issue-focused stories, n = 170, 76.2%), while only a quarter of them (n = 53, 23.8%) appeared in stories dedicated to science communication. The vast majority of the references appeared in original stories (n = 188, 84.3%), with only 15.7% of references (n = 35) appearing in stories that had been previously published by another source. Most references included a hyperlink to the original study (n = 192, 86.1%). However, in more than 70% of republished cases (n = 25), the original news story included an outbound link to the study, but the republished version did not.

The majority of references did not include bibliographic information about the research study—such as author names, affiliations, or journal titles—which can function as heuristic cues for scientific credibility and enable readers to locate the original study (Table 4). One in four research references included author affiliations (n = 56, 25.1%), one in five named the authors (n = 45, 20.2%), and 39 (17.5%) named both. References that noted journal titles were equally rare (n = 43, 19.3%). Likewise, most references did not provide any information about the research design or process, and only reported the findings (n = 187, 84%; Table 5). Merely 36 references include any details about the study design, most frequently sample sizes (n = 27, 75%) or study timeframes (n = 19, 52.8%). Various other details, such as the methodological approach or sample specifications, were equally rare (n = 20, 55.6%). Generally, such references provided only one or two study details (n = 16, 44.4% and n = 13, 36.1%, respectively), and more often appeared in science communication stories (stories: n = 26, 52%; references: n = 26, 49%) than in issue-focused ones (stories: n = 9, 9%; references: n = 10, 6%). A logistic regression model confirms that mentions found in science communication stories were statistically more likely to be accompanied by both bibliographic details (odds ratio [OR] = 11.14, p < 0.001) and study details (OR = 13.92, p < 0.001) when compared to mentions found in issue-focused stories.

TABLE 4

Table 4. Research references that provided bibliographic information about the study, grouped by frame.

TABLE 5

Table 5. Research references that included details about the research design and methodology, grouped by frame.

How Are Scholarly Publications About Opioid-Related Disorders Framed?

The vast majority of the 223 research references in our sample framed science as valid (n = 184, 82.5%). The uncertain (n = 17, 7.6%) and controversial (n = 22, 9.8%) science frames appeared much less frequently (see Tables 4, 5).