Carried out by experts, heuristic evaluations assess components of a product or service, comparing it against best practices to produce a rating and recommendations, if necessary (Rosenzweig, 2015). While not an in-depth form of evaluation, usability heuristics provide a quick assessment of an interface. To conduct the heuristic evaluations, the recommended approach when using it alongside other methods of data collection is to conduct it early in the research process (Jeffries and Desurvire, 1992). Thus, as a first step, research team members (n = 4) individually assessed each tool using established heuristics (Nielsen, 1994; Law and Hvannberg, 2004). In cases of disagreement in assessment, tools are revisited to reach a mutually agreed upon severity rating. Further cases of disagreement result in the mean severity rating being taken as the final rating. With guidelines recommending three to four evaluators to increase the reliability of the mean severity rating (Nielsen, 1994), our use of four evaluators meets these criteria.

To facilitate the testing of software tools through potential scenarios that are typically encountered by end users, black-box software testing methods (Nidhra, 2012) using a task-based approach are used to assess how a user would interact with digital disinformation-countering tools using a real-life situation (Rosenzweig, 2015). This approach assists in the assessment of the overall efficacy of disinformation-countering tools to meet the contextual, technical, and cognitive needs of individuals using them as part of their daily information consumption activities. Remote-moderated testing using a think-aloud protocol was selected as it provides for direct observation of user experiences, note-taking, asking clarifying questions, and assisting if an explanation is needed (Rosenzweig, 2015; Riihiaho, 2017; Lewis and Sauro, 2021). We use the platform Loop11 to facilitate the sessions as it prompts scenarios and tasks while recording participants’ screens and audio.

Before conducting the usability evaluation with the main participants, moderators (n = 2) created an evaluation script for use during evaluations. A pilot study with four participants identified deficiencies in scenarios and questions. Modifications were made according to feedback.

While conducting research into the success of information systems, it is recommended to consider the technical abilities of the users (Weigel and Hazen, 2014). Thus, considering our intent to explore the usability of digital tools to represent users with varying technical proficiency, we establish three categories of technical capability, “beginner,” “intermediate,” or “advanced” information technology users. Selected participants self-identified according to the high-level definition provided.

To determine a representative number of participants, we refer to the opinion that 5–8 people can find most of the usability issues for qualitative evaluations (Nielsen and Landauer, 1993; Rosenzweig, 2015), whereas qualitative survey evaluations are recommended to have a minimum of 20 participants for a statistically representative sample (Rosenzweig, 2015). Thus, we aimed to have 15–20 participants. A call for participation was shared with peers and publicly posted on networking and social media platforms to solicit participants. Those interested were asked to complete an online form with their name, email, and level of technical competence.

Using purposive sampling, we ensured that the minimum requirement for each persona was fulfilled (Tashakkori and Teddlie, 2003). Moreover, to seek heterogeneity in the final sample and maximize the generalizability of the results, participants of varying age and education levels were preferred (Lewis and Sauro, 2021). As compensation, a $40 participant-selected Amazon or Starbucks gift card was given to each participant. Eighteen participants took part in the usability evaluation, and their demographics are provided in Table 1.

To represent real-life situations, scenario-based tasks were used to uncover potential issues experienced while information credibility checking. As summarized in Table 2, three articles were used for fact-checking and were selected based on the diverse forms of disinformation that individuals could be subjected to in their daily lives, either through social media posts or received from friends or family. As news with sensational headlines is most likely to be accessed and shared (Burkhardt, 2017), articles with such headlines were selected. They include Chinese Herb Kills Cancer Cells (pseudoscience), Biggest Lie in World History: There Never Was A Pandemic (conspiracy theory), and Hairdresser Arrested for Making Voodoo Dolls from Customers’ Hair (satire).

We chose to use browser extensions and stand-alone websites to investigate different modalities of use for countering disinformation, as well as to ascertain user preferences and the functional efficacy of each style of tool. Based on our initial testing, three browser extensions and two websites were selected (Table 2; Figures 1–3).

Participants first met via Zoom to receive information on the study and tasks to be completed. Once agreed, participants electronically signed the consent form and returned it to the moderator via email. Upon receipt, the evaluation was carried out in Loop11.

Participants began by answering several demographic questions (5 min), followed by the tasks (30–45 min). To validate claims made within the identified articles, participants first visited selected websites, followed by the browser extensions. As the browser extensions require additional steps for installation, participants were instructed to install the extension and had an opportunity to review any developer-provided information guides. Participants then opened identified articles to assess how the extension interacts with the website and/or authenticates information.

Upon completion of tasks for each tool, summative questions were asked to collect participants’ final opinions, which included the shortened SUS (5 min) and multiple choice questions (5 min). Debriefings were held via Zoom to thank participants and answer final questions.

To analyze the task-based usability evaluation data, thematic content analysis was used to identify ideas, patterns, or trends (Krippendorff, 2012, 2018). Extracted data were then quantified for quasi-statistical consideration to demonstrate the frequency of occurrence (Becker, 1970). Results of the quantitative questions collected in Loop11 are to be exported to Excel for analysis. SUS questionnaire results are to be analyzed using a programmed spreadsheet (Excel Spreadsheet for Calculating SUS Scores, 2008) and then represented using adjective-based scoring to facilitate understanding (Bangor et al., 2008).

As highlighted in Table 3, a numeric score ranging from zero, where no issues were found, to four, which represents a severe issue, was assigned for each heuristic. Considering the severity of ratings, the poorest performing heuristic is the lack of help and documentation, followed by a lack of match between the system and the real world. The third worst-rated heuristic is a tie between the visibility of system status and consistency and standards.

To elaborate on the lack of help, while some tools have some form of “faq,” “how it works,” or an overview of the extension, Table 3 shows that apart from MBFC-E, the tools have little support on how to use it. Considering the lack of match between the system and the real world, all tools had poorly represented information and had an insufficient explanation of what terms meant, leading to confusion. The visibility of system status was rated poorly across all but the ITC-W, due to either a lack of knowing how to open the extension or whether a search is being executed when queries are entered. Finally, except for TNR-E, the lack of consistency and standards was rated poorly again across tools due to their lack of consistency in the terms used and forms of interaction. While the tools should reduce user’s cognitive load by being designed in a way that users are familiar with, and are intuitive (Nielsen, 2020), most tools were not designed in a way that is easy to understand, particularly when receiving results.

Considering ITC-W has the lowest overall severity ratings, it may suggest that the website is a more user-friendly tool than the others, potentially indicating that it may provide a more appreciable experience and may be a better option for disinformation-countering. Nevertheless, with the severity rankings being high across several tools, this suggests serious deficiencies, which may hinder their use. Acknowledging the importance of these tools and the need for amplifying their use to circumvent the harms of disinformation, it is evident that improvements are required.

Given that a major aspect of user experience relates to users’ overall satisfaction and how efficiently tasks can be performed, we collected participants’ feedback and challenges experienced while attempting to verify claims made within the articles provided.

Upon analysis of the data collected, it became evident that the challenges experienced were concentrated across four main categories, including, issues related to the design of the interface, how to verify information, receiving results, and then understanding the results received. Each of these will be discussed, with a sample of participant commentary highlighted in Table 4.

Once participants completed tasks, they were asked to provide SUS ratings and answer multiple choice questions. Thus, in addition to participants’ thoughts, opinions, and a count of the frequency of which an issue occurred, these scores provide a quantitative score for each tool assessed.

To determine whether certain users fare better or worse in the use of disinformation-countering tools, our objective was to identify usability challenges faced by users of varying technical proficiency. In this regard, we have found inconsistency in experiences, suggesting that all users face challenges regardless of their technical abilities.

To elaborate, we mention several occurrences where participants had greater discrepancies in their experiences. First, considering challenges experienced within the user interface, as seen in Figure 4, beginners more often complained of being distracted by the content in MBFC-W than their counterparts. While attempting to verify information, advanced users had more difficulty figuring out how to perform a search in MBFC-W and required multiple attempts (Figure 5). When results of search queries or feedback from the browser extension were received, due to their inability to execute a successful search query or recognize that results had been received, the intermediate participants were more likely to end the task without verifying information in MBFC-W and TNR-E (Figure 6).

Considering whether participants understood the meaning of the search results or feedback from the extensions, beginners only slightly lacked an understanding of the results in TNR-E, compared to others, whereas intermediates were behind their peers while interacting with MBFC-E (Figure 7). For those who complained of not enough information being provided, intermediate and advanced were more likely to complain of not enough information in TNR-E, whereas the advanced users had similar complaints using ITC-W and further sought an instruction guide to try to understand the results received (Figure 7).

Thus, given the inconsistencies in challenges experienced, it appears that user performance relates more to the individual tool interacted with, the depth of information they wish to see, and their ability to interpret results, rather than their technical proficiency. Moreover, many cases wherein participant experiences were all closely aligned suggest the severity of the usability issues in the disinformation-countering tools. In this case, it cannot be stated that one group of user profiles fares better than the others.

To address our second research objective, in this section, we address recommendations for both tool developers and researchers to optimize the creation of disinformation-countering tools to improve resilience to disinformation. To the best of our knowledge, recommended design principles and guidelines have yet to be put forth for the design of disinformation-countering tools. Therefore, whereas our findings in the results and discussion sections pertain specifically to our study and investigation of disinformation-countering tools, our recommendations section integrates our findings with existing literature to put forth comprehensive suggestions for tool design.

Our study is not without limitations. We first mention that had we used mainstream news sources, the tools generally would have provided better information and a particularly much-improved experience. Our results would likely have fewer identified usability challenges and are strongly in favor of ITC-W. Moreover, consideration was not given to the format in which the news articles may be received and read, as well as any pre-existing opinions the participants may hold, nor their awareness of and caution toward potential disinformation. Furthermore, the tools selected for our study only form a small sample of the disinformation-countering websites, browser extensions, and applications created. Therefore, with the limited number of articles and types of information verified, our findings may not present a full image of the capabilities offered by disinformation-countering tools. As such, our results cannot be generalized to all currently available tools.

On this note, we also mention that the NT-E tool was discontinued at the end of our period of study and the MBFC-W has since been updated with fewer advertisements, thus limiting future replicability of our study. Finally, given that we had less than the 20 participants required to establish statistically significant findings for our summative questions, the results can be considered informative, however, not statistically significant. Financial and time limitations prevented the inclusion of additional participants in our study.