Online Information Acquisition (OIA) focuses on the searching and accessing of online information, for example by using general and specialized search engines and databases, specifying search queries, opening specific websites. Beyond these more technical aspects, COR focuses in particular on searching for specific platform entries and passages and terms on a website in as far as they contribute to an (efficient) accessing of relevant and trustworthy information and avoidance of untrustworthy information (Braten et al., 2018; the Information Search Process model, Kuhlthau et al., 2008).

Critical Information Evaluation (CIE) is crucial for self-directed, cross-sectional learning based on online information. This facet focuses on students' selection of information sources and evaluation of information and sources based on website features or specific cues (e.g., text, graphics, audio-visuals). Following comprehension-oriented reception and processing, CIE is used to differentiate and select high- instead of low-quality information (relative to one's subjective standards and interpretation of task requirements). A cue can be any meaningful pattern in the online environment interpreted as an indicator of (trustworthy or untrustworthy) online media or communicative means. Examples of cues may be a URL, title or keyword on the search engine results page, a layout or design element, media properties, an article title, information about author, publisher or founder, publication date, certain phrasings, legal or technical information. Trustworthiness “evaluations” typically include targeted verification behavior, which results in a (defeasible) “judgment” about a web medium or piece of information, which may be based on an initial heuristic appraisal without further (re-)evaluation. However, CIE as “evaluation” can require a more systematic analytical, criteria-based judgment process for students, possibly using multiple searches to establish reliable and warranted knowledge (for an overview on related multiple document comprehension frameworks, see Braten et al., 2018; e.g., the Discrepancy-Induced Source Comprehension (D-ISC) model, Braasch and Bråten, 2017).

Reasoning with Evidence, Argumentation, and Synthesis (REAS) is probably the most important facet of COR, which distinguishes this construct from “literacy” constructs (e.g., digital, information or media literacy). This facet focuses on uniting the initially appraised information, weighting it against further indications and perspectives, and using it as evidence to construct a convincing argument that accounts for uncertainty (Walton, 2006). Argumentation is a well-suited discourse format for deliberating whether to accept a proposition (e.g., to trust or distrust). Evidence-based argumentation imposes certain quality standards for a well-founded judgment (e.g., rationality) and requires minimal components of a claim, reasons, evidence (and data) and conventional inferential connections between them (e.g., Argumentation Schemes, Walton, 2006; Walton et al., 2008; Fischer et al., 2014; Fischer, 2018).

These three main facets, OIA, CIE, and REAS, are primarily considered cognitive abilities. Each of them can also take on a metacognitive quality within the COR process, for example as reasoners (internally) comment on their ongoing search, evaluation or argument construction (e.g., “I would not trust this website”), or (self-)reflect on previously acquired knowledge to identify incorrectness or inconsistencies (e.g., “This sentence here contradicts that other source/what I know about the subject”). The latter reflection can become epistemic if it turns to the method of information acquisition and reasoning itself (e.g., “How did I end up believing this scam?”).

These main facets are accompanied by an overarching, self-regulative, metacognitive COR component that activates deliberate COR behavior and coordinates (transitions) between the COR facets in the progression of COR activity, particularly for activating a critical evaluation and deciding when to terminate it, in relation to other events (e.g., during a learning experience, social communication)³. Self-regulation can be applied to monitor and maintain focus (noticing unfocused processing, returning to task) and handle environmental signals (identifying and minimizing distracting information features) (Blummer and Kenton, 2015). As reasoners may have affective responses (Kuhlthau, 1993) to their task progress and to specific information (particularly on controversial topics), affective self-regulation can not only support them in staying on task and keeping an open mind, but they can use it meta-cognitively for COR to gain an insight into unconsciously processed information (e.g., identifying and coping with triggered avoidance reactions or anxiety induced by ambiguity or manipulation attempts) and can critically reflect on triggers in the source cues.

Thus, Metacognitive Activation (MCA) is assumed to be an ability required to activate COR in relevant contexts. (Epistemic) metacognition can be characterized by gradations of self-awareness regarding information acquisition, evaluation and reasoning processes, which may activate a “vigilance state” in students and lead to certain (subconscious) reactions (and a habitual affective response, e.g., anxiety, excitement), or can also be interpreted as an indicator of a potential problem with processed information (“am I being lied to/at risk after misjudging the information?”) at the metacognitive level (on uncertainty and emotions when searching for information, see Kuhlthau, 1993; on ambiguity experience as the first stage in a general critical reasoning process, see Jahn and Kenner, 2018), which may lead to the activation of an evaluative COR process.

The main facets of COR and the overarching metacognitive self-regulative component are understood to determine COR performance (and are the focus of the CORA, Section Test Definition and Operationalization of COR: Design and Characteristics of CORA Tasks). The main COR facets are assumed to rely on “secondary” sub-facets that provide support in cases where related specific problems occur, including self-regulation for minimizing distractions and on-task focus, as well as diverse knowledge sub-facets.

Knowledge sub-facets may include, for OIA, knowledge of resources and techniques for credibility verification; for CIE, knowledge of credibility indicators and potentially misleading contexts and framings, manipulative genres and communication strategies; for REAS, knowledge of reasoning standards as well as fallacies, heuristics, and perceptual, reasoning and memory biases as well as of epistemic limitations for trustworthiness assertions. The list is non-exhaustive, and the knowledge and skills are problem-dependent (e.g., checking for media bias will yield conclusive results only if there is in fact a bias in the stimulus material); they can be expected to impact COR in related cases. Hence, controlling for corresponding stimuli encompassed in the task is recommended.

Attitudinal dispositions for critical reasoning and thinking, such as open-mindedness, fairness, and intellectual autonomy (Facione, 1990; Paul and Elder, 2005) are equally likely candidates for COR influences. These secondary facets are not examined in the current conceptualization.

For acquiring information online in a warranted way, students need to successfully identify and use trustworthy sources and information and avoid untrustworthy ones. In contrast, unsuccessful performance is marked by trusting untrustworthy information, a gullibility error, or refusing to accept trustworthy sources, an incredulity error (Tseng and Fogg, 1999). To decide which information to trust and use, students need to judge information in regard to several criteria, including at least the following: usefulness, accessibility, relevance, and trustworthiness. Information may be judged as useful if it advances the inquiry, for instance by supporting the construction of an argument; usefulness may also be understood as a holistic appraisal based on all other criteria. Lack of accessibility (or comprehensibility) limits students to the parts of the information landscape that they can confidently access and process (e.g., students may ignore a search result in a foreign language or leave a website with a paywall, but also abandon a text they deem too difficult to locate or understand in the given task time). In an open information environment, successfully judging relevance as relatedness or specificity to the topic of inquiry and trustworthiness or quality of information enables students to select and spend more time on high-quality sources and avoid untrustworthy sources. Assuming students will attempt to ignore information they judge as untrustworthy, any decision in this regard affects their available information pool for reasoning and learning.

The judgment of trustworthiness as an (inter-)subjective judgment of the objectively verifiable quality of an online media product against an evidential or epistemic standard is central to COR. In more descriptively oriented “web credibility” research, a credibility judgment is understood as a subjective attribution of trust to an online media product; trustworthiness in COR is closely related, but presupposes that the judgment can be based on valid or invalid reasoning (acceptable or unacceptable based on a normative standard) and hence can be evaluated as a skill. Trustworthiness in COR can be considered a warranted credibility judgment. Consequently, COR enables students to distinguish trustworthy from untrustworthy information and, more specifically, various sub-types based on assumed expertise and communicative intent, for example: accidental misinformation due to error, open or hidden bias, deliberate disinformation, and (non-epistemic) “bullshitting.” A more fine-grained judgment is assumed to afford higher certainty, a more precise information selection, and more adequate response to an information problem. To successfully infer the type of information, reasoners may evaluate cues from at least three major strands of evidence about an online medium, including cues on content, logic, and evidence; cues on design, surface structure, and other representational factors; and cues on author, source, funding, and other media production and publication-related factors. Reasoners may evaluate these themselves (using their own judgment), trust the judgment of experts (external judgment), or a combination of the two; when accepting external judgment, instead of the information itself, reasoners need to judge at least their chosen expert's topic-related expertise and truth-oriented intent.

Based on the CORA framework, COR is modeled for generic critical online reasoning (GEN-COR) on tasks and websites that do not require specialized domain knowledge and are suited for young adults after secondary education. The construct can be specified for study domains (DOM-COR), for instance by defining domain standards of evidence for distinguishing trustworthy from untrustworthy information and typical domain problems regarding the judgment of online information.

Different gradations can be derived based on task difficulty, complexity, time, and aspired specificity of reasoning (Sections Test Definition and Operationalization of COR: Design and Characteristics of CORA Tasks and Scoring Rubrics). COR ability levels were distinguished to fit the main construct facets depending on students' performance in (sub-)tasks tapping OIA, CIE, and REAS (see rubrics in Section Scoring Rubrics; Table 1).

Low-quality information on the linked websites can be caused by a lack of expertise of the author(s), belief-related bias, or accidental errors when drawing inferences or citing from other sources. Moreover, the linked sources offer contradictory information or inconsistencies between multiple online texts, which learners need to resolve in the process of acquiring consistent knowledge. In our example (Figure 2), the provided link leads to a website that offers information about vegan protein sources. At first glance, the website seems to provide accurate and scientifically sound information about vegan nutrition and protein sources, but upon closer inspection, the information turns out to be biased in favor of vegan protein sources. The article is shaped by a commercial interest, since specific products are advertised. This bias can be noticed by reading the content of the website carefully and critically. The existence of an online shop is another indication of a commercial interest motivating the article. In contrast, the references to scientific studies give a false sense of reliability.

As the construct definition of COR states (Section Construct Definition of Critical Online Reasoning), if students wonder about the trustworthiness of certain online information during the inquiry, this should be a sufficient initial stimulus to activate their COR abilities. Thus, we explicitly include the stimulus at the beginning of an inquiry task prompt of all CORA tasks. The in-task cues can tap these activation routes even if the students did not respond to the initial prompt at the beginning of the task (Figure 2). In the example, the participants are also asked whether the website is reliable to stimulate the COR process and a web search.

Following the ECD (Mislevy, 2017), we describe the task model and the student model of the CORA in more detail.

Task difficulty in terms of the cognitive requirements of the construct dimensions of COR varies through the task properties and the prompt (i.e., difficulty of deciding on a specific solution by considering pros/cons or both). For instance, in the dimension of OIA, task difficulty varies in terms of whether students are required to evaluate a website and related online sources or only a claim and related online sources. The quality of the websites found in the free web searches is likely to significantly vary between test participants, which is not explicitly controlled for in the task and in the scoring of the task performance. This information is only examined in additional process analyses using the recorded log files (Section Analyses of Response Processes and Longitudinal Studies).

In the easy CORA tasks, the web authors were aware that they may be biased and alerted their audience to this fact, for instance by stating their stance directly or by acknowledging their affiliation to a certain position or perspective—the students then had to take these statements into account in their evaluation. In the difficult CORA tasks, the web authors actively tried to conceal the manipulative or biased nature of their published content—and the students had to recognize the techniques these authors employed. In addition, they had to identify the severity of this manipulation and to autonomously decide which information was untrustworthy and should therefore not be taken into consideration. This untrustworthy information can comprise a single word or paragraph, an entire document, all content by a specific author or organization, or even entire platforms (e.g., if their publication guidelines, practices, and filters allow for low-quality information) or entire geographical areas (e.g., due to biased national discourse).

For each CORA task, we developed a rubric scheme that describes the aforementioned specific features of the websites linked in the task, for instance in terms of credibility and trustworthiness of the information they contained (for details, see Section Scoring Rubrics). To develop the psychological stimuli encountered in CORA tasks (in accordance with the construct definition; Section Construct Definition of Critical Online Reasoning), we based our approach on a specific classification of misinformation by Karlova and Fisher (2013) and on classifications of evaluative criteria of information quality (e.g., topicality, accuracy, trustworthiness, completeness, precision, objectivity) by Arazy and Kopak (2011), Rieh (2010, 2014), and Paul and Elder (2005).

Cues indicating trustworthiness or lack thereof were systematized in evidence strands according to the Information Trust model (Lucassen and Schraagen, 2011, 2013). The model distinguishes evidence on author, content, and presentation, which are aligned with classical routes of persuasion in rhetoric; each requires a different evaluation process. We expand this model by a distinction of personal evaluation vs. trust in a secondary source of information (Table 1).

The task difficulty level was gauged in particular by the scope and extent of misinformation based on an adaptation of the Hierarchy of Influences model by Shoemaker and Reese (2014), which assesses agents in the media production process and their relative power to shape the media message—and hence introduce error or manipulation, which need to be judged by students to discern the limits of warranted trust (e.g., at the bottom end are obvious deceptions and errors by the author such as SPAM emails or simple transcription mistakes in a paragraph, while at the top end are high-level secret service operations or a society-wide cultural misconception).

Task difficulty in terms of required argumentative reasoning in CORA was varied in three ways: (1) Scaffolding was added to the task prompts by asking students only for part of the argument (e.g., only pro side, con side, or only specific sub-criteria) to reduce the necessary reasoning steps. (2) The stimuli websites were selected by controlling for (i) scope and (ii) order of bias or misinformation, and for how difficult it is to detect it. Scope refers to the comprehensiveness of biases or misinformation based on the adapted Hierarchy of Influences model by Shoemaker and Reese (2014). The order is the level of meta-cognition that needs to be assumed in relation to a bias or misinformation. (3) The composition of sources that can be consulted for information (i.e., number of supporting and opposing, or high-quality and low-quality sources) can again be modified only in a closed Internet-like environment (Shavelson et al., 2019; Zlatkin-Troitschanskaia et al., 2019), but it can hardly be controlled for on the open Internet.

The natural online environment used in this assessment constitutes a crucial aspect of the CORA task difficulty (that is also related to the reliable scoring of task performance; Section Scoring Rubrics). In a closed information environment with a finite number of sources, a comprehensive evaluation of all sources is possible. On the Internet, an indefinitely large number of sources are available. Hence, when solving the CORA tasks, students also need to constantly decide whether to continue examining a selected source to extract more information (and how deeply to process this information, e.g., reading vs. scanning) or whether to attempt to find a more suitable source, and a sample of search hits on a search engine results page, and whether they should use different search terms or even switch to a more specialized search engine or a specific database that might yield more useful information. This aspect is related to the student model and the primary aim of students in the context of inquiry-based learning based on online information, which is to gather information to “fill” their knowledge gaps while carrying out a task. Learning in an online environment requires students' initial (and later updated) understanding of the problem in relation to a specific generic or domain-specific task, and recognition of the types of information that are needed to solve a given problem, and then carrying out the steps to locate, access, use, and reason based on information, and finally formulate an evidence-based solution to the problem.

The expected response processes while solving the CORA tasks can be described with a focus on their basic phases based on the abovementioned Information Problem-Solving using the Internet (IPS-I) model (Brand-Gruwel et al., 2009): (1) Defining the information problem, (2) Searching information, (3) Scanning information, (4) Processing Information, (5) Organizing and presenting information. These phases are quite common in many other models and categorizations of information search and also media and digital literacy (e.g., Eisenberg and Berkowitz, 1990; Fisher et al., 2005). For a multi-source information problem, we expect that the processes will be iterated for each new source. An additional meta-cognitive Regulation component guides orientation, steering, and evaluation, and can be active throughout interacting at each phase (Brand-Gruwel et al., 2009). Required judgments of information trustworthiness can be situated in the meta-cognitive component of evaluation. Within the evaluative process, trustworthiness judgments might be juxtaposed with judgments of accessibility, relevance and/or usefulness at several points in addition to the ongoing collection of information for the inquiry. Based on these categorizations, we developed a fine-grained description of the (sub)processes the students are expected to perform while solving the CORA tasks (Table 2).

In the following, we describe the student model with regard to the four main COR facets in more detail.

In CORA, the test takers are required to produce an argumentative conclusive written response based on the consulted and critically evaluated online sources. In line with the older IPS model (Brand-Gruwel et al., 2005), we have added a reflective metacognitive review as an expected process, which may occur at any moment but possibly upon response verification, to highlight that COR may be activated even after an iteration of the IPS-I process or after the whole CORA task has been completed without critical consideration.

Judgments of usefulness, accessibility, relevance, and trustworthiness of online information can be attributed to the COR facet CIE that is represented as a (meta-)cognitive evaluating component in the IPS-I model. A judgment may require a more elaborate evaluation based on additional information searches. Hence, we assume that a spontaneous trustworthiness judgment can occur at any stage in the IPS-I model. Additionally, a more deliberate, likely criterion-based, reflective evaluation of information, for instance in terms of its trustworthiness, can be performed as a specific (scheduled) sub-stage—if the student is aware of the need to evaluate the information.

The sections in the IPS-I process for evaluations of trustworthiness and other judgments also indicate that they can be interwoven with comprehension and reasoning activities and with each other (Section Construct Definition of Critical Online Reasoning, Figure 1). However, they are also likely to be distributed across several stages and differ in the content of partial evaluations and possible inferences drawn. The more detailed view of judgments and evaluations by search phase indicate that several judgments are likely to occur per phase and judgments of accessibility, relevance, trustworthiness, and usefulness are differentially important across phases and touch upon different sub-questions per phase. For instance, trustworthiness evaluations can be both fast, if an exclusion criterion is found, or gradual over one or several stages, including the collection of multiple cues. We assume this to be the case for information and sources that students evaluate as part of the CORA task. For other additional sources found during web searches, it is likely the student will evaluate trustworthiness just once and with little effort, i.e., heuristically, if they know they can go back to searching a more trustworthy source faster. (i.e., it is not the student's intention to find and determine every untrustworthy source on the Internet, but find one that is not untrustworthy and meets their needs). We therefore expect that the CORA tasks tap a judgment of trustworthiness, including either a systematic criterion-based evaluation (to the extent to which the test-taker is aware of criteria for trustworthy or untrustworthy information) and/or a vigilant recognition of the specific information features that may help the participants identify bias and misinformation.

In this context, the Information Search Process model (Kuhlthau, 1993) links behavior, cognition, and affective responses, with cognition being characterized by gradations of self-awareness regarding information search process (Figure 1). Here, again, we assume multifold interrelations with the metacognitive facet of COR. Therefore, we expect that recognizing a cue in the linked information in the CORA task (i.e., stimuli), indicating possible bias or misinformation, may activate a “vigilance state” in the students and lead to certain (subconscious) reactions (and a habitual affective response, e.g., anxiety, excitement) or can also be interpreted as an indicator of a potential problem (“am I being lied to?/at risk after misjudging the information?”) at the metacognitive level, which may lead to the activation of the facet of COR, i.e., (meta)cognition for critical reasoning activation (on the role of uncertainty and emotions when searching for information, see Kuhlthau, 1993). In this regard, we consider the ambiguity experience as the initial stage in a general critical reasoning and evaluation process, i.e., a cognitive appraisal marked by uncertainty about the validity of one's interpretation of the current situation that leads to a need for more clarity (or to avoid the problem-solving situation, e.g., in case of a low self-efficacy), which may prompt an expected response behavior during the CORA tasks, i.e., critical reflection and evaluation.

In terms of the task model, this ambiguity is tapped by the CORA task description and the prompt, which explicitly asks students to judge the trustworthiness of a given website or claim. Thus, the task prompt is the initial stimulus for students to activate their trustworthiness evaluation since the question of whether or not information is trustworthy is explicitly given by the task prompt; the second are the cues offered by the stimulus materials embedded in the CORA tasks; the third is the reminder in the response field of the CORA task to formulate a short statement to the task questions and to list the consulted online sources (Section Scoring Rubrics). The CORA task prompts explicitly require students to formulate a response, justify it with reasons and arguments, and back these up by citing URLs of sources used to reach their decision. Thus, students' responses comprise the fundamental components of argumentative reasoning (Section Theoretical Components of COR). In CORA, we framed the trustworthiness evaluation through an argumentative model, and modeled (possible) stances on a (trustworthiness) issue and their supporting reasons and evidence (cues). Alternatively, students might not reason deeply about it, but apply cognitive heuristics (Kahneman et al., 1982; Metzger and Flanagin, 2013). However, given that it is explicitly prompted in the task, we expect students to apply argumentative reasoning and to be able to identify cognitive heuristics (e.g., authority biases) within their argumentation.

In this context, one aspect is particular important in terms of the interpretative model (Section Scoring Rubrics). Assuming that cognitive biases (e.g., confirmation bias), and motivated reasoning can be tapped by controversial topics as presented in the CORA tasks, an opposing stance toward a given topic (i.e., skeptical) affords more stimuli to be critical and motivates the student to find evidence of misinformation. This is why a balanced selection of various topics was established in CORA. We assume students' initial personal stance on the task and its topic will depend on a number of influences, controlled for in CORA (e.g., prior domain knowledge, attitude toward the task topic). This aspect is crucial since students may pass different credibility judgments and follow diverse reasoning approaches depending on their initial stance (Kahneman et al., 1982; Flanagin et al., 2018). At a later, longitudinal research stage (e.g., in the context of formative assessments; Section Analyses of Response Processes and Longitudinal Studies), attitude-dependent tasks can be administered to assess COR levels among students for topics they explicitly support or oppose. Not solving the task in a way that accounts for both perspectives would therefore yield a lower CORA test score (Section Scoring Rubrics). This in turn would strengthen the high ecological validity of CORA.

Whichever stance the students choose, they will not be awarded points unless they provide warrant through reasons and arguments, and back them up with evidence from the evaluated website and further consulted online sources (Section Scoring Rubrics). Thus, an evaluation supported by reason and evidence (such as a link to an authoritative website), judged by raters as acceptable against a generic or domain-specific quality standard, is used to infer the extent to which students' have critically reasoned with and about online information. The call to justify is explicitly prompted in the task (“provide a justification”) and the backing with evidence is required in a separate field, asking for the URLs of consulted further websites. Providing citations is a common form of evidence in academic writing, and copying a URL does not require an elaborate evidential standard. The reasons and arguments students cite in their written responses are scored for plausibility and validity based on a few rules (e.g., “trusting only the source's own claims about itself is not sufficient reason”). The indicated URLs are also evaluated in terms of their trustworthiness (Nagel et al., 2020). We assume that students with advanced COR abilities cite only the best sources they found and used to back up their argument. Conversely, indicating many relevant and trustworthy sources as well as irrelevant and untrustworthy ones was considered an indicator of reasoning that was not fully sufficient (see scoring rubrics in Section Scoring Rubrics).

According to the fundamentals of argumentation, the main claim, reasons, and backing (e.g., evidence) are the basic elements of a reasonable argument (Toulmin, 2003; Walton, 2006). Hence, we considered indications of these, which are also explicitly prompted in the CORA task, in a somewhat aligned manner in the students' responses as evidence that students performed argumentative reasoning. Some argumentation frameworks include further basic components, such as rebuttal and undercut as types of opposing reasons or inclusion of consequences (Toulmin, 2003). These components can be included in further CORA tasks (Section Refining and Expanding CORA), but were not required for the short online evaluation tasks. Moreover, in terms of metacognitive evaluation, students are expected to engage the evaluative critical reflection, i.e., “self-reflective review” of their task solution after formulating their response to the task.

Overall, with regard to the suitability and validity of the CORA tasks for higher education in Germany, most experts agreed and confirmed the content and ecological validity of the CORA tasks and recommended further expansions. For instance: “The task is clear, the instruction is also clear, and it seems obvious that they need to formulate a response.”

One expert, after pondering how to translate and adapt the U.S. tasks, and worrying about cultural suitability, considered the CORA tasks and commented: “These [German] tasks are really a hundred times better for Germany.”

One question critically discussed with experts addresses the domain-specificity of the CORA tasks. Here, the experts confirmed that the six tasks cover generic COR ability. For instance: “No domain-specific knowledge is required. It's a good selection for the news/science context.”

One concern that was raised by most expects regards the suitability of the testing time to assess all facets of COR, and in particular the REAS facet. However, experts also agreed that students may not dedicate more time to the task when evaluating an information source in a real setting. As one expert notes: “There are 10 min to conduct a search. One may doubt if people would commit as much time in everyday life, unless they really took the time to carry out a more detailed search.” At the same time, the natural online environment of the assessment was praised in terms of the high ecological validity of CORA by all experts: “The mode of administration as given here is important, since it enables assessing internet search behavior.”

The new rating scheme with the subscores and evaluation categories was positively evaluated by the experts, although they stressed the high complexity of the scoring rubrics. For instance: “It is also good that you have different degrees, not only “right or wrong.” Of course, this places high demands on coders, but with training, it is doable.”

Most experts positively evaluated the representativeness of the chosen media, i.e., media that students frequently use online. However, one expert criticized that “scientific and journalistic media were indeed covered, but the selection could include more reputable media as well as some media more on the lower quality end of the spectrum. The ones here are well chosen; one cannot immediately tell if they are fabricated or not.” Another expert proposed: “These are common media sources. However, you may include even more social media, and not only evaluate news by institutions and organizations, but also by individual users or from the “alternative” news outlets. Influencers on Instagram who present products are another option.”

In terms of the presented misinformation, the overall judgment by the experts was positive. For instance: “Item topics are nicely varied; tasks are not too simple, so one does not get bored; and I could not decide right away, I had to click on the [background source] and take a look. Even as a media-competent person I had to examine it to make a judgment.” Another expert stated: “I could not solve the items without checking. I had heard nothing about these cases. With unknown issues, ideology also plays a smaller role.” In terms of potential biases and DIFs, the experts did not express any concerns. For instance: “I do not think that, given equal competence, it would be easier for students with typically liberal or left-wing attitudes to solve the tasks. The selection of topics in the tasks covers some stances typically accepted in the left and green camp, some typically accepted by the conservative camp.... It is a good mix.” In addition, one expert recommended expanding the item pool by a clearly untrustworthy website and one clearly trustworthy website, so that lack of trustworthiness would not be predictable on post-tests. Another expert proposed: “Some other frequently shared information of low trustworthiness can include memes, misattributed or completely wrong quotes, or quotes taken out of context.”

At the same time, however, it was questioned whether the task prompts might be too difficult for beginning or undergraduate students. For instance, “Even as a frequent evaluator, I was not always skeptical of the given information.” In this context, the appropriateness of the limited testing time was once again questioned. Only one expert was of the opinion that the tapped skills are mastered early on during the course of studies: “What you assess here is what we call study of sources. […] We teach this the first year in our degree course, and from then on, students should know it, and it is basically part of practice from then on.” In this regard, some experts recommend splitting the task into parts that focus on particular facets of COR. For instance, “You could ask for an ad hoc judgment, and have additional tasks [for more detailed search].” Another expert proposed: “If students do not find suitable sources, they may get stuck. Perhaps, it is worth including a separate task format or hints.”

Another aspect addressed by most experts concerns participants' prior knowledge, beliefs and critical stances, which may significantly influence their CORA test performance. In this context, one expert stated: “Whether people evaluate sources can also depend on their motivation to put in the time for checking them. Hence, need for cognition could be an influence, people's proclivity to get to the bottom of things and not avoid complexity.” Similarly, another expert commented: “People may also carry out a detailed search just to confirm their worldview or to form an opinion. This can occur despite existing search skills (but they would still be ideologically stuck). Hence, motivation to be open to other positions is key, and it then matters how much time I'm willing to invest in a search.” In this context, most experts stressed the need to control for participants' prior knowledge and attitudes. For instance: “Political orientation can be used as a control variable if completely anonymized, for instance asking where they would position themselves on a 1-to-10 left-to-right-wing scale (on a voluntary basis) appears less invasive.” Another expert proposed: “You may also want to specify whether it is the successful judgment of a first impression or openness to changing one's opinion. In that case, personality traits would be controlled for. So, a different option would be to assess who changes their mind when they come across new leads.”

The interviewees did not recommend the exclusion of any tasks. In few cases, the experts recommended removing certain task features. However, the experts provided a number of recommendations in terms of refining the CORA. For instance: “The role of content shared by friends could be expanded, where it is unclear if it has been checked or not... User comments can be read and might influence more passive users... So to increase difficulty, you could add social credibility cues. It would be an even more realistic setting, but you need to see how additional information would influence difficulty.” This suggestion is in line with credibility research that highlights the huge role that social persuasion by peers plays in today's social media (Fogg, 2003). Although cues exist in few of the tasks, social persuasion and learning was purposely left for future CORA expansions (Section Refining and Expanding CORA).

As task prompts shared a similar structure and wording with only differing topics and source links, the evaluative and argumentative requirements were assumed to be similar as well. The closer content analysis, however, revealed two distinct types of tasks: (1) “website evaluation tasks,” tapping particularly CIE, but less OIA if students did not search beyond the presented website; and (2) “fact-checking tasks” that only presented a claim, but no linked website as a stimulus, and therefore forced an Internet search. Fact-checking emphasized OIA more in comparison to CIE since students were not bound to evaluate one particular website; if they were uncertain about a source, they could abandon it to find a better alternative. In this way, the task types afforded use of all three facets but, respectively prioritized one in particular; consequently, a third format emphasizing REAS to complement the other two would be a further development step.

The task response sheet provides students with a clear structure, with the sections overall trustworthiness judgment, warrant (sometimes with separate pro and con sections), and URLs. The scoring rubrics did not contain any specific language requirements. Nonetheless, the students had to fill in the response sheet sections in a coherent way and formulate a conclusive statement to be awarded points. While the strands of evidence varied systematically, content-related aspects and difficulty were not systematically varied across tasks. At the current stage, given the large number of available topics and types of biases, it is still a small, to-be-expanded task pool (Section Refining and Expanding CORA).

Regarding the individual COR facets, the content analysis showed the following findings:

In terms of the activation of COR, all task prompts offered clear instructions to evaluate the trustworthiness of the sources at the beginning of the task. Mid-task activation depended on the presence of specific cues. All tasks contained at least one explicit initial and one implicit mid-task cue that might alert students to the need to use their COR. End-of-task activation, for instance a prompt to explicitly review and reflect, was not employed. Moreover, there were no tasks with implicit mid-task or end-of-task activation only, which is a characteristic of deception in online information in real life (i.e., there rarely are prior warnings that a website might contain misinformation, compared to automated warnings and filters for, e.g., malware detection). The primary aim of CORA is to measure performance during Internet searches, critical evaluation, and argumentative reasoning; it would be hardly possible to assess these facets if students missed the activation cue.

In regard to the aspect of “problem definition,” while problems were clearly stated in the task prompts, the students need to determine the evaluation criteria for trustworthiness and untrustworthiness themselves. Some experts critically noted that students may be unsure about the required evidence standards. It remains an open question whether deriving criteria for one's trustworthiness judgment should be part of the COR ability. This aspect has been scaffolded in some think-aloud studies, though we are not aware of scaffolding in other assessments. In think-aloud studies, the evaluation of criteria has been separated into different steps; for instance consecutive filtering of sources based first on relevance, then trustworthiness, then usefulness (Walraven et al., 2009; Goldman et al., 2013).

Regarding expected search skills, content analyses indicated that students can find a suitable source, and in one task a complete website review, even without specific search terms apart from the titles as long as they searched for external sources at all. Only for the fact-checking task did we find an expected larger share of relevant distracting SERP results. For selection of sources for reading, features also varied as expected. Even though some stimuli are quite short (e.g., a tweet), not all students may open the linked background article with more information. However, as this is clearly included as the main piece of evidence backing up the claim in the stimulus, students' attention to the link as a cue and to the background article can be considered a legitimate part of the tapped COR ability. An examination of the SERPs for major keywords showed significant variation in terms of the available information on the first SERP page and across tasks (see section Descriptive Features), which usually included some supporting, but also multiple irrelevant or misleading sources on the first page. Thus, the tasks appear to tap students' skills in SERP evaluation, as desired; for instance, students need to actively decide which websites to focus on.

As expected, some websites contain too much text to process in the limited time and require students to search for or skim the content. Most webpages contained more text on the landing page than fit on a single screen, and had common sub-pages, such as the “legal notice” or “about” section. For their own orientation and for a fast trustworthiness judgment, students need to gain a comprehensive overview of the websites first to be able to deliberately focus on specific sections. Some tasks also required students to recognize and understand cues outside the main text (e.g., an organization logo at the top). This indicates that simply starting to read the text might be an unsuccessful strategy on these tasks and would take too much time.

In terms of strands of evidence, cues were well distributed across tasks, in fact more regularly than expected. There were usually at least two strands of evidence with relevant cues available, so students could take different routes through the task. The linked background webpages usually contained cues that need to be understood and evaluated. Suitable information was also available in (purposefully selected) external sources to help students solve the tasks and, for instance verify the reputation of an unknown author. While tasks could be solved using just one of the available strands of evidence (e.g., only cues on author), combining two or more converging strands could potentially afford higher confidence in task response and possibly minimize effects of interpretation errors. This supports the intended interpretation of task performance.

In terms of the argumentative component of COR, students needed to make a judgment in all tasks, mostly by weighting the pros and cons, although some tasks also scaffolded these, asking for both pros and cons separately rather than a final integrated decision. These requirements can be varied more systematically based on empirical evidence regarding task difficulty. All tasks required students to find disconfirming evidence or arguments, which supports the interpretation that “critical” reasoning skills are tapped, and some tasks required students to find both confirming and disconfirming evidence or arguments. This could place students who rely only on their confirmation bias at a disadvantage, as intended. However, one expert called for the inclusion of clearly trustworthy or untrustworthy websites to better discriminate performance at the lower skill range and prevent re-testing effects (i.e., students assuming that all websites in the assessment are untrustworthy). While citing external sources is required on all tasks and is often beneficial to building an evidence-based argument, it implies a certain trade-off, as evaluating these external sources takes time and requires a higher cognitive effort. A REAS-focused task format might juxtapose several pre-selected sources with potentially contradictory information that would need to be argumentatively weighted and synthesized. Such tasks have been developed in the iPAL project (Shavelson et al., 2019; Zlatkin-Troitschanskaia et al., 2019) and MSC research (for an overview, see Braasch et al., 2018).

Task topics were varied, as expected, albeit not all societal spheres were equally covered—experts did not judge this aspect as particularly important. Sources of misinformation, including websites by associations and individuals, small enterprises, and editorial teams, were at the lower to medium level in the Hierarchy of Influences. These sources of misinformation were still mostly identifiable as individual entities within a pluralistic information environment. The CORA tasks did not focus on entities at higher levels of influence, such as media corporations or government agencies. Thus, there were no high-level “scandals” involved (as often referenced in conspiracy-related misinformation). This may imply that the highest levels of COR related to analyzing societal and funding contexts, as typically required from investigative journalists, are not tapped in the CORA tasks. This is reasonable given the time limit and the lack of content-related knowledge requirements. However, as the experts noted, the selection of topics, contexts, and genres covered in the tasks could be varied more systematically (Section Refining and Expanding CORA).

The preliminary validity evidence from content analysis and expert interviews indicated some important implications for the CORA. Overall, both the content analysis and the expert interviews indicated that it taps higher education students' abilities to search, access, critically evaluate, use, and reason based on online information in the German context, with a slightly stronger focus on the evaluation components. The preliminary evidence supports our validity claim that the CORA measures the participants' personal construct-relevant abilities in the sense of the defined construct definition (RQ3). Moreover, the expert interviews indicated that the CORA tasks cover a significant portion of the online media landscape relevant for higher education students in Germany as well as typical problems and genres of websites and online texts that call for COR skills in the German higher education context.