Scientific toxicity data, including from alternative non-animal in vitro methods should be systematically reused to the extent possible during risk assessment processes (More et al., 2021b; EFSA, 2021). This is particularly critical and urgently needed in the nanosafety community, where large amounts of complex and multidisciplinary safety data have been generated, but harmonized strategies for data reuse are currently limited (Jeliazkova et al., 2021; Ji et al., 2021). Nanosafety addresses the risks of nanomaterials (NMs), which differ markedly from traditional chemicals, since variations of physicochemical properties of NMs challenges the identification of specific features driving hazard (Fadeel et al., 2018). Even though the general health risk assessment paradigm and principles used for conventional chemicals are also applicable for NMs (OECD, 2022a), NMs risk assessment should be adapted to address the complexity associated with their identity, and their biological and environmental behaviors (Laux et al., 2018). Hence, several frameworks and tools specifically for risk assessment of NMs (in some contexts referred to as nanoforms) have been proposed (Hristozov et al., 2012; Stone et al., 2014; Bos et al., 2015; Hristozov et al., 2016b; Dekkers et al., 2016), with the aim of structured data collection and evaluation, considering their unique physicochemical properties and distinct behavior (Hristozov et al., 2016a).

However, among the available risk assessment approaches, the highly variable levels and techniques used for physicochemical and biological characterization of NMs across different studies, is considered the main bottleneck for the assessment (Laux et al., 2018). Thus, there is a need for methods that facilitate systematic evaluation of data quality and relevance to inform efficient risk assessment for regulatory purposes (e.g., decision-making relating to restriction/approval of chemicals) (Nymark et al., 2017). In the regulatory context, quality of data is commonly referred to as the reliability of data (Molander et al., 2015). Reliability entails the study’s inherent scientific quality, demonstrated by the robustness of the methods used, reproducibility of the results and adequate description of the study (OECD, 2002). Data relevance describes whether a set of data are relevant for evaluating the health risks that are being assessed, i.e., relates to a specific problem formulation (Hardy et al., 2017). In general, the evaluation of “reliability” and “relevance” of toxicity studies is an integral part of the weight-of-evidence (WoE) process in regulatory chemical risk assessment context (Hardy et al., 2017).

Nevertheless, challenges regarding the evaluation of data reliability and relevance remains in the nanosafety community (Savage et al., 2019). Firstly, harmonized data quality assessment approaches are lacking, impeding both adequate health risk assessment of NMs and the reuse of data from alternative methods for risk assessment. The lack of valid experimental protocols and models for regulatory decisions, and difficulties in identifying key physicochemical properties of NMs to predict human hazards as well as reliable Test Guidelines for use under the Mutual Acceptance of Data has hindered the use of nano-derived toxicity data (OECD, 2016). To promote the efficient use of nanotoxicity data, evaluation of data quality should be performed in a transparent and structured way. Secondly, differences in use of terminology within nanosafety research and chemical risk assessment approaches have caused confusion. In addition to data quality (reliability) and relevance, the term “completeness” is also a specific (meta)data requirement to serve the intended purpose in the nanosafety community due to the necessity of extensive physicochemical characterization of NMs before and during toxicity testing (Robinson et al., 2016). (Meta)data should be “complete”, illustrated by the adequate characterization of the test NM in both physicochemical and biological aspect. However, the definition of “completeness” overlaps with data “quality” (Robinson et al., 2016). Thus, there is a need to clarify the terminology to support risk assessment of NMs.

Science in Risk Assessment and Policy (SciRAP) is a freely available online platform (www.scirap.org) which contains different tools to evaluate the reliability and relevance of in vivo, in vitro and ecotoxicity data for hazard and risk assessment of chemicals. The tools are structured around sets of predefined criteria for evaluating reliability, based on requirements and recommendations in OECD Test Guidelines, OECD Guidance Documents (e.g., Guidance Document for Describing Non-Guideline In Vitro Test Methods–GIVIMP (OECD, 2017)), as well as previous methods such as the Klimisch method (Klimisch et al., 1997) and the ToxRtool (Schneider et al., 2009). The SciRAP tools have been rigorously tested through external review by experts and subsequently refined, providing high scientific soundness and applicability in the regulatory setting (Molander et al., 2015; Roth et al., 2021). Reliability evaluation using the SciRAP approach consists of evaluating reporting quality (RQ) and methodological quality (MQ) separately, and is also explicitly separated from evaluation of relevance. The aim is to facilitate structured and transparent use of toxicological data in regulatory risk assessment of chemicals (Molander et al., 2015). Apart from the evaluation of reliability and relevance, SciRAP provides reporting checklists for researchers with a set of items that is necessarily included in the test report for ensuring sufficient detail and transparency (Molander et al., 2015).

The MQ and RQ of an evaluated study or dataset are presented quantitatively, with a numerical score, and qualitatively with a set of color-coded Excel charts. The output of the evaluation of relevance is presented as a qualitative color profile. The SciRAP method for evaluating in vivo toxicity studies was first published in 2014 and further refined in 2018 (Molander et al., 2015; Beronius et al., 2018). Recently, the SciRAP approach was also refined to enable evaluation of data quality of in vitro toxicity and was published and available online in 2021 (Roth et al., 2021). The tool also includes criteria for evaluation of ecotoxicity studies on NMs (Hartmann et al., 2017). However, the criteria for in vitro studies are not optimized for NM-specific context due to the unique requirements on physicochemical characterization (Roth et al., 2021).

Different from previous data quality assessment approaches such as the Klimisch method (Klimisch et al., 1997) which gives higher scores to studies performed with internationally standardized methods by default, SciRAP facilitates structured evaluation for non-standard studies. This is especially useful for most NMs’ toxicity studies, since availability of standardized test guidelines suitable for NM testing are currently limited (Rasmussen et al., 2016), despite the progress made in validating methods and updating test guidelines for NMs risk assessment (OECD 2018b; OECD 2018a; Gao and Lowry, 2018; OECD, 2022b). Apart from the scarcity of “standard methods” within the nanosafety community, it has been pointed out that OECD Test Guidelines revised for toxicity testing of NMs still result in new challenges for both scientists and for quality assessment when conducting GLP (good laboratory practice) studies (Lee et al., 2021). Therefore, there is an urgent need for inclusion of non-standard studies with high quality and relevance for NMs risk assessment, indicating that SciRAP is urgently needed for the NM-specific context.

The aim of the present study was to refine the SciRAP criteria towards applicability for evaluating reliability (reporting and methodological quality) and relevance of in vitro toxicity data for NM in support of both research and risk assessment. A number of previously published efforts and reviews aimed at identifying key physicochemical parameters relevant to quality assessment of nanosafety data were utilized as a starting point for refinement of the SciRAP criteria, including documents from EU-funded projects (caLIBRAte, 2023; GRACIOUS, 2023; GUIDEnano, 2023), as well as reviews and analyses of existing standards, regulatory guidelines and other NM-specific recommendations (ECHA, 2008; ISO, 2012; Rasmussen et al., 2016; Elberskirch et al., 2022). Finally, another aim of the study was to clarify the terminology utilized within nanosafety when referring to data quality and relevance (e.g., “completeness”) in relation to the terminology used within the regulatory context.

To facilitate structured and transparent use of in vitro toxicity data for regulatory hazard and risk assessment of NMs, SciRAPnano was developed to provide a pragmatic, harmonized and user-friendly approach to evaluate data reliability and relevance for regulatory purpose. The tool was developed based on the previously established SciRAP in vitro tool, which has rigorously been tested through internal and external review (Roth et al., 2021). The SciRAPnano in vitro approach is also in accordance with both the trend of Next-Generation Risk Assessment of NMs and recommendation from regulatory authorities that non-animal alternatives should be applied whenever possible to avoid animal experimentation (EC, 2009; Browne et al., 2019).

Within the EU, efforts have been made to ensure that data are of high quality in NM’s risk assessment. For example, the German-funded DaNa 2.0 project (DaNa, 2023) created a Literature Criteria Checklist which evaluates whether NMs’ physicochemical properties and experimental details are fully described in toxicological publications. In addition, Elberskirch et al. (Elberskirch et al., 2022) created a metadata schema to establish a representation standard for nanotoxicity studies, including 300 parameters in 6 modules. These measures, to a large extent, contribute to ensuring studies with high reporting quality being included for NM’s risk assessment. However, adequate data reporting may not automatically ensure data quality, and experimental design and performance also play a vital role in determining the overall reliability and relevance of studies (Grafström et al., 2021; Jeliazkova et al., 2021; Saarimäki et al., 2021). To this end, SciRAPnano allows for a pragmatic and harmonized evaluation, considering the most critical parameters in line with the characterization requirement of nanoforms under REACH and guidance documents from EU chemical regulatory bodies. Moreover, the detailed criteria in SciRAPnano contribute to ensure that central aspects for evaluating the inherent scientific quality of the study (i.e., experimental design and study performance) are considered across all studies, regardless of compliance with standardized test guidelines, which remain limited in nanosafety community. In addition, the approach is harmonized with quality assessment of chemical toxicity studies, providing a basis for comparable quality assessments between the fields, to the extent possible.

SciRAPnano enables assessors to evaluate the data in a semi-automated and structured fashion. Easy-to-follow instructions for how to fulfill MQ criteria requirements are provided. In addition, SciRAPnano users can provide comments (e.g., the rationale of each judgement) under each criterion during data evaluation providing means for transparency. Individual evaluation reports for each evaluated study can be generated and color-coding charts and quantitative scores indicate the quality of studies. In addition to risk assessors, SciRAPnano is also useful for researchers by supporting appropriate study design and sufficient reporting, and therefore improving the usefulness of data for regulatory purposes.

The presented pragmatic approach also leaves room for flexibility (i.e., expert judgement) which is particularly important in its implementation as many aspects still lack consensus within the nanosafety community. For example, owing to the complexity of the protein corona profile and the lack of understanding regarding its effect on toxicity, it remains unclear if and how the bio-corona should be taken into account in NMs’ risk assessment (Westmeier et al., 2015; Lin et al., 2017). In SciRAPnano, the “bio-corona” aspect is covered by the aspect of transformation (RQ#16 and MQ#3) and would be evaluated on a case-by-case basis with expert judgement. Moreover, expert judgement is also needed beyond SciRAPnano in evaluating nanosafety data quality. For instance, apart from the key parameters identified in the present study, other physicochemical properties (photocatalysis, redox potential, etc.) are also potential descriptors for NM’s toxicity (Fu et al., 2014). Some toxicity determinants should only be considered specific to the type of NM or to the adverse outcomes and thus, were not prioritized in SciRAPnano (e.g., dustiness determines the toxicity of airborne NMs, but is not related to risk from other exposure routes (Donaldson and Poland, 2013)). These potential parameters were not included in the approach but should be considered and evaluated on a case-by-case scenario.

Overall it should be noted that a pragmatic approach does not necessarily refer to complete exclusion of expert judgement, but rather aims to bring transparency, consistency and structure to the process (Wandall et al., 2007; Beronius and Ågerstrand, 2017). The SciRAPnano approach ensures that data quality and relevance assessment are systematically performed while allowing for flexibility, which is particularly important for non-standard studies, e.g., during WoE processes in risk assessment (Ingre-Khans et al., 2020).

Relating to the above, it was noted at the start of this study that the evolution of terminology within the nanosafety field has caused difficulties and unclarities associated with quality and relevance assessment of data and literature from the nanotoxicology field within regulatory risk assessment (OECD, 2022a). For example, within nanosafety, a commonly used term to indicate data requirements is “completeness”, which in general refers to the necessity of adequate physicochemical characterization of NMs in toxicity testing. The term has proved difficult to fit within the chemical regulatory context, where data should be “reliable” and “relevant”, and often causes confusion as to whether it relates to data being complete enough in order to be reliable (i.e., sufficient physicochemical characterization to reliably assess toxicity in a given test system), or complete enough in order to be relevant (i.e., sufficient physicochemical characterization to relevantly address a specific problem formulation) (Robinson et al., 2016; Hardy et al., 2017). Here, the term “completeness” of (meta)data was clarified from a regulatory perspective using the WoE assessment approach as a reference, contributing to the alignment of the nanosafety community with the regulatory context. Overall, the term was strongly associated with a dependency on the questions posed (e.g., the problem formulation in a risk assessment context) and therefore, from a regulatory perspective, related to reporting quality and relevance, but not to methodological quality (cf. Table 1). However, further effective communication is still needed among regulatory agencies, industries and nanosafety experts to progress data quality and relevance assessment for risk assessment of NMs.

Here we present the newly developed SciRAPnano tool for evaluation of reliability and relevance of in vitro studies on NMs. It is based on the previously available SciRAP tool for in vitro studies and covers 38 RQ criteria (involving 15 new or adjusted criteria to evaluate key physicochemical properties, sample preparation methodology, cellular uptake, dose metrics, dispersion stability, NM’s transformation and measures taken to address the potential interference) and 19 MQ criteria (involving 4 new or adjusted criteria covering NM’s transformation, dispersant, NMs’ interference with test method and sample preparation). In addition, the tool involves four relevance items, which remain unchanged from the original SciRAP in vitro tool, except for an addition to the guidance of the first item (the identity of the test item), which was revised based on the consideration of NMs potential for transformation. Other aspects relevant to NMs, involving delivery form, viscosity, dustiness, labelling information of NMs, as well as potential contamination within the test sample, and extra substances added to the test system, are included in the “other information” sections of the RQ and MQ criteria, respectively.

Overall, SciRAPnano allows for pragmatic, harmonized and user-friendly evaluation of data reliability and relevance in order to support structured and transparent use of toxicological data within, for example, WoE processes for risk assessment of NMs. In addition to support for regulatory risk assessment of NMs, the approach can be expected to provide extensive support to the recently proposed framework aimed to guide innovation involving chemicals and materials towards becoming Safe and Sustainable by Design (SSbD) (Caldeira et al., 2022). The framework is heavily dependent on the reuse of existing data and knowledge, indicating an urgent need for harmonized and structured data quality and relevance assessment across diverse agents, including NMs (Nymark et al., 2020).