SciRAPnano: a pragmatic and harmonized approach for quality evaluation of in vitro toxicity data to support risk assessment of nanomaterials

Abstract

Large amounts of nanotoxicity data from alternative non-animal (in vitro) test methods have been generated, but there is a lack of harmonized quality evaluation approaches for these types of data. Tools for scientifically sound and structured evaluation of the reliability and relevance of in vitro toxicity data to effectively inform regulatory hazard assessment of nanomaterials (NMs), are needed. Here, we present the development of a pragmatic approach to facilitate such evaluation. The tool was developed based on the Science in Risk Assessment and Policy (SciRAP) tool currently applicable to quality evaluation of chemical toxicity studies. The approach taken to develop the tool, referred to as SciRAPnano, included refinement of the original SciRAP in vitro tool through implementation of identified NM-relevant criteria, and further refined based on a set of case studies involving evaluation of 11 studies investigating in vitro toxicity of nano-sized titanium dioxide. Parameters considered cover key physicochemical properties as well as assay-specific aspects that impact NM toxicity, including NM interference with test methods and NM transformation. The final SciRAPnano tool contains 38 criteria for reporting quality, 19 criteria for methodological quality, and 4 guidance items to evaluate relevance. The approach covers essential parameters for pragmatic and harmonized evaluation of NM in vitro toxicity studies and allows for structured use of in vitro data in regulatory hazard assessment of NMs, including transparency on data quality.

1 Introduction

Scientific toxicity data, including from alternative non-animal in vitro methods should be systematically reused to the extent possible during risk assessment processes (More et al., 2021b; EFSA, 2021). This is particularly critical and urgently needed in the nanosafety community, where large amounts of complex and multidisciplinary safety data have been generated, but harmonized strategies for data reuse are currently limited (Jeliazkova et al., 2021; Ji et al., 2021). Nanosafety addresses the risks of nanomaterials (NMs), which differ markedly from traditional chemicals, since variations of physicochemical properties of NMs challenges the identification of specific features driving hazard (Fadeel et al., 2018). Even though the general health risk assessment paradigm and principles used for conventional chemicals are also applicable for NMs (OECD, 2022a), NMs risk assessment should be adapted to address the complexity associated with their identity, and their biological and environmental behaviors (Laux et al., 2018). Hence, several frameworks and tools specifically for risk assessment of NMs (in some contexts referred to as nanoforms) have been proposed (Hristozov et al., 2012; Stone et al., 2014; Bos et al., 2015; Hristozov et al., 2016b; Dekkers et al., 2016), with the aim of structured data collection and evaluation, considering their unique physicochemical properties and distinct behavior (Hristozov et al., 2016a).

However, among the available risk assessment approaches, the highly variable levels and techniques used for physicochemical and biological characterization of NMs across different studies, is considered the main bottleneck for the assessment (Laux et al., 2018). Thus, there is a need for methods that facilitate systematic evaluation of data quality and relevance to inform efficient risk assessment for regulatory purposes (e.g., decision-making relating to restriction/approval of chemicals) (Nymark et al., 2017). In the regulatory context, quality of data is commonly referred to as the reliability of data (Molander et al., 2015). Reliability entails the study’s inherent scientific quality, demonstrated by the robustness of the methods used, reproducibility of the results and adequate description of the study (OECD, 2002). Data relevance describes whether a set of data are relevant for evaluating the health risks that are being assessed, i.e., relates to a specific problem formulation (Hardy et al., 2017). In general, the evaluation of “reliability” and “relevance” of toxicity studies is an integral part of the weight-of-evidence (WoE) process in regulatory chemical risk assessment context (Hardy et al., 2017).

Nevertheless, challenges regarding the evaluation of data reliability and relevance remains in the nanosafety community (Savage et al., 2019). Firstly, harmonized data quality assessment approaches are lacking, impeding both adequate health risk assessment of NMs and the reuse of data from alternative methods for risk assessment. The lack of valid experimental protocols and models for regulatory decisions, and difficulties in identifying key physicochemical properties of NMs to predict human hazards as well as reliable Test Guidelines for use under the Mutual Acceptance of Data has hindered the use of nano-derived toxicity data (OECD, 2016). To promote the efficient use of nanotoxicity data, evaluation of data quality should be performed in a transparent and structured way. Secondly, differences in use of terminology within nanosafety research and chemical risk assessment approaches have caused confusion. In addition to data quality (reliability) and relevance, the term “completeness” is also a specific (meta)data requirement to serve the intended purpose in the nanosafety community due to the necessity of extensive physicochemical characterization of NMs before and during toxicity testing (Robinson et al., 2016). (Meta)data should be “complete”, illustrated by the adequate characterization of the test NM in both physicochemical and biological aspect. However, the definition of “completeness” overlaps with data “quality” (Robinson et al., 2016). Thus, there is a need to clarify the terminology to support risk assessment of NMs.

Science in Risk Assessment and Policy (SciRAP) is a freely available online platform (www.scirap.org) which contains different tools to evaluate the reliability and relevance of in vivo, in vitro and ecotoxicity data for hazard and risk assessment of chemicals. The tools are structured around sets of predefined criteria for evaluating reliability, based on requirements and recommendations in OECD Test Guidelines, OECD Guidance Documents (e.g., Guidance Document for Describing Non-Guideline In Vitro Test Methods–GIVIMP (OECD, 2017)), as well as previous methods such as the Klimisch method (Klimisch et al., 1997) and the ToxRtool (Schneider et al., 2009). The SciRAP tools have been rigorously tested through external review by experts and subsequently refined, providing high scientific soundness and applicability in the regulatory setting (Molander et al., 2015; Roth et al., 2021). Reliability evaluation using the SciRAP approach consists of evaluating reporting quality (RQ) and methodological quality (MQ) separately, and is also explicitly separated from evaluation of relevance. The aim is to facilitate structured and transparent use of toxicological data in regulatory risk assessment of chemicals (Molander et al., 2015). Apart from the evaluation of reliability and relevance, SciRAP provides reporting checklists for researchers with a set of items that is necessarily included in the test report for ensuring sufficient detail and transparency (Molander et al., 2015).

The MQ and RQ of an evaluated study or dataset are presented quantitatively, with a numerical score, and qualitatively with a set of color-coded Excel charts. The output of the evaluation of relevance is presented as a qualitative color profile. The SciRAP method for evaluating in vivo toxicity studies was first published in 2014 and further refined in 2018 (Molander et al., 2015; Beronius et al., 2018). Recently, the SciRAP approach was also refined to enable evaluation of data quality of in vitro toxicity and was published and available online in 2021 (Roth et al., 2021). The tool also includes criteria for evaluation of ecotoxicity studies on NMs (Hartmann et al., 2017). However, the criteria for in vitro studies are not optimized for NM-specific context due to the unique requirements on physicochemical characterization (Roth et al., 2021).

Different from previous data quality assessment approaches such as the Klimisch method (Klimisch et al., 1997) which gives higher scores to studies performed with internationally standardized methods by default, SciRAP facilitates structured evaluation for non-standard studies. This is especially useful for most NMs’ toxicity studies, since availability of standardized test guidelines suitable for NM testing are currently limited (Rasmussen et al., 2016), despite the progress made in validating methods and updating test guidelines for NMs risk assessment (OECD 2018b; OECD 2018a; Gao and Lowry, 2018; OECD, 2022b). Apart from the scarcity of “standard methods” within the nanosafety community, it has been pointed out that OECD Test Guidelines revised for toxicity testing of NMs still result in new challenges for both scientists and for quality assessment when conducting GLP (good laboratory practice) studies (Lee et al., 2021). Therefore, there is an urgent need for inclusion of non-standard studies with high quality and relevance for NMs risk assessment, indicating that SciRAP is urgently needed for the NM-specific context.

The aim of the present study was to refine the SciRAP criteria towards applicability for evaluating reliability (reporting and methodological quality) and relevance of in vitro toxicity data for NM in support of both research and risk assessment. A number of previously published efforts and reviews aimed at identifying key physicochemical parameters relevant to quality assessment of nanosafety data were utilized as a starting point for refinement of the SciRAP criteria, including documents from EU-funded projects (caLIBRAte, 2023; GRACIOUS, 2023; GUIDEnano, 2023), as well as reviews and analyses of existing standards, regulatory guidelines and other NM-specific recommendations (ECHA, 2008; ISO, 2012; Rasmussen et al., 2016; Elberskirch et al., 2022). Finally, another aim of the study was to clarify the terminology utilized within nanosafety when referring to data quality and relevance (e.g., “completeness”) in relation to the terminology used within the regulatory context.

2 Methods and materials

2.1 Terminology

In order to clarify the diverse terminologies, evaluation of the interrelation among “reliability”, “quality”, “reporting quality”, “methodological quality”, and “completeness” was performed utilizing the terminology applied in WoE assessment approaches as a reference (Hardy et al., 2017). Details of the literature that were used to define the above terms are shown in Supplementary Table S1A1.

2.2 Literature search for the development of SciRAPnano

To develop the NM-applicable reliability assessment criteria and relevance items, several existing approaches and other guidance documents were reviewed. These materials were used to identify the critical parameters and topics that needed to be included in the NM-specific reliability and relevance evaluation approach. Details are shown in Supplementary Table S1A2.

2.2.1 Key physicochemical parameters

Diverse reports and publications from the EU-funded projects caLIBRAte (caLIBRAte), GUIDEnano (GUIDEnano), and GRACIOUS (GRACIOUS), as well as from the OECD, were used as a basis to identify key parameters to include in the current study.

The caLIBRAte quality assessment method involves the evaluation of relevance, reliability, and completeness of nanotoxicity data in the regulatory context (Nymark et al., 2017). The GUIDEnano hazard assessment strategy, developed for industrial stakeholders, provides a scoring system for evaluating the test design and reporting of NMs toxicity studies, including the physicochemical properties that need to be characterized and reported (Fernández-Cruz et al., 2018). GRACIOUS has identified the key physicochemical properties that are critical to be included in NMs’ toxicity study reporting, involving the physicochemical properties required by REACH for nanoform identification and physicochemical properties recommended by ECHA for grouping and read-across (Robinson et al., 2016). In addition, OECD WPMN has published the Physical-chemical Decision Framework to Inform Decisions for Risk Assessment of Manufactured Nanomaterials, to clarify requirements and reduce uncertainty in the applicability of testing and measurements for resolving knowledge gaps (OECD, 2019). The physicochemical properties involved in this framework were also considered to be integrated into the approach.

2.2.2 Consideration of additional physicochemical parameters

Elberskirch et al. have analyzed existing standards, regulatory guidelines (e.g., OECD Test Guidelines), and other NM-specific guidelines to create a minimum information table (MIT) which is considered necessary to be reported in nanotoxicity testing, including a total of 300 parameters (Elberskirch et al., 2022). Besides the key physicochemical parameters covered by the other central documents described above, further NM-specific physicochemical properties within the second module (material information) of the MIT were considered for inclusion in SciRAPnano.

2.2.3 NM-specific aspects of in vitro toxicity studies

Apart from the physicochemical properties, other NM-specific recommendations and considerations in designing and performing in vitro nanotoxicity studies were reviewed. In this step, guidance documents from international organizations (e.g., OECD, ISO), regulatory bodies (e.g., ECHA, EFSA), and non-regulatory academic literature were included.

2.3 Development of SciRAPnano in vitro tool v1.0

The original SciRAP criteria and items for assessment of reliability (both RQ and MQ) and relevance of in vitro studies provided the basis for developing the NM-applicable evaluation approach. Hence, the key physicochemical properties of NM and other NM-specific aspects identified in the previous step were integrated into the original SciRAP in vitro criteria. The existing SciRAP criteria were reviewed and adjusted in terms of both content and wording to be applicable to NM-data. The newly generated RQ and MQ criteria and relevance items of SciRAPnano in vitro tool version 1.0 (v1.0) are included in Supplementary Table S1.

2.4 Case study for testing SciRAPnano v1.0 and refinement to v2.0

The SciRAPnano in vitro tool v1.0 was tested and evaluated through assessment of a set of selected in vitro studies focusing on titanium dioxide (TiO₂) nanomaterials.

2.4.1 Selection of TiO₂ studies

Based on the above considerations, 11 studies were selected. Details are shown in Supplementary Table S2B1.

2.4.2 Evaluation of the selected studies

The reliability and relevance of the studies were evaluated according to the proposed v1.0 RQ and MQ criteria, and relevance items. The evaluation was done manually using Excel to consolidate the results and to further analyze the soundness of each criterion (or item) across studies. For each study, the quality criteria were judged as “fulfilled, F” “partially fulfilled, PF” “not fulfilled, NF” “not determined, ND”, whereas the relevance items were judged as “directly relevant, DR” “indirectly relevant, IR” “not relevant, NR” “not determined, ND”. Evaluation of all studies in the case study was carried out by one evaluator.

2.4.3 Refinement of SciRAPnano v1.0 into v2.0

The evaluation of the case studies supported further refinement and development of SciRAPnano v1.0 into v2.0 based on the following goals: i) to obtain understanding of the general quality of recent nanotoxicity studies, especially the level of physicochemical characterization of the selected NM, ii) to identify potential refinement needs in the practical implementation of each newly proposed SciRAP criteria and item, iii) to identify potential overlaps or interconnection among criteria, and iv) to evaluate the overall adequacy and consistency of the whole approach.

3 Results

3.1 Clarifying terminology

The interrelationship between the terms “reliability”, “relevance” and “completeness” was clarified utilizing the structure and terminology of the WoE process as an example of common terminology used within regulatory risk assessment processes (Hardy et al., 2017; SHEER, 2018). The WoE process is initiated by a problem formulation process, which is coupled to the setting of a clear definition of the minimum data required to support health risk assessment. Problem formulation is followed by three steps of evidence assembling, weighing and integration to conclude the overall assessment (Hardy et al., 2017). During weighing of evidence, reliability and relevance of each single study (single evidence) is evaluated in order to be used as a basis for the assessment. It should be noted that the problem formulation sets the basis for the evaluation of data relevance. The definitions of “reliability” and “relevance” within the WoE process are shown in Table 1. During integration of the evidence, (meta)data should be curated and integrated in line with the original minimum data required during the setting of the problem, which can be considered criteria for completeness, in order to be sufficient for the specific risk assessment scenario at hand. The definition of “completeness” as guided by the WoE process is shown in Table 1.

3.2 SciRAPnano in vitro tool v1.0

SciRAPnano v1.0 provided the draft of criteria (items) development, based on the newly identified physicochemical properties and other NM-specific aspects in experimental design, performance and reporting, as described in the following.

3.2.1 Eleven key physicochemical parameters

Eleven key physicochemical parameters were identified based on the literature review, including size, size distribution, crystallinity, shape, surface chemistry, surface area/specific surface area, surface charge, agglomeration/aggregation, stability, solubility and dissolution rate (shown in Table 2). In addition, Table 3 shows how key physicochemical properties dictate the NM’s interaction with biological system and how these parameters interlink with each other. These parameters were included among the RQ criteria within SciRAPnano. The terminology used for formulating the RQs corresponded to the eNanoMapper ontology (Hastings et al., 2015). Composition and purity were already covered by existing SciRAP criteria and were not added to the RQ. It should be noted that dustiness, density, surface hydrophobicity, free radical generation capacity, conduction band energy level, corrosivity were only mentioned once in existing minimum information checklists and thus regarded as low priority at this point.

3.2.2 Four additional physicochemical properties

3.2.3 Eight critical aspects of in vitro toxicity testing of nanomaterials

3.2.4 SciRAPnano v1.0 reliability criteria

The above parameters and aspects were formulated and integrated in the original SciRAP in vitro section, generating the first version of SciRAPnano in vitro quality assessment approach. The details about the new criteria and rewording of existing SciRAP RQ and MQ criteria are shown in Supplementary Tables S1A3, A4. RQ#5-15, MQ#2, MQ#3 MQ#10 and MQ#16 are new criteria and not present in the original SciRAP in vitro section. MQ#4, MQ#5, MQ#11, MQ#12, MQ#13 were reworded owing to the identified NM-specific aspects, based on the original SciRAP criteria. In addition, the original SciRAP MQ#2 criterion: It was likely that the test compound was soluble at the concentrations used was deleted since this is not relevant for NMs, which are not in general soluble in common solvents.

3.2.5 SciRAPnano v1.0 relevance item(s)

The four relevance items in the original SciRAP in vitro tool are i) the identity of the tested substance; ii) the test system used; iii) the endpoint studied; and iv) the concentrations used. These items were not changed for NM-specific reasons. However, since NMs may transform, the test item in toxicity studies may not be relevant to the risk assessment of the nanoform of interest. Hence, this aspect was added to the guidance of the first relevance item (the identity of the test item). Details are provided below in chapter 3.5.

3.3 Case studies

An overview of ratings for each of the 11 studies evaluated in the test round for RQ and MQ reliability criteria, and relevance items, is shown in Tables 4-6, respectively.

For RQ, the least reported parameters among all studies were “#12 water solubility of the test item”, “#13 dissolution rate of the test item” “#18 metabolic competence of the test system”, and “#22 measures taken for avoiding or screening for contamination” (Table 4). In addition, the whole “physicochemical properties of the test item” section was poorly reported, with #5 “size” being the only fully reported parameter within all 11 physicochemical properties across all studies.

For MQ, “#16 NMs physicochemical properties characterization was adequately performed at least ‘as received’, ‘as administered’ and ‘after administration or in situ’” is the least fulfilled criterion, indicating the poor characterization of physicochemical properties of the tested NM (Table 5). This finding corresponds to the inadequately reported physicochemical properties (cf. Table 4). In addition, the NM interference of the test item with the test condition (MQ#12) was not fully considered in most of the studies. Besides, due to the agglomeration of nano-TiO₂ dispersion, MQ#3 which assesses whether a stable dispersion was maintained, was scarcely fulfilled.

For the purpose of relevance evaluation, a generic problem was formulated as follows: “human health risk assessment of nano-size TiO₂ in real-life exposure level”. Items #1 and #3 were both judged as “directly relevant” for all studies which investigates the toxicity of nano-TiO₂ for different endpoints (Table 6). In contrast, many of the studies do not explain the rationale for the chosen exposure level, making it difficult to evaluate item #4 (the concentration used).

3.4 SciRAPnano in vitro tool v2.0

3.4.1 SciRAPnano reliability criteria (v2.0)

Based on the refinement needs identified in the case study, SciRAPnano in vitro tool v1.0 was refined to SciRAPnano in vitro tool v2.0, which involves 38 criteria for RQ and 19 criteria for MQ. The full lists of all criteria are shown in Table 7 and Table 8. In addition, a revised guidance for the MQ section is presented in Supplementary Table S3C1.

The overview and the detailed description of the refinement and reasons behind the refinement can be found in Supplementary Table S1. Briefly, the refinement included considerations of i) harmonization between the NM-specific criteria and the original SciRAP terminology/wording as far as possible, ii) overlaps with existing criteria, iii) a need to integrate or separate NM-specific criteria from existing criteria, and iv) clarification of NM-specific terminology and their positioning in the tool (e.g., NM-specific RQ vs. MQ, interference vs. interaction).

In total, 15 RQ criteria (#5-16, #24, #29, #31) and 3 MQ criteria (#3, #6, #14) were added based on the literature review and the case study. Further, one new MQ criterion (#2) replaced the original MQ criterion (#2) in the SciRAP in vitro tool. It should be noted that both RQ and MQ have an open criterion for evaluators to provide other aspects apart from the existing criteria (e.g., study design, performance, and reporting) that affects the overall reliability. These functions are retained in the final SciRAPnano tool and users are recommended to use the “comment” function for each criterion and to provide the rationale behind each evaluation.

3.5 SciRAPnano relevance items (v 2.0)

The SciRAPnano in vitro tool 2.0 involves 4 items for evaluating relevance. No alteration or refinement of items formulations themselves were done in NM-specific context. However, the guidance for the first item (the identity of the test item) was revised in SciRAPnano v1.0 owing to the distinct physicochemical properties of NMs, details are shown in Table 9. Due to the generic problem formulation relating to the case studies (see section 3.3) no further refinement of the relevance items was required in v2.0 of the SciRAPnano tool.

4 Discussion

To facilitate structured and transparent use of in vitro toxicity data for regulatory hazard and risk assessment of NMs, SciRAPnano was developed to provide a pragmatic, harmonized and user-friendly approach to evaluate data reliability and relevance for regulatory purpose. The tool was developed based on the previously established SciRAP in vitro tool, which has rigorously been tested through internal and external review (Roth et al., 2021). The SciRAPnano in vitro approach is also in accordance with both the trend of Next-Generation Risk Assessment of NMs and recommendation from regulatory authorities that non-animal alternatives should be applied whenever possible to avoid animal experimentation (EC, 2009; Browne et al., 2019).

Within the EU, efforts have been made to ensure that data are of high quality in NM’s risk assessment. For example, the German-funded DaNa 2.0 project (DaNa, 2023) created a Literature Criteria Checklist which evaluates whether NMs’ physicochemical properties and experimental details are fully described in toxicological publications. In addition, Elberskirch et al. (Elberskirch et al., 2022) created a metadata schema to establish a representation standard for nanotoxicity studies, including 300 parameters in 6 modules. These measures, to a large extent, contribute to ensuring studies with high reporting quality being included for NM’s risk assessment. However, adequate data reporting may not automatically ensure data quality, and experimental design and performance also play a vital role in determining the overall reliability and relevance of studies (Grafström et al., 2021; Jeliazkova et al., 2021; Saarimäki et al., 2021). To this end, SciRAPnano allows for a pragmatic and harmonized evaluation, considering the most critical parameters in line with the characterization requirement of nanoforms under REACH and guidance documents from EU chemical regulatory bodies. Moreover, the detailed criteria in SciRAPnano contribute to ensure that central aspects for evaluating the inherent scientific quality of the study (i.e., experimental design and study performance) are considered across all studies, regardless of compliance with standardized test guidelines, which remain limited in nanosafety community. In addition, the approach is harmonized with quality assessment of chemical toxicity studies, providing a basis for comparable quality assessments between the fields, to the extent possible.

SciRAPnano enables assessors to evaluate the data in a semi-automated and structured fashion. Easy-to-follow instructions for how to fulfill MQ criteria requirements are provided. In addition, SciRAPnano users can provide comments (e.g., the rationale of each judgement) under each criterion during data evaluation providing means for transparency. Individual evaluation reports for each evaluated study can be generated and color-coding charts and quantitative scores indicate the quality of studies. In addition to risk assessors, SciRAPnano is also useful for researchers by supporting appropriate study design and sufficient reporting, and therefore improving the usefulness of data for regulatory purposes.

The presented pragmatic approach also leaves room for flexibility (i.e., expert judgement) which is particularly important in its implementation as many aspects still lack consensus within the nanosafety community. For example, owing to the complexity of the protein corona profile and the lack of understanding regarding its effect on toxicity, it remains unclear if and how the bio-corona should be taken into account in NMs’ risk assessment (Westmeier et al., 2015; Lin et al., 2017). In SciRAPnano, the “bio-corona” aspect is covered by the aspect of transformation (RQ#16 and MQ#3) and would be evaluated on a case-by-case basis with expert judgement. Moreover, expert judgement is also needed beyond SciRAPnano in evaluating nanosafety data quality. For instance, apart from the key parameters identified in the present study, other physicochemical properties (photocatalysis, redox potential, etc.) are also potential descriptors for NM’s toxicity (Fu et al., 2014). Some toxicity determinants should only be considered specific to the type of NM or to the adverse outcomes and thus, were not prioritized in SciRAPnano (e.g., dustiness determines the toxicity of airborne NMs, but is not related to risk from other exposure routes (Donaldson and Poland, 2013)). These potential parameters were not included in the approach but should be considered and evaluated on a case-by-case scenario.

Overall it should be noted that a pragmatic approach does not necessarily refer to complete exclusion of expert judgement, but rather aims to bring transparency, consistency and structure to the process (Wandall et al., 2007; Beronius and Ågerstrand, 2017). The SciRAPnano approach ensures that data quality and relevance assessment are systematically performed while allowing for flexibility, which is particularly important for non-standard studies, e.g., during WoE processes in risk assessment (Ingre-Khans et al., 2020).

Relating to the above, it was noted at the start of this study that the evolution of terminology within the nanosafety field has caused difficulties and unclarities associated with quality and relevance assessment of data and literature from the nanotoxicology field within regulatory risk assessment (OECD, 2022a). For example, within nanosafety, a commonly used term to indicate data requirements is “completeness”, which in general refers to the necessity of adequate physicochemical characterization of NMs in toxicity testing. The term has proved difficult to fit within the chemical regulatory context, where data should be “reliable” and “relevant”, and often causes confusion as to whether it relates to data being complete enough in order to be reliable (i.e., sufficient physicochemical characterization to reliably assess toxicity in a given test system), or complete enough in order to be relevant (i.e., sufficient physicochemical characterization to relevantly address a specific problem formulation) (Robinson et al., 2016; Hardy et al., 2017). Here, the term “completeness” of (meta)data was clarified from a regulatory perspective using the WoE assessment approach as a reference, contributing to the alignment of the nanosafety community with the regulatory context. Overall, the term was strongly associated with a dependency on the questions posed (e.g., the problem formulation in a risk assessment context) and therefore, from a regulatory perspective, related to reporting quality and relevance, but not to methodological quality (cf. Table 1). However, further effective communication is still needed among regulatory agencies, industries and nanosafety experts to progress data quality and relevance assessment for risk assessment of NMs.

5 Conclusion

Here we present the newly developed SciRAPnano tool for evaluation of reliability and relevance of in vitro studies on NMs. It is based on the previously available SciRAP tool for in vitro studies and covers 38 RQ criteria (involving 15 new or adjusted criteria to evaluate key physicochemical properties, sample preparation methodology, cellular uptake, dose metrics, dispersion stability, NM’s transformation and measures taken to address the potential interference) and 19 MQ criteria (involving 4 new or adjusted criteria covering NM’s transformation, dispersant, NMs’ interference with test method and sample preparation). In addition, the tool involves four relevance items, which remain unchanged from the original SciRAP in vitro tool, except for an addition to the guidance of the first item (the identity of the test item), which was revised based on the consideration of NMs potential for transformation. Other aspects relevant to NMs, involving delivery form, viscosity, dustiness, labelling information of NMs, as well as potential contamination within the test sample, and extra substances added to the test system, are included in the “other information” sections of the RQ and MQ criteria, respectively.

Overall, SciRAPnano allows for pragmatic, harmonized and user-friendly evaluation of data reliability and relevance in order to support structured and transparent use of toxicological data within, for example, WoE processes for risk assessment of NMs. In addition to support for regulatory risk assessment of NMs, the approach can be expected to provide extensive support to the recently proposed framework aimed to guide innovation involving chemicals and materials towards becoming Safe and Sustainable by Design (SSbD) (Caldeira et al., 2022). The framework is heavily dependent on the reuse of existing data and knowledge, indicating an urgent need for harmonized and structured data quality and relevance assessment across diverse agents, including NMs (Nymark et al., 2020).

References

• 1

  BeroniusA.ÅgerstrandM. (2017). Making the most of expert judgment in hazard and risk assessment of chemicals. Toxicol. Res. (Camb)6, 571–577. 10.1039/c7tx00114b

  • CrossRef
  • Google Scholar

• 2

  BeroniusA.MolanderL.ZilliacusJ.RudénC.HanbergA. (2018). Testing and refining the science in risk assessment and policy (scirap) web-based platform for evaluating the reliability and relevance of in vivo toxicity studies. J. Appl. Toxicol.38, 1460–1470. 10.1002/jat.3648

  • CrossRef
  • Google Scholar

• 3

  BosP. M. J.GottardoS.Scott-FordsmandJ. J.Van TongerenM.SemenzinE.FernandesT. F.et al (2015). The marina risk assessment strategy: a flexible strategy for efficient information collection and risk assessment of nanomaterials. Collect. Risk Assess. Nanomater.12, 15007–15021. 10.3390/ijerph121214961

  • CrossRef
  • Google Scholar

• 4

  BrowneP.DelrueN.GourmelonA. (2019). Regulatory use and acceptance of alternative methods for chemical hazard identification. Curr. Opin. Toxicol.15, 18–25. 10.1016/j.cotox.2019.02.003

  • CrossRef
  • Google Scholar

• 5

  CaldeiraC.FarcalR.MorettiC.ManciniL.RauscherH.RasmussenK.et al (2022). Safe and sustainable by design chemicals and materials: review of safety and sustainability dimensions, aspects, methods, indicators, and tools EC joint research centre.

  • Google Scholar

• 6

  caLIBRAte (2023). Available: https://www.nanosafetycluster.eu/nsc-overview/nsc-structure/steering-group/calibrate/.

  • Google Scholar

• 7

  ComandellaD.GottardoS.Rio-EchevarriaI. M.RauscherH. (2020). Quality of physicochemical data on nanomaterials: an assessment of data completeness and variability. Nanoscale12, 4695–4708. 10.1039/C9NR08323E

  • CrossRef
  • Google Scholar

• 8

  DaNa (2023). Lit. Criteria Checkl. – Wissensplattform Nanopartikel.Info. Available at: https://nanopartikel.info/en/knowledge/literature-criteria-checklist/.

  • Google Scholar

• 9

  DekkersS.OomenA. G.BleekerE. A. J.VandebrielR. J.MichelettiC.CabellosJ.et al (2016). Towards a nanospecific approach for risk assessment. Regul. Toxicol. Pharmacol.80, 46–59. 10.1016/j.yrtph.2016.05.037

  • CrossRef
  • Google Scholar

• 10

  DonaldsonK.PolandC. (2013). Nanotoxicity: challenging the myth of nano-specific toxicity. Curr. Opin. Biotechnol.24, 724–734. 10.1016/j.copbio.2013.05.003

  • CrossRef
  • Google Scholar

• 11

  EC (2009). Regulation (ec) No 1223/2009 of the European parliament and of the council of 30 november 2009 on cosmetic products (recast) (text with eea relevance), 342. OJ L.

  • Google Scholar

• 12

  EC (2018). Commission regulation (Eu) 2018/1881 of 3 december 2018 amending regulation (ec) No 1907/2006 of the European parliament and of the council on the registration, evaluation, authorisation and restriction of chemicals (reach) as regards annexes I, Iii,Vi, vii, viii, ix, X, xi, and xii to address nanoforms of substances (text with eea relevance.), 308. OJ L.

  • Google Scholar

• 13

  ECHA (2008). Guidance on information requirements and chemical safety assessment. Helsinki: European Chemicals Agency. Available at: https://echa.europa.eu/guidance-documents/guidance-on-information-requirements-and-chemical-safety-assessment

  • Google Scholar

• 14

  EFSA (2021). Nanomaterials in reach and clp - environment - European commission.

  • Google Scholar

• 15

  ElberskirchL.BinderK.RieflerN.SofrankoA.LiebingJ.MinellaC. B.et al (2022). Digital research data: from analysis of existing standards to a scientific foundation for a modular metadata schema in nanosafety. Part Fibre Toxicol.19, 1. 10.1186/s12989-021-00442-x

  • CrossRef
  • Google Scholar

• 16

  FadeelB.FarcalL.HardyB.Vázquez-CamposS.HristozovD.MarcominiA.et al (2018). Advanced tools for the safety assessment of nanomaterials. Nat. Nanotechnol.13, 537–543. 10.1038/s41565-018-0185-0

  • CrossRef
  • Google Scholar

• 17

  Fernández-CruzM. L.Hernández-MorenoD.CatalánJ.CrossR. K.Stockmann-JuvalaH.CabellosJ.et al (2018). Quality evaluation of human and environmental toxicity studies performed with nanomaterials – the guidenano approach. Environ. Sci. Nano5, 381–397. 10.1039/C7EN00716G

  • CrossRef
  • Google Scholar

• 18

  FröhlichE. (2012). The role of surface charge in cellular uptake and cytotoxicity of medical nanoparticles. Int. J. Nanomedicine7, 5577–5591. 10.2147/IJN.S36111

  • CrossRef
  • Google Scholar

• 19

  FuP. P.XiaQ.HwangH.-M.RayP. C.YuH. (2014). Mechanisms of nanotoxicity: generation of reactive oxygen species. J. Food Drug Analysis22, 64–75. 10.1016/j.jfda.2014.01.005

  • CrossRef
  • Google Scholar

• 20

  GaoX.LowryG. V. (2018). Progress towards standardized and validated characterizations for measuring physicochemical properties of manufactured nanomaterials relevant to nano health and safety risks. NanoImpact9, 14–30. 10.1016/j.impact.2017.09.002

  • CrossRef
  • Google Scholar

• 21

  GRACIOUS (2023). Available: https://www.nanosafetycluster.eu/nsc-overview/nsc-structure/steering-group/gracious/.

  • Google Scholar

• 22

  GrafströmR.HaaseA.KohonenP.JeliazkovaN.NymarkP. (2021). Reply to: prospects and challenges for fair toxicogenomics data. Nat. Nanotechnol.17, 19–20. 10.1038/s41565-021-01050-8

  • CrossRef
  • Google Scholar

• 23

  GUIDEnano (2023). Available: https://www.guidenano.eu/.

  • Google Scholar

• 24

  HardyA.BenfordD.HalldorssonT.JegerM. J.KnutsenH. K.MoreS.et al (2017). Guidance on the use of the weight of evidence approach in scientific assessments. EFSA J.15, e04971. 10.2903/j.efsa.2017.4971

  • CrossRef
  • Google Scholar

• 25

  HartmannN. B.ÅgerstrandM.LützhøftH.-C. H.BaunA. (2017). Nanocred: a transparent framework to assess the regulatory adequacy of ecotoxicity data for nanomaterials – relevance and reliability revisited. NanoImpact6, 81–89. 10.1016/j.impact.2017.03.004

  • CrossRef
  • Google Scholar

• 26

  HastingsJ.JeliazkovaN.OwenG.TsilikiG.MunteanuC. R.SteinbeckC.et al (2015). Enanomapper: harnessing ontologies to enable data integration for nanomaterial risk assessment. J. Biomed. Semant.6, 10. 10.1186/s13326-015-0005-5

  • CrossRef
  • Google Scholar

• 27

  HristozovD.GottardoS.SemenzinE.OomenA.BosP.PeijnenburgW.et al (2016a). Frameworks and tools for risk assessment of manufactured nanomaterials. Environ. Int.95, 36–53. 10.1016/j.envint.2016.07.016

  • CrossRef
  • Google Scholar

• 28

  HristozovD.ZabeoA.Alstrup JensenK.GottardoS.IsigonisP.MaccalmanL.et al (2016b). Demonstration of a modelling-based multi-criteria decision analysis procedure for prioritisation of occupational risks from manufactured nanomaterials. Nanotoxicology10, 1215–1228. 10.3109/17435390.2016.1144827

  • CrossRef
  • Google Scholar

• 29

  HristozovD. R.GottardoS.CrittoA.MarcominiA. (2012). Risk assessment of engineered nanomaterials: a review of available data and approaches from a regulatory perspective. Nanotoxicology6, 880–898. 10.3109/17435390.2011.626534

  • CrossRef
  • Google Scholar

• 30

  Ingre-KhansE.ÅgerstrandM.RudénC.BeroniusA. (2020). Improving structure and transparency in reliability evaluations of data under REACH: suggestions for a systematic method. an Int. J.26, 212–241. 10.1080/10807039.2018.1504275

  • CrossRef
  • Google Scholar

• 31

  ISO (2012). Iso/tr 13014:2012(en), nanotechnologies — guidance on physico-chemical characterization of engineered nanoscale materials for toxicologic assessment.

  • Google Scholar

• 32

  JeliazkovaN.ApostolovaM. D.AndreoliC.BaroneF.BarrickA.BattistelliC.et al (2021). Towards fair nanosafety data. Nat. Nanotechnol.16, 644–654. 10.1038/s41565-021-00911-6

  • CrossRef
  • Google Scholar

• 33

  JiZ.GuoW.SakkiahS.LiuJ.PattersonT. A.HongH. (2021). Nanomaterial databases: data sources for promoting design and risk assessment of nanomaterials. Nanomaterials11, 1599. 10.3390/nano11061599

  • CrossRef
  • Google Scholar

• 34

  JiangJ.OberdörsterG.ElderA.GeleinR.MercerP.BiswasP. (2008). Does nanoparticle activity depend upon size and crystal phase?Nanotoxicology2, 33–42. 10.1080/17435390701882478

  • CrossRef
  • Google Scholar

• 35

  KhanI.SaeedK.KhanI. (2019). Nanoparticles: properties, applications and toxicities. Arabian J. Chem.12, 908–931. 10.1016/j.arabjc.2017.05.011

  • CrossRef
  • Google Scholar

• 36

  KlimischH. J.AndreaeM.TillmannU. (1997). A systematic approach for evaluating the quality of experimental toxicological and ecotoxicological data. Regul. Toxicol. Pharmacol.25, 1–5. 10.1006/rtph.1996.1076

  • CrossRef
  • Google Scholar

• 37

  LauxP.TentschertJ.RiebelingC.BraeuningA.CreutzenbergO.EppA.et al (2018). Nanomaterials: certain aspects of application, risk assessment and risk communication. Arch. Toxicol.92, 121–141. 10.1007/s00204-017-2144-1

  • CrossRef
  • Google Scholar

• 38

  LeeS. K.JoM. S.KimH. P.KimJ. C.YuI. J. (2021). Quality assurance for nanomaterial inhalation toxicity testing. Inhal. Toxicol.33, 161–167. 10.1080/08958378.2021.1926602

  • CrossRef
  • Google Scholar

• 39

  LinS.MortimerM.ChenR.KakinenA.RiviereJ. E.DavisT. P.et al (2017). Nanoehs beyond toxicity – focusing on biocorona. Environ. Sci. Nano4, 1433–1454. 10.1039/C6EN00579A

  • CrossRef
  • Google Scholar

• 40

  MolanderL.ÅgerstrandM.BeroniusA.HanbergA.RudénC. (2015). Science in risk assessment and policy (scirap): an online resource for evaluating and reporting in vivo (Eco)Toxicity studies. Hum. Ecol. Risk Assess. an Int. J.21, 753–762. 10.1080/10807039.2014.928104

  • CrossRef
  • Google Scholar

• 41

  MoreS.BampidisV.BenfordD.BragardC.HalldorssonT.Hernández-JerezA.et al (2021a). Guidance on technical requirements for regulated food and feed product applications to establish the presence of small particles including nanoparticles. EFSA J.19, e06769. 10.2903/j.efsa.2021.6769

  • CrossRef
  • Google Scholar

• 42

  MoreS.BampidisV.BenfordD.BragardC.HalldorssonT.Hernández-JerezA.et al (2021b). Guidance on risk assessment of nanomaterials to Be applied in the food and feed chain: human and animal health. EFSA J.19, e06768. 10.2903/j.efsa.2021.6768

  • CrossRef
  • Google Scholar

• 43

  NelA.XiaT.MädlerL.LiN. (2006). Toxic potential of materials at the nanolevel, 311. New York, NY: Science, 622–627. 10.1126/science.1114397

  • CrossRef
  • Google Scholar

• 44

  NelA. E.MädlerL.VelegolD.XiaT.HoekE. M. V.SomasundaranP.et al (2009). Understanding biophysicochemical interactions at the nano–bio interface. Nat. Mater8, 543–557. 10.1038/nmat2442

  • CrossRef
  • Google Scholar

• 45

  NymarkP.BakkerM.DekkersS.FrankenR.FransmanW.García-BilbaoA.et al (2020). Toward rigorous materials production: new approach methodologies have extensive potential to improve current safety assessment practices. Small16, 1904749. 10.1002/smll.201904749

  • CrossRef
  • Google Scholar

• 46

  NymarkP.GrafströmR.NoorlanderC.CatalánJ.Rodriguez-LlopisI.Suárez-MerinoB.et al (2017). Calibrate D5.3 - document on quality criteria for data. 10.5281/zenodo.3859951

  • CrossRef
  • Google Scholar

• 47

  OECD (2016). Future challenges related to the safety of manufactured nanomaterials: report from the special session, 75. Paris:Organisation for Economic Co-operation and Development. Available at: https://one.oecd.org/document/ENV/JM/MONO(2016)58/en/pdf.

  • Google Scholar

• 48

  OECD (2017). Guidance document for describing non-guideline in vitro test methods. Paris:Organisation for Economic Co-operation and Development. Available at: https://www.oecd.org/publications/guidance-document-for-describing-non-guideline-in-vitro-test-methods-9789264274730-en.htm.

  • Google Scholar

• 49

  OECD (2018a). Test No. 412: subacute inhalation toxicity: 28-day study. Paris: Organisation for Economic Co-operation and Development.

  • Google Scholar

• 50

  OECD (2018b). Test No. 413: subchronic inhalation toxicity: 90-day study. Paris: Organisation for Economic Co-operation and Development.

  • Google Scholar

• 51

  OECD (2019). Physical-chemical decision framework to inform decisions for risk assessment of manufactured nanomaterials, 90. Paris:Organisation for Economic Co-operation and Development. Available at: https://one.oecd.org/document/env/jm/mono(2019)12/en/pdf.

  • Google Scholar

• 52

  OECD (2022a). Important issues on risk assessment of manufactured nanomaterials, 103. Paris:Organisation for Economic Co-operation and Development.

  • Google Scholar

• 53

  OECD (2022b). Test No. 125: nanomaterial particle size and size distribution of nanomaterials. Paris: Organisation for Economic Co-operation and Development.

  • Google Scholar

• 54

  OECD (2002). Guidance notes for analysis and evaluation of repeat-dose toxicity studies. Paris: Organisation for Economic Co-operation and Development.

  • Google Scholar

• 55

  PalR.WangL.-M.PeiY.WangL.-S.ZengX. C. (2012). Unraveling the mechanisms of O2 activation by size-selected gold clusters: transition from superoxo to peroxo chemisorption. J. Am. Chem. Soc.134, 9438–9445. 10.1021/ja302902p

  • CrossRef
  • Google Scholar

• 56

  PalomäkiJ.VälimäkiE.SundJ.VippolaM.ClausenP. A.JensenK. A.et al (2011). Long, needle-like carbon nanotubes and asbestos activate the Nlrp3 inflammasome through a similar mechanism. ACS Nano5, 6861–6870. 10.1021/nn200595c

  • CrossRef
  • Google Scholar

• 57

  RasmussenK.GonzálezM.KearnsP.SintesJ. R.RossiF.SayreP. (2016). Review of achievements of the oecd working party on manufactured nanomaterials' testing and assessment programme. From exploratory testing to test guidelines. Regul. Toxicol. Pharmacol. RTP74, 147–160. 10.1016/j.yrtph.2015.11.004

  • CrossRef
  • Google Scholar

• 58

  RobinsonR. L. M.LynchI.PeijnenburgW.RumbleJ.KlaessigF.MarquardtC.et al (2016). How should the completeness and quality of curated nanomaterial data Be evaluated?Nanoscale8, 9919–9943. 10.1039/C5NR08944A

  • CrossRef
  • Google Scholar

• 59

  RothN.ZilliacusJ.BeroniusA. (2021). Development of the scirap approach for evaluating the reliability and relevance of in vitro toxicity data, 3. 10.3389/ftox.2021.746430

  • CrossRef
  • Google Scholar

• 60

  SaarimäkiL. A.MelagrakiG.AfantitisA.LynchI.GrecoD. (2021). Prospects and challenges for fair toxicogenomics data. Nat. Nanotechnol.17, 17–18. 10.1038/s41565-021-01049-1

  • CrossRef
  • Google Scholar

• 61

  SavageD. T.HiltJ. Z.DziublaT. D. (2019). Methods in molecular biology. Clifton, NJ, 1–29. 10.1007/978-1-4939-8916-4_1Vitro methods for assessing nanoparticle toxicity

  • CrossRef
  • Google Scholar

• 62

  SchneiderK.SchwarzM.BurkholderI.Kopp-SchneiderA.EdlerL.Kinsner-OvaskainenA.et al (2009). "ToxRTool", a new tool to assess the reliability of toxicological data. a New Tool Assess Reliab. Toxicol. Data. Toxicol. Lett.189, 138–144. 10.1016/j.toxlet.2009.05.013

  • CrossRef
  • Google Scholar

• 63

  ScownT. M.SantosE. M.JohnstonB. D.GaiserB.BaaloushaM.MitovS.et al (2010). Effects of aqueous exposure to silver nanoparticles of different sizes in rainbow trout. Toxicol. Sci.115, 521–534. 10.1093/toxsci/kfq076

  • CrossRef
  • Google Scholar

• 64

  SelvamaniV. (2019). “Chapter 15 - stability studies on nanomaterials used in drugs,” in Characterization and biology of nanomaterials for drug delivery. Editors MohapatraS. S.RanjanS.DasguptaN.MishraR. K.ThomasS. (Elsevier), 425–444.

  • Google Scholar

• 65

  SHEER (2018). Memorandum on weight of evidence and uncertainties - revision 2018. 10.2875/386011

  • CrossRef
  • Google Scholar

• 66

  StoneV.Pozzi-MucelliS.TranL.AschbergerK.SabellaS.VogelU.et al (2014). Its-nano - prioritising nanosafety research to develop a stakeholder driven intelligent testing strategy. Part Fibre Toxicol.11, 9. 10.1186/1743-8977-11-9

  • CrossRef
  • Google Scholar

• 67

  WandallB.HanssonS. O.RudénC. (2007). Bias in Toxicology. Arch. Toxicol.81, 605–617. 10.1007/s00204-007-0194-5

  • CrossRef
  • Google Scholar

• 68

  WangX.CuiX.ZhaoY.ChenC. (2020). Nano-bio interactions: the implication of size-dependent biological effects of nanomaterials. Sci. China Life Sci.63, 1168–1182. 10.1007/s11427-020-1725-0

  • CrossRef
  • Google Scholar

• 69

  WestmeierD.ChenC.StauberR. H.DocterD. (2015). The bio-corona and its impact on nanomaterial toxicity. Eur. J. Nanomedicine7, 153–168. 10.1515/ejnm-2015-0018

  • CrossRef
  • Google Scholar

• 70

  YinH.ChenR.CaseyP. S.KeP. C.DavisT. P.ChenC. (2015). Reducing the cytotoxicity of zno nanoparticles by a pre-formed protein corona in a supplemented cell culture medium. RSC Adv.5, 73963–73973. 10.1039/C5RA14870G

  • CrossRef
  • Google Scholar