<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article article-type="research-article" dtd-version="2.3" xml:lang="EN" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">Front. Pharmacol.</journal-id>
<journal-title>Frontiers in Pharmacology</journal-title>
<abbrev-journal-title abbrev-type="pubmed">Front. Pharmacol.</abbrev-journal-title>
<issn pub-type="epub">1663-9812</issn>
<publisher>
<publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="publisher-id">708050</article-id>
<article-id pub-id-type="doi">10.3389/fphar.2021.708050</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Pharmacology</subject>
<subj-group>
<subject>Original Research</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group>
<article-title>A Comparison of Nine Machine Learning Mutagenicity Models and Their Application for Predicting Pyrrolizidine Alkaloids</article-title>
<alt-title alt-title-type="left-running-head">Helma et&#x20;al.</alt-title>
<alt-title alt-title-type="right-running-head">Prediction of Pyrrolizidine Alkaloid Mutagenicity</alt-title>
</title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes">
<name>
<surname>Helma</surname>
<given-names>Christoph</given-names>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
<xref ref-type="corresp" rid="c001">&#x2a;</xref>
<uri xlink:href="https://loop.frontiersin.org/people/63846/overview"/>
</contrib>
<contrib contrib-type="author">
<name>
<surname>Sch&#xf6;ning</surname>
<given-names>Verena</given-names>
</name>
<xref ref-type="aff" rid="aff2">
<sup>2</sup>
</xref>
<uri xlink:href="https://loop.frontiersin.org/people/1132621/overview"/>
</contrib>
<contrib contrib-type="author" corresp="yes">
<name>
<surname>Drewe</surname>
<given-names>J&#xfc;rgen</given-names>
</name>
<xref ref-type="aff" rid="aff3">
<sup>3</sup>
</xref>
<xref ref-type="aff" rid="aff4">
<sup>4</sup>
</xref>
<xref ref-type="corresp" rid="c001">&#x2a;</xref>
<uri xlink:href="https://loop.frontiersin.org/people/717792/overview"/>
</contrib>
<contrib contrib-type="author">
<name>
<surname>Boss</surname>
<given-names>Philipp</given-names>
</name>
<xref ref-type="aff" rid="aff5">
<sup>5</sup>
</xref>
</contrib>
</contrib-group>
<aff id="aff1">
<label>
<sup>1</sup>
</label>In Silico Toxicology Gmbh, <addr-line>Basel</addr-line>, <country>Switzerland</country>
</aff>
<aff id="aff2">
<label>
<sup>2</sup>
</label>Clinical Pharmacology and Toxicology, Department of General Internal Medicine, University Hospital Bern, University of Bern, Inselspital, <addr-line>Bern</addr-line>, <country>Switzerland</country>
</aff>
<aff id="aff3">
<label>
<sup>3</sup>
</label>Max Zeller S&#xf6;hne AG, <addr-line>Romanshorn</addr-line>, <country>Switzerland</country>
</aff>
<aff id="aff4">
<label>
<sup>4</sup>
</label>Department of Clinical Pharmacology, University Hospital Basel, University of Basel, <addr-line>Basel</addr-line>, <country>Switzerland</country>
</aff>
<aff id="aff5">
<label>
<sup>5</sup>
</label>Berlin Institute for Medical Systems Biology, Max Delbr&#xfc;ck Center for Molecular Medicine in the Helmholtz Association, <addr-line>Berlin</addr-line>, <country>Germany</country>
</aff>
<author-notes>
<fn fn-type="edited-by">
<p>
<bold>Edited by:</bold> <ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/680622/overview">Monika Batke</ext-link>, Fraunhofer Institute for Toxicology and Experimental Medicine (FHG), Germany</p>
</fn>
<fn fn-type="edited-by">
<p>
<bold>Reviewed by:</bold> <ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/31350/overview">Emilio Benfenati</ext-link>, Istituto di Ricerche Farmacologiche Mario Negri (IRCCS), Italy</p>
<p>
<ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/1068318/overview">Birgit Mertens</ext-link>, Sciensano, Belgium</p>
</fn>
<corresp id="c001">&#x2a;Correspondence: Christoph Helma, <email>helma@in-silico.ch</email>; J&#xfc;rgen Drewe, <email>juergen.drewe@zellerag.ch</email>
</corresp>
<fn fn-type="other">
<p>This article was submitted to Predictive Toxicology, a section of the journal Frontiers in Pharmacology</p>
</fn>
</author-notes>
<pub-date pub-type="epub">
<day>22</day>
<month>07</month>
<year>2021</year>
</pub-date>
<pub-date pub-type="collection">
<year>2021</year>
</pub-date>
<volume>12</volume>
<elocation-id>708050</elocation-id>
<history>
<date date-type="received">
<day>11</day>
<month>05</month>
<year>2021</year>
</date>
<date date-type="accepted">
<day>07</day>
<month>07</month>
<year>2021</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright &#xa9; 2021 Helma, Sch&#xf6;ning, Drewe and Boss.</copyright-statement>
<copyright-year>2021</copyright-year>
<copyright-holder>Helma, Sch&#xf6;ning, Drewe and Boss</copyright-holder>
<license xlink:href="http://creativecommons.org/licenses/by/4.0/">
<p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these&#x20;terms.</p>
</license>
</permissions>
<abstract>
<p>Random forest, support vector machine, logistic regression, neural networks and k-nearest neighbor (lazar) algorithms, were applied to a new <italic>Salmonella</italic> mutagenicity dataset with 8,290 unique chemical structures utilizing MolPrint2D and Chemistry Development Kit (CDK) descriptors. Crossvalidation accuracies of all investigated models ranged from 80 to 85% which is comparable with the interlaboratory variability of the <italic>Salmonella</italic> mutagenicity assay. Pyrrolizidine alkaloid predictions showed a clear distinction between chemical groups, where otonecines had the highest proportion of positive mutagenicity predictions and monoesters the lowest.</p>
</abstract>
<kwd-group>
<kwd>mutagenicity</kwd>
<kwd>lazar</kwd>
<kwd>openbabel</kwd>
<kwd>CDK</kwd>
<kwd>machine learning</kwd>
<kwd>tensorflow</kwd>
<kwd>pyrrolizidine alkaloids</kwd>
</kwd-group>
</article-meta>
</front>
<body>
<sec id="s1">
<title>1 Introduction</title>
<p>The assessment of mutagenicity is an important part in the safety assessment of chemical structures, because mutations may lead to cancer and germ cells damage. The bacterial reverse mutation test (Ames test) is capable to identify substances that cause mutations (e.g., base-pair substitutions, frameshifts, insertions, deletions) and is generally used as the first step in genotoxicity and carcinogenicity assessments.</p>
<p>Computer based (<italic>in silico</italic>) mutagenicity predictions can be used in the early screening of novel compounds (e.g., drug candidates), but they are also gaining regulatory acceptance e.g. for the registration of industrial chemicals within REACH (<xref ref-type="bibr" rid="B7">European Chemical Agency, 2017</xref>) or the assessment of impurities in pharmaceuticals (<xref ref-type="bibr" rid="B14">ICH, 2017</xref>).</p>
<p>Currently, mutagenicity is the toxicological endpoint with the largest amount of public data for almost 10000 structures, whereas datasets for other endpoints contain typically only a few hundred compounds. The Ames test itself is relatively reproducible with an interlaboratory variability of 80&#x2013;85% (<xref ref-type="bibr" rid="B24">Piegorsch and Zeiger, 1991</xref>).</p>
<p>This makes the development of mutagenicity models also interesting from a computational chemistry and machine learning point of view. The relatively large amount of public data reduces the probability of chance effects due to small sample sizes and the reliability of the underlying assay reduces the risk of overfitting experimental errors.</p>
<p>Within this study we attempted:<list list-type="simple">
<list-item>
<p>&#x2022; to generate a new public mutagenicity training dataset focusing on <italic>Salmonella typhimurium</italic>, by combining the most comprehensive public datasets</p>
</list-item>
<list-item>
<p>&#x2022; to compare the performance of MolPrint2D (<italic>MP2D</italic>) fingerprints with Chemistry Development Kit (<italic>CDK</italic>) descriptors for mutagenicity predictions</p>
</list-item>
<list-item>
<p>&#x2022; to compare the performance of global QSAR models (random forests (<italic>RF</italic>), support vector machines (<italic>SVM</italic>), logistic regression (<italic>LR</italic>), neural nets (<italic>NN</italic>) with local models (lazar)</p>
</list-item>
</list>
</p>
<p>To demonstrate the application of mutagenicity models to compounds with very limited experimental data and to show their strengths and weaknesses we decided to apply them to Pyrrolizidine alkaloids (PAs).</p>
<p>Pyrrolizidine alkaloids (PAs) are characteristic metabolites of some plant families, mainly: Asteraceae, Boraginaceae, Fabaceae and Orchidaceae (<xref ref-type="bibr" rid="B11">Hartmann and Witte, 1995</xref>; <xref ref-type="bibr" rid="B16">Langel et&#x20;al., 2011</xref>) and form a powerful defence mechanism against herbivores. PAs are heterocyclic ester alkaloids composed of a necine base (two fused five-membered rings joined by a single nitrogen atom) and a necic acid (one or two carboxylic ester arms), occurring principally in two forms, tertiary base PAs and PA N-oxides.</p>
<p>In mammals, PAs are mainly metabolized in the liver. There are three principal metabolic pathways for 1,2-unsaturated PAs (<xref ref-type="bibr" rid="B3">Chen et&#x20;al.,2010</xref>):<list list-type="simple">
<list-item>
<p>&#x2022; Detoxification by hydrolysis of the ester bond on positions C7 and C9 by non-specific esterases to release necine base and necic&#x20;acid.</p>
</list-item>
<list-item>
<p>&#x2022; N-oxidation of the necine base to form PA N-oxides, which can be either conjugated by phase II enzymes and then excreted or converted back into the corresponding parent PA (<xref ref-type="bibr" rid="B29">Wang et&#x20;al, 2005</xref>). This detoxification pathway is not possible for otonecine-type PAs, as they are N-methylated (see <xref ref-type="fig" rid="F1">Figure&#x20;1</xref>).</p>
</list-item>
<list-item>
<p>&#x2022; Metabolic activation or toxification by oxidation (for retronecine-type PAs) or oxidative N-demethylation (for otonecine-type Pas) by cytochromes P450 isoforms CYP2B and 3A (<xref ref-type="bibr" rid="B18">Lin et&#x20;al, 1998</xref>; <xref ref-type="bibr" rid="B25">Ruan et&#x20;al, 2014</xref>).</p>
</list-item>
</list>
</p>
<fig id="F1" position="float">
<label>FIGURE 1</label>
<caption>
<p>Structural features of pyrrolizidine alkaloids (modified after <xref ref-type="bibr" rid="B28">Sch&#xf6;ning et&#x20;al., 2017</xref>).</p>
</caption>
<graphic xlink:href="fphar-12-708050-g001.tif"/>
</fig>
<p>The latter reactions result in the formation of dehydropyrrolizidine (DHP) that is highly reactive and causes damage by building adducts with protein, lipids and DNA (<xref ref-type="bibr" rid="B3">Chen et&#x20;al, 2010</xref>). On the other hand, open diesters and macrocyclic PAs have a reduced detoxification due to steric hinderance of the respective esterases (<xref ref-type="bibr" rid="B25">Ruan et&#x20;al, 2014</xref>). However, due to limited availability of pure substances, only a small number of PAs have been investigated experimentally in an Ames test. To overcome this bottleneck, the application of different machine learning models to predict mutagenic probabilities based on structures and properties could provide further insights into the mutagenicity mechanisms of&#x20;PAs.</p>
</sec>
<sec sec-type="materials|methods" id="s2">
<title>2 Materials and Methods</title>
<sec id="s2-1">
<title>2.1 Data</title>
<sec id="s2-1-1">
<title>2.1.1 Mutagenicity Training Data</title>
<p>An identical training dataset was used for all models. The training dataset was compiled from the following sources:<list list-type="simple">
<list-item>
<p>&#x2022; Kazius/Bursi Dataset (4,337 compounds, (<xref ref-type="bibr" rid="B15">Kazius et&#x20;al, 2005</xref>)): <ext-link ext-link-type="uri" xlink:href="http://cheminformatics.org/datasets/bursi/cas_4337.zip">http://cheminformatics.org/datasets/bursi/cas_4337.zip</ext-link>
</p>
</list-item>
<list-item>
<p>&#x2022; Hansen Dataset (6,513 compounds, <xref ref-type="bibr" rid="B10">Hansen et&#x20;al, 2009</xref>)): <ext-link ext-link-type="uri" xlink:href="http://doc.ml.tu-berlin.de/toxbenchmark/Mutagenicity_N6512.csv">http://doc.ml.tu-berlin.de/toxbenchmark/Mutagenicity_N6512.csv</ext-link>
</p>
</list-item>
<list-item>
<p>&#x2022; EFSA Dataset (695 compounds <xref ref-type="bibr" rid="B4">EFSA 2016</xref>)): <ext-link ext-link-type="uri" xlink:href="https://data.europa.eu/euodp/data/storage/f/2017-0719T142131/GENOTOX\%20data\%20and\%20dictionary.xls">https://data.europa.eu/euodp/data/storage/f/2017-0719T142131/GENOTOX\%20data\%20and\%20dictionary.xls</ext-link>
</p>
</list-item>
</list>
</p>
<p>Mutagenicity classifications from Kazius and Hansen datasets were used without further processing. According to these publications, compounds were classified as mutagenic if at least one positive result has been obtained in <italic>Salmonella typhimurium</italic> strains TA97, TA98, TA100, TA102, TA1535, TA1537 and TA1538 either with or without metabolic activation by S9. <italic>E.&#x20;coli</italic> results were not considered in these databases. To achieve consistency with these datasets, EFSA compounds were classified as mutagenic, if at least one positive result was found for the same <italic>Salmonella</italic> strains either with or without metabolic activation and as non-mutagenic if no positive result was found. The complete dataset contains chemicals from very diverse application areas (e.g., pharmaceuticals, pesticides, industrial chemicals, environmental contaminants).</p>
<p>Dataset merges were based on unique SMILES (<italic>Simplified Molecular Input Line Entry Specification</italic>, (<xref ref-type="bibr" rid="B30">Weininger et&#x20;al, 1989</xref>)) strings of the compound structures. Duplicated experimental data with the same outcome was merged into a single value, because it is likely that it originated from the same experiment. Contradictory results were kept as multiple measurements in the database. The combined training dataset contains 8,290 unique structures and 8,309 individual measurements. Contradictory results were found for 19 substances.</p>
<p>Source code for all data download, extraction and merge operations is publicly available from the git repository <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper">https://git.in-silico.ch/mutagenicity-paper</ext-link> under a GPL3 License. The new combined dataset can be found at <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/mutagenicity/mutagenicity.csv">https://git.in-silico.ch/mutagenicity-paper/tree/mutagenicity/mutagenicity.csv</ext-link>.</p>
</sec>
<sec id="s2-1-2">
<title>2.1.2 Pyrrolizidine Alkaloid Dataset</title>
<p>The pyrrolizidine alkaloid dataset was created from five independent, necine base substructure searches in PubChem (<ext-link ext-link-type="uri" xlink:href="https://pubchem.ncbi.nlm.nih.gov/">https://pubchem.ncbi.nlm.nih.gov/</ext-link>) and compared to the PAs listed in (<xref ref-type="bibr" rid="B5">EFSA, 2011</xref>) and the book by (<xref ref-type="bibr" rid="B21">Mattocks, 1986</xref>), to ensure, that all major PAs were included. PAs mentioned in these publications, which were not found in the downloaded substances were searched individually in PubChem and, if available, downloaded separately. Non-PA substances, duplicates, and isomers were removed from the files, but artificial PAs, even if unlikely to occur in nature, were kept. The resulting PA dataset comprised a total of 602 different PAs. Further details about the compilation of the PA dataset are described in (<xref ref-type="bibr" rid="B28">Sch&#xf6;ning et&#x20;al, 2017</xref>).</p>
<p>The PAs in the dataset were classified according to structural features. A total of nine different structural features were assigned to the necine base, to modifications of the necine base and to the necic acid (<xref ref-type="fig" rid="F1">Figure&#x20;1</xref>):</p>
<p>For the necine base, the following structural features were chosen:<list list-type="simple">
<list-item>
<p>&#x2022; Retronecine-type (1,2-unstaturated necine base, 392 compounds)</p>
</list-item>
<list-item>
<p>&#x2022; Otonecine-type (1,2-unstaturated necine base, 46 compounds)</p>
</list-item>
<list-item>
<p>&#x2022; Platynecine-type (1,2-saturated necine base, 140 compounds)</p>
</list-item>
</list>
</p>
<p>For the modifications of the necine base, the following structural features were chosen:<list list-type="simple">
<list-item>
<p>&#x2022; N-oxide-type (84 compounds)</p>
</list-item>
<list-item>
<p>&#x2022; Dehydropyrrolizidine-type (DHP, pyrrolic ester, 23 compounds)</p>
</list-item>
<list-item>
<p>&#x2022; Tertiary-type (PAs which were neither from the N-oxide- nor DHP-type, 495 compounds)</p>
</list-item>
</list>
</p>
<p>For the necic acid, the following structural features were chosen:<list list-type="simple">
<list-item>
<p>&#x2022; Monoester-type (154 compounds)</p>
</list-item>
<list-item>
<p>&#x2022; Open-ring diester-type (163 compounds)</p>
</list-item>
<list-item>
<p>&#x2022; Macrocyclic diester-type (255 compounds)</p>
</list-item>
</list>
</p>
</sec>
</sec>
<sec id="s2-2">
<title>2.2 Descriptors</title>
<sec id="s2-2-1">
<title>2.2.1 MolPrint2D Fingerprints</title>
<p>MolPrint2D fingerprints (<xref ref-type="bibr" rid="B23">O&#x2019;Boyle et&#x20;al, 2011</xref>) use atom environments as molecular representation. They determine for each atom in a molecule, the atom types of its connected atoms to represent their chemical environment. This resembles basically the chemical concept of functional groups.</p>
<p>In contrast to predefined lists of fragments (e.g., FP3, FP4 or MACCs fingerprints) or descriptors (e.g., CDK) they are generated dynamically from chemical structures. This has the advantage that they can capture unknown substructures of toxicological relevance that are not included in other descriptors. In addition, they allow the efficient calculation of chemical similarities (e.g., Tanimoto indices) with simple set operations.</p>
<p>MolPrint2D fingerprints were calculated with the OpenBabel cheminformatics library (<xref ref-type="bibr" rid="B23">O&#x2019;Boyle et&#x20;al, 2011</xref>) for the complete training dataset with 8,290 unique structures. They can be obtained from the following locations:</p>
<p>Training data:<list list-type="simple">
<list-item>
<p>&#x2022; sparse representation (<ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/mutagenicity/mutagenicity-mp2d">https://git.in-silico.ch/mutagenicity-paper/tree/mutagenicity/mutagenicity-mp2d</ext-link>)</p>
</list-item>
<list-item>
<p>&#x2022; descriptor matrix (<ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/mutagenicity/mutagenicity-mp2d.csv.gz">https://git.in-silico.ch/mutagenicity-paper/tree/mutagenicity/mutagenicity-mp2d.csv.gz</ext-link>)</p>
</list-item>
</list>
</p>
<p>Pyrrolizidine alkaloids:<list list-type="simple">
<list-item>
<p>&#x2022; sparse representation (<ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/pa-mp2d">https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/pa-mp2d</ext-link>)</p>
</list-item>
<list-item>
<p>&#x2022; descriptor matrix (<ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/pa-mp2d.csv">https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/pa-mp2d.csv</ext-link>)</p>
</list-item>
</list>
</p>
</sec>
<sec id="s2-2-2">
<title>2.2.2 Chemistry Development Kit Descriptors</title>
<p>Molecular 1D and 2D descriptors were calculated with the PaDEL-Descriptors program (<ext-link ext-link-type="uri" xlink:href="http://www.yapcwsoft.com">http://www.yapcwsoft.com</ext-link> version 2.21, (<xref ref-type="bibr" rid="B34">Yap, 2011</xref>)). PaDEL uses the Chemistry Development Kit (<italic>CDK</italic>, <ext-link ext-link-type="uri" xlink:href="https://cdk.github.io/index.html">https://cdk.github.io/index.html</ext-link>) library for descriptor calculations.</p>
<p>As the training dataset contained 8,309 instances, it was decided to delete all instances where CDK descriptor calculations failed during pre-processing. Furthermore, 19 substances with contradictory experimental results were removed. The final training dataset contained 1,442 descriptors for 8,083 compounds.</p>
<p>CDK training data can be obtained from <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/mutagenicity/mutagenicity-cdk.csv">https://git.in-silico.ch/mutagenicity-paper/tree/mutagenicity/mutagenicity-cdk.csv</ext-link>.</p>
<p>The same procedure was applied for the pyrrolizidine dataset yielding descriptors for compounds. CDK features for pyrrolizidine alkaloids are available at <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/pa-cdk.csv">https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/pa-cdk.csv</ext-link>.</p>
</sec>
</sec>
<sec id="s2-3">
<title>2.3 Algorithms</title>
<sec id="s2-3-1">
<title>2.3.1 Lazar</title>
<p>Lazar (<italic>lazy structure activity relationships</italic>) is a modular framework for read-across model development and validation. It follows the following basic workflow: For a given chemical structure lazar:<list list-type="simple">
<list-item>
<p>&#x2022; searches in a database for similar structures (neighbors) with experimental data,</p>
</list-item>
<list-item>
<p>&#x2022; builds a local QSAR model with these neighbors&#x20;and</p>
</list-item>
<list-item>
<p>&#x2022; uses this model to predict the unknown activity of the query compound.</p>
</list-item>
</list>
</p>
<p>This procedure resembles an automated version of read across predictions in toxicology. In machine learning terms it would be classified as a k-nearest-neighbor algorithm.</p>
<p>Apart from this basic workflow, lazar is completely modular and allows the researcher to use arbitrary algorithms for similarity searches and local QSAR (<italic>Quantitative structure&#x2013;activity relationship</italic>) modeling. Algorithms used within this study are described in the following sections.</p>
<sec id="s2-3-1-1">
<title>Feature Preprocessing</title>
<p>MolPrint2D features were used without preprocessing. Near zero variance and strongly correlated CDK descriptors were removed and the remaining descriptor values were centered and scaled. Preprocessing was performed with the R caret preProcess function using the methods &#x201c;nzv&#x201d;,&#x201c;corr&#x201d;,&#x201c;center&#x201d; and &#x201c;scale&#x201d; with default settings.</p>
</sec>
<sec id="s2-3-1-2">
<title>Neighbor Identification</title>
<p>Utilizing this modularity, similarity calculations were based both on MolPrint2D fingerprints and on CDK descriptors.</p>
<p>For MolPrint2D fingerprints chemical similarity between two compounds <italic>a</italic> and <italic>b</italic> is expressed as the proportion between atom environments common in both structures <inline-formula id="inf1">
<mml:math id="m1">
<mml:mrow>
<mml:mi>A</mml:mi>
<mml:mo>&#x2229;</mml:mo>
<mml:mi>B</mml:mi>
</mml:mrow>
</mml:math>
</inline-formula> and the total number of atom environments <inline-formula id="inf2">
<mml:math id="m2">
<mml:mrow>
<mml:mi>A</mml:mi>
<mml:mo>&#x222a;</mml:mo>
<mml:mi>B</mml:mi>
</mml:mrow>
</mml:math>
</inline-formula> (Jaccard/Tanimoto index).<disp-formula id="e1">
<mml:math id="m3">
<mml:mrow>
<mml:mi>s</mml:mi>
<mml:mi>i</mml:mi>
<mml:mi>m</mml:mi>
<mml:mo>&#x3d;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:mrow>
<mml:mo>&#x7c;</mml:mo>
<mml:mrow>
<mml:mi>A</mml:mi>
<mml:mtext>&#x2009;</mml:mtext>
<mml:mo>&#x2229;</mml:mo>
<mml:mi>B</mml:mi>
</mml:mrow>
<mml:mo>&#x7c;</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo>&#x7c;</mml:mo>
<mml:mrow>
<mml:mi>A</mml:mi>
<mml:mtext>&#x2009;</mml:mtext>
<mml:mo>&#x222a;</mml:mo>
<mml:mi>B</mml:mi>
</mml:mrow>
<mml:mo>&#x7c;</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:math>
<label>(1)</label>
</disp-formula>
</p>
<p>For CDK descriptors chemical similarity between two compounds <italic>a</italic> and <italic>b</italic> is expressed as the cosine similarity between the descriptor vectors <italic>A</italic> for <italic>a</italic> and <italic>B</italic> for <italic>b</italic>.<disp-formula id="e2">
<mml:math id="m4">
<mml:mrow>
<mml:mi>s</mml:mi>
<mml:mi>i</mml:mi>
<mml:mi>m</mml:mi>
<mml:mo>&#x3d;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:mi>A</mml:mi>
<mml:mo>&#x22c5;</mml:mo>
<mml:mi>B</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo>&#x7c;</mml:mo>
<mml:mi>A</mml:mi>
<mml:mo>&#x7c;</mml:mo>
</mml:mrow>
<mml:mrow>
<mml:mo>&#x7c;</mml:mo>
<mml:mi>B</mml:mi>
<mml:mo>&#x7c;</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:math>
<label>(2)</label>
</disp-formula>
</p>
<p>Threshold selection is a trade-off between prediction accuracy (high threshold) and the number of predictable compounds (low threshold). As it is in many practical cases desirable to make predictions even in the absence of closely related neighbors, we follow a tiered approach:<list list-type="simple">
<list-item>
<p>&#x2022; First a similarity threshold of 0.5 (MP2D/Tanimoto) or 0.9 (CDK/Cosine) is used to collect neighbors, to create a local QSAR model and to make a prediction for the query compound. This are predictions with high confidence.</p>
</list-item>
<list-item>
<p>&#x2022; If any of these steps fails, the procedure is repeated with a similarity threshold of 0.2 (MP2D/Tanimoto) or 0.7 (CDK/Cosine) and the prediction is flagged with a warning that it might be out of the applicability domain of the training data (low confidence).</p>
</list-item>
<list-item>
<p>&#x2022; These similarity thresholds are the default values chosen by software developers and remained unchanged during the course of these experiments.</p>
</list-item>
</list>
</p>
<p>Compounds with the same structure as the query structure are automatically eliminated from neighbors to obtain unbiased predictions in the presence of duplicates.</p>
</sec>
<sec id="s2-3-1-3">
<title>Local QSAR Models and Predictions</title>
<p>Only similar compounds (neighbors) above the threshold are used for local QSAR models. In this investigation, we are using a weighted majority vote from the neighbor&#x2019;s experimental data for mutagenicity classifications. Probabilities for both classes (mutagenic/non-mutagenic) are calculated according to the following formula and the class with the higher probability is used as prediction outcome.<disp-formula id="e3">
<mml:math id="m5">
<mml:mrow>
<mml:msub>
<mml:mi>p</mml:mi>
<mml:mi>c</mml:mi>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:mstyle displaystyle="true">
<mml:mo>&#x2211;</mml:mo>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mtext>sim</mml:mtext>
</mml:mrow>
<mml:mrow>
<mml:mi mathvariant="italic">n,c</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mstyle>
</mml:mrow>
<mml:mrow>
<mml:mstyle displaystyle="true">
<mml:mo>&#x2211;</mml:mo>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mtext>sim</mml:mtext>
</mml:mrow>
<mml:mi mathvariant="italic">n</mml:mi>
</mml:msub>
</mml:mrow>
</mml:mstyle>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:math>
<label>(3)</label>
</disp-formula>
<inline-formula id="inf3">
<mml:math id="m6">
<mml:mrow>
<mml:msub>
<mml:mi mathvariant="italic">p</mml:mi>
<mml:mi mathvariant="italic">c</mml:mi>
</mml:msub>
</mml:mrow>
</mml:math>
</inline-formula> Probability of class c (e.g. mutagenic or non-mutagenic) <inline-formula id="inf4">
<mml:math id="m7">
<mml:mrow>
<mml:msup>
<mml:mstyle displaystyle="true">
<mml:mrow>
<mml:mstyle displaystyle="true">
<mml:mo>&#x2211;</mml:mo>
<mml:mrow>
<mml:mi mathvariant="normal">si</mml:mi>
<mml:msub>
<mml:mi mathvariant="normal">m</mml:mi>
<mml:mrow>
<mml:mi mathvariant="italic">n,c</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mstyle>
</mml:mrow>
</mml:mstyle>
<mml:mtext>&#x200b;</mml:mtext>
</mml:msup>
</mml:mrow>
</mml:math>
</inline-formula> Sum of similarities of neighbors with class c <inline-formula id="inf5">
<mml:math id="m8">
<mml:mrow>
<mml:msup>
<mml:mstyle displaystyle="true">
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mstyle displaystyle="true">
<mml:mo>&#x2211;</mml:mo>
<mml:mrow>
<mml:mtext>sim</mml:mtext>
</mml:mrow>
</mml:mstyle>
</mml:mrow>
<mml:mi mathvariant="italic">n</mml:mi>
</mml:msub>
</mml:mrow>
</mml:mstyle>
<mml:mtext>&#x200b;</mml:mtext>
</mml:msup>
</mml:mrow>
</mml:math>
</inline-formula> Sum of all neighbors.</p>
</sec>
<sec id="s2-3-1-4">
<title>Applicability Domain</title>
<p>The applicability domain (AD) of lazar models is determined by the structural diversity of the training data. If no similar compounds are found in the training data no predictions will be generated. Warnings are issued if the similarity threshold had to be lowered from 0.5 to 0.2 in order to enable predictions. Predictions without warnings can be considered as close to the applicability domain (<italic>high confidence</italic>) and predictions with warnings as more distant from the applicability domain (<italic>low confidence</italic>). Quantitative applicability domain information can be obtained from the similarities of individual neighbors.</p>
</sec>
<sec id="s2-3-1-5">
<title>Validation</title>
<p>10-fold cross validation was performed for model evaluation.</p>
</sec>
<sec id="s2-3-1-6">
<title>Pyrrolizidine Alkaloid Predictions</title>
<p>For the prediction of pyrrolizidine alkaloids models were generated with the MP2D and CDK training datasets. The complete feature set was used for MP2D predictions, for CDK predictions the intersection between training and pyrrolizidine alkaloid features was&#x20;used.</p>
</sec>
<sec id="s2-3-1-7">
<title>Availability</title>
<p>
<list list-type="simple">
<list-item>
<p>&#x2022; Source code for this manuscript (GPL3): <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/lazar/tree/?h=mutagenicity-paper">https://git.in-silico.ch/lazar/tree/?h&#x3d;mutagenicity-paper</ext-link>
</p>
</list-item>
<list-item>
<p>&#x2022; Crossvalidation experiments (GPL3): <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/lazar/tree/models/?h=mutagenicity-paper">https://git.in-silico.ch/lazar/tree/models/?h&#x3d;mutagenicity-paper</ext-link>
</p>
</list-item>
<list-item>
<p>&#x2022; Pyrrolizidine alkaloid predictions (GPL3): <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/lazar/tree/predictions/?h=mutagenicity-paper">https://git.in-silico.ch/lazar/tree/predictions/?h&#x3d;mutagenicity-paper</ext-link>
</p>
</list-item>
<list-item>
<p>&#x2022; Public web interface: <ext-link ext-link-type="uri" xlink:href="https://lazar.in-silico.ch">https://lazar.in-silico.ch</ext-link>
</p>
</list-item>
</list>
</p>
</sec>
</sec>
<sec id="s2-3-2">
<title>2.3.2 Tensorflow Models</title>
<sec id="s2-3-2-1">
<title>Feature Preprocessing</title>
<p>For preprocessing of the CDK features we used a quantile transformation to a uniform distribution. MP2D features were not preprocessed.</p>
</sec>
<sec id="s2-3-2-2">
<title>Random Forests (<italic>RF</italic>)</title>
<p>For the random forest classifier we used the parameters n_estimators &#x3d; 1,000 and max_leaf_nodes &#x3d; 200. For the other parameters we used the scikit-learn default values.</p>
</sec>
<sec id="s2-3-2-3">
<title>Logistic Regression (SGD) (<italic>LR-Ssgd</italic>)</title>
<p>For the logistic regression we used a combination of five trained models. For each model we used a batch size of 64 and trained for 50 epochs. As an optimizer ADAM was chosen. For the other parameters we used the tensorflow default values.</p>
</sec>
<sec id="s2-3-2-4">
<title>Logistic Regression (Scikit) (<italic>LR-Scikit</italic>)</title>
<p>For the logistic regression we used as parameters the scikit-learn default values.</p>
</sec>
<sec id="s2-3-2-5">
<title>Neural Nets</title>
<p>For the neural network we used a combination of five trained models. For each model we used a batch size of 64 and trained for 50 epochs. As an optimizer ADAM was chosen. The neural network had four hidden layers with 64 nodes each and a ReLu activation function. For the other parameters we used the tensorflow default values.</p>
</sec>
<sec id="s2-3-2-6">
<title>Support Vector Machines</title>
<p>We used the SVM implemented in scikit-learn. We used the parameters kernel &#x3d; &#x201c;rbf,&#x201d; gamma &#x3d; &#x201c;scale&#x201d;. For the other parameters we used the scikit-learn default values.</p>
</sec>
<sec id="s2-3-2-7">
<title>Validation</title>
<p>10-fold cross-validation was used for all Tensorflow models.</p>
</sec>
<sec id="s2-3-2-8">
<title>Pyrrolizidine Alkaloid Predictions</title>
<p>For the prediction of pyrrolizidine alkaloids we trained the model described above on the training data. For training and prediction only the features were used that were in the intersection of features from the training data and the pyrrolizidine alkaloids.</p>
</sec>
<sec id="s2-3-2-9">
<title>Availability</title>
<p>Jupyter notebooks for these experiments can be found at the following locations.</p>
</sec>
<sec id="s2-3-2-10">
<title>Crossvalidation:</title>
<p>
<list list-type="simple">
<list-item>
<p>&#x2022; MolPrint2D fingerprints: <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/crossvalidations/tensorflow/prediction-v5-norm.ipynb">https://git.in-silico.ch/mutagenicity-paper/tree/crossvalidations/tensorflow/prediction-v5-norm.ipynb</ext-link>
</p>
</list-item>
<list-item>
<p>&#x2022; CDK descriptors: <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/crossvalidations/tensorflow/prediction-v5-ext.ipynb">https://git.in-silico.ch/mutagenicity-paper/tree/crossvalidations/tensorflow/prediction-v5-ext.ipynb</ext-link>
</p>
</list-item>
</list>
</p>
</sec>
<sec id="s2-3-2-11">
<title>Pyrrolizidine Alkaloids:</title>
<p>
<list list-type="simple">
<list-item>
<p>&#x2022; MolPrint2D fingerprints: <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/tensorflow/prediction-v5-ext-ext-Padel-2D.ipynb">https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/tensorflow/prediction-v5-ext-ext-Padel-2D.ipynb</ext-link>
</p>
</list-item>
<list-item>
<p>&#x2022; CDK descriptors: <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/tensorflow/prediction-v5-ext-Padel-2D.ipynb">https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/tensorflow/prediction-v5-ext-Padel-2D.ipynb</ext-link>
</p>
</list-item>
</list>
</p>
</sec>
</sec>
</sec>
</sec>
<sec sec-type="results" id="s3">
<title>3 Results</title>
<sec id="s3-1">
<title>3.1&#x20;10-Fold Crossvalidations</title>
<p>Crossvalidation results are summarized in the following tables: <xref ref-type="table" rid="T1">Table&#x20;1</xref> shows results with MolPrint2D descriptors and <xref ref-type="table" rid="T2">Table&#x20;2</xref> with CDK descriptors.</p>
<table-wrap id="T1" position="float">
<label>TABLE 1</label>
<caption>
<p>Summary of crossvalidation results with MolPrint descriptors (lazar-HC, lazar with high confidence, lazar-all, all lazar predictions, RF, random forests, LR-sgd, logistic regression (stochastic gradient descent), LR-scikit, logistic regression (scikit), NN, neural networks, SVM, support vector machines).</p>
</caption>
<table>
<thead>
<tr>
<td align="left"/>
<td align="center">Lazar-HC</td>
<td align="center">Lazar-all</td>
<td align="center">RF</td>
<td align="center">LR-sgd</td>
<td align="center">LR-scikit</td>
<td align="center">NN</td>
<td align="center">SVM</td>
</tr>
</thead>
<tbody valign="top">
<tr>
<td align="left">Accuracy</td>
<td align="center">84</td>
<td align="center">82</td>
<td align="center">80</td>
<td align="center">84</td>
<td align="center">84</td>
<td align="center">84</td>
<td align="center">84</td>
</tr>
<tr>
<td align="left">True positive rate</td>
<td align="center">89</td>
<td align="center">85</td>
<td align="center">78</td>
<td align="center">83</td>
<td align="center">83</td>
<td align="center">82</td>
<td align="center">83</td>
</tr>
<tr>
<td align="left">True negative rate</td>
<td align="center">78</td>
<td align="center">78</td>
<td align="center">82</td>
<td align="center">84</td>
<td align="center">85</td>
<td align="center">85</td>
<td align="center">86</td>
</tr>
<tr>
<td align="left">Positive predictive value</td>
<td align="center">83</td>
<td align="center">80</td>
<td align="center">81</td>
<td align="center">84</td>
<td align="center">84</td>
<td align="center">84</td>
<td align="center">85</td>
</tr>
<tr>
<td align="left">Negative predictive value</td>
<td align="center">86</td>
<td align="center">84</td>
<td align="center">80</td>
<td align="center">84</td>
<td align="center">84</td>
<td align="center">83</td>
<td align="center">84</td>
</tr>
<tr>
<td align="left">Nr. predictions</td>
<td align="center">5,864</td>
<td align="center">7,782</td>
<td align="center">8,303</td>
<td align="center">8,303</td>
<td align="center">8,303</td>
<td align="center">8,303</td>
<td align="center">8,303</td>
</tr>
</tbody>
</table>
</table-wrap>
<table-wrap id="T2" position="float">
<label>TABLE 2</label>
<caption>
<p>Summary of crossvalidation results with CDK descriptors (lazar-HC, lazar with high confidence, lazar-all: all lazar predictions, RF, random forests, LR-sgd, logistic regression (stochastic gradient descent), LR-scikit, logistic regression (scikit), NN, neural networks, SVM, support vector machines).</p>
</caption>
<table>
<thead>
<tr>
<td align="left"/>
<td align="center">Lazar-HC</td>
<td align="center">Lazar-all</td>
<td align="center">RF</td>
<td align="center">LR-sgd</td>
<td align="center">LR-scikit</td>
<td align="center">NN</td>
<td align="center">SVM</td>
</tr>
</thead>
<tbody valign="top">
<tr>
<td align="left">Accuracy</td>
<td align="center">85</td>
<td align="center">82</td>
<td align="center">84</td>
<td align="center">79</td>
<td align="center">80</td>
<td align="center">85</td>
<td align="center">82</td>
</tr>
<tr>
<td align="left">True positive rate</td>
<td align="center">87</td>
<td align="center">84</td>
<td align="center">81</td>
<td align="center">81</td>
<td align="center">80</td>
<td align="center">85</td>
<td align="center">82</td>
</tr>
<tr>
<td align="left">True negative rate</td>
<td align="center">82</td>
<td align="center">80</td>
<td align="center">86</td>
<td align="center">78</td>
<td align="center">80</td>
<td align="center">85</td>
<td align="center">82</td>
</tr>
<tr>
<td align="left">Positive predictive value</td>
<td align="center">85</td>
<td align="center">81</td>
<td align="center">85</td>
<td align="center">79</td>
<td align="center">80</td>
<td align="center">85</td>
<td align="center">82</td>
</tr>
<tr>
<td align="left">Negative predictive value</td>
<td align="center">85</td>
<td align="center">82</td>
<td align="center">82</td>
<td align="center">80</td>
<td align="center">80</td>
<td align="center">85</td>
<td align="center">82</td>
</tr>
<tr>
<td align="left">Nr. predictions</td>
<td align="center">4,872</td>
<td align="center">7,353</td>
<td align="center">8,077</td>
<td align="center">8,077</td>
<td align="center">8,077</td>
<td align="center">8,077</td>
<td align="center">8,077</td>
</tr>
</tbody>
</table>
</table-wrap>
<p>
<xref ref-type="fig" rid="F2">Figure&#x20;2</xref> depicts the position of all crossvalidation results in receiver operating characteristic (ROC)&#x20;space.</p>
<fig id="F2" position="float">
<label>FIGURE 2</label>
<caption>
<p>ROC plot of crossvalidation results (lazar-HC, lazar with high confidence, lazar-all: all lazar predictions, RF, random forests, LR-sgd, logistic regression (stochastic gradient descent), LR-scikit, logistic regression (scikit), NN, neural networks, SVM, support vector machines).</p>
</caption>
<graphic xlink:href="fphar-12-708050-g002.tif"/>
</fig>
<p>Confusion matrices for all models are available from the git repository <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/crossvalidations/confusion-matrices/">https://git.in-silico.ch/mutagenicity-paper/tree/crossvalidations/confusion-matrices/</ext-link>, individual predictions can be found in <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/crossvalidations/predictions/">https://git.in-silico.ch/mutagenicity-paper/tree/crossvalidations/predictions/</ext-link>.</p>
<p>All investigated algorithm/descriptor combinations give accuracies between (80 and 85%) which is equivalent to the experimental variability of the <italic>Salmonella typhimurium</italic> mutagenicity bioassay (80&#x2013;85%, <xref ref-type="bibr" rid="B24">Piegorsch and Zeiger, 1991)</xref>). Sensitivities and specificities are balanced in all of these models.</p>
</sec>
<sec id="s3-2">
<title>3.2 Pyrrolizidine Alkaloid Mutagenicity Predictions</title>
<p>Mutagenicity predictions of 602 pyrrolizidine alkaloids (PAs) from all investigated models can be downloaded from <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/pa-predictions.csv">https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/pa-predictions.csv</ext-link>. A visual representation of all PA predictions can be found at <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/pa-predictions.pdf">https://git.in-silico.ch/mutagenicity-paper/tree/pyrrolizidine-alkaloids/pa-predictions.pdf</ext-link>.</p>
<p>For the visualization of the position of pyrrolizidine alkaloids in respect to the training data set we have applied t-distributed stochastic neighbor embedding (t-SNE, (<xref ref-type="bibr" rid="B20">Maaten and Hinton, 2008</xref>) for MolPrint2D and CDK descriptors. t-SNE maps each high-dimensional object (chemical) to a two-dimensional point, maintaining the high-dimensional distances of the objects. Similar objects are represented by nearby points and dissimilar objects are represented by distant points. t-SNE coordinates were calculated with the R Rtsne package using the default settings (perplexity &#x3d; 30, theta &#x3d; 0.5, max_iter &#x3d; 1,000).</p>
<p>
<xref ref-type="fig" rid="F3">Figure&#x20;3</xref> shows the t-SNE of pyrrolizidine alkaloids (PA) and the mutagenicity training data in MP2D space (Tanimoto/Jaccard similarity), which resembles basically the structural diversity of the investigated compounds.</p>
<fig id="F3" position="float">
<label>FIGURE 3</label>
<caption>
<p>t-SNE visualization of mutagenicity training data and pyrrolizidine alkaloids (PA) in MP2D&#x20;space.</p>
</caption>
<graphic xlink:href="fphar-12-708050-g003.tif"/>
</fig>
<p>
<xref ref-type="fig" rid="F4">Figure&#x20;4</xref> shows the t-SNE of pyrrolizidine alkaloids (PA) and the mutagenicity training data in CDK space (Euclidean similarity), which resembles basically the physical-chemical properties of the investigated compounds.</p>
<fig id="F4" position="float">
<label>FIGURE 4</label>
<caption>
<p>t-SNE visualization of mutagenicity training data and pyrrolizidine alkaloids (PA) in CDK&#x20;space.</p>
</caption>
<graphic xlink:href="fphar-12-708050-g004.tif"/>
</fig>
<p>
<xref ref-type="fig" rid="F5">Figure&#x20;5</xref> and <xref ref-type="fig" rid="F6">Figure&#x20;6</xref> depict two example pyrrolizidine alkaloid mutagenicity predictions in the context of training data. t-SNE visualisations of all investigated models can be downloaded from <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/fures">https://git.in-silico.ch/mutagenicity-paper/figures</ext-link>.</p>
<fig id="F5" position="float">
<label>FIGURE 5</label>
<caption>
<p>t-SNE visualization of MP2D random forest predictions.</p>
</caption>
<graphic xlink:href="fphar-12-708050-g005.tif"/>
</fig>
<fig id="F6" position="float">
<label>FIGURE 6</label>
<caption>
<p>t-SNE visualization of all CDK lazar predictions.</p>
</caption>
<graphic xlink:href="fphar-12-708050-g006.tif"/>
</fig>
<p>
<xref ref-type="table" rid="T3">Table&#x20;3</xref> summarizses the outcome of pyrrolizidine alkaloid predictions from all models with MolPrint2D and CDK descriptors.</p>
<table-wrap id="T3" position="float">
<label>TABLE 3</label>
<caption>
<p>Summary of pyrrolizidine alkaloid predictions.</p>
</caption>
<table>
<thead>
<tr>
<td align="left">Model</td>
<td align="center">MP2D mutagenic</td>
<td align="center">Nr. predictions</td>
<td align="center">CDK mutagenic</td>
<td align="center">Nr. predictions</td>
</tr>
</thead>
<tbody valign="top">
<tr>
<td align="left">Lazar-all</td>
<td align="center">20% (111)</td>
<td align="center">93% (560)</td>
<td align="center">39% (193)</td>
<td align="center">83% (500)</td>
</tr>
<tr>
<td align="left">Lazar-HC</td>
<td align="center">25% (76)</td>
<td align="center">50% (301)</td>
<td align="center">45% (111)</td>
<td align="center">41% (246)</td>
</tr>
<tr>
<td align="left">RF</td>
<td align="center">5% (28)</td>
<td align="center">100% (602)</td>
<td align="center">2% (10)</td>
<td align="center">100% (602)</td>
</tr>
<tr>
<td align="left">LR-sgd</td>
<td align="center">21% (127)</td>
<td align="center">100% (602)</td>
<td align="center">16% (97)</td>
<td align="center">100% (602)</td>
</tr>
<tr>
<td align="left">LR-scikit</td>
<td align="center">20% (118)</td>
<td align="center">100% (602)</td>
<td align="center">15% (88)</td>
<td align="center">100% (602)</td>
</tr>
<tr>
<td align="left">NN</td>
<td align="center">21% (124)</td>
<td align="center">100% (602)</td>
<td align="center">25% (150)</td>
<td align="center">100% (602)</td>
</tr>
<tr>
<td align="left">SVM</td>
<td align="center">14% (82)</td>
<td align="center">100% (602)</td>
<td align="center">3% (19)</td>
<td align="center">100% (602)</td>
</tr>
</tbody>
</table>
</table-wrap>
<p>
<xref ref-type="fig" rid="F7">Figure&#x20;7</xref> displays the proportion of positive mutagenicity predictions from all models for the different pyrrolizidine alkaloid groups. Tensorflow models predicted all 602 pyrrolizidine alkaloids, lazar MP2D models predicted 560 compounds (301 with high confidence) and lazar CDK models 500 compounds (246 with high confidence).</p>
<fig id="F7" position="float">
<label>FIGURE 7</label>
<caption>
<p>Summary of pyrrolizidine alkaloid predictions.</p>
</caption>
<graphic xlink:href="fphar-12-708050-g007.tif"/>
</fig>
<p>For the lazar-HC model, only 50/41% of the PA dataset were within the stricter similarity thresholds of 0.5/0.9 (MP2D/CDK). Reduction of the similarity threshold to 0.2/0.5 in the lazar-all model increased the amount of predictable PAs to 93/83%. As the other ML models do not consider applicability domains, all PAs were predicted.</p>
<p>Although most of the models show similar accuracies, sensitivities and specificities in crossvalidation experiments some of the models (MPD-RF, CDK-RF and CDK-SVM) predict a lower number of mutagens (2&#x2013;5%) than the majority of the models (14&#x2013;25%, <xref ref-type="table" rid="T3">Table&#x20;3</xref>, <xref ref-type="fig" rid="F7">Figure&#x20;7</xref>).</p>
<p>Over all models, the mean value of mutagenic predicted PAs was highest for otonecines (65%, 407/623), followed by macrocyclic diesters (31%, 1,042/3,356), dehydropyrrolizidines (27%, 74/268), tertiary PAs (19%, 1,201/6,307) and retronecines (15%, 762/5,054).</p>
<p>When excluding the aforementioned three deviating models, the rank order stays the same, but the percentage of mutagenic PAs is higher.</p>
<p>The following rank order for mutagenic probability can be deduced from the results of all models taken together:</p>
<p>Necine base: Platynecine &#x3c; Retronecine &#x3c;&#x3c; Otonecine.</p>
<p>Necic acid: Monoester &#x3c; Diester &#x3c;&#x3c; Macrocyclic diester.</p>
<p>Modification of necine base: N-oxide &#x3c; Tertiary PA &#x3c; Dehydropyrrolizidine.</p>
</sec>
</sec>
<sec sec-type="discussion" id="s4">
<title>4 Discussion</title>
<sec id="s4-1">
<title>4.1 Data</title>
<p>A new training dataset for <italic>Salmonella</italic> mutagenicity was created from three different sources (<xref ref-type="bibr" rid="B15">Kazius et&#x20;al, 2005</xref>; <xref ref-type="bibr" rid="B10">Hansen et&#x20;al, 2009</xref>); <xref ref-type="bibr" rid="B4">EFSA, 2016</xref>). It contains 8,290 unique chemical structures, which is according to our knowledge the largest public mutagenicity dataset presently available. The new training data can be downloaded from <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/mutagenicity/mutagenicity.csv">https://git.in-silico.ch/mutagenicity-paper/tree/mutagenicity/mutagenicity.csv</ext-link>.</p>
</sec>
<sec id="s4-2">
<title>4.2 Algorithms</title>
<p>Lazar is formally a <italic>k-nearest-neighbor</italic> algorithm that searches for similar structures for a given compound and calculates the prediction based on the experimental data for these structures. The QSAR literature calls such models frequently <italic>local models</italic>, because models are generated specifically for each query compound. The investigated tensorflow models are in contrast <italic>global models</italic>, i.e. a single model is used to make predictions for all compounds. It has been postulated in the past, that local models are more accurate, because they can account better for mechanisms that affect only a subset of the training&#x20;data.</p>
<p>
<xref ref-type="table" rid="T1">Table&#x20;1</xref>, <xref ref-type="table" rid="T2">Table&#x20;2</xref> and <xref ref-type="fig" rid="F2">Figure&#x20;2</xref> show that the crossvalidation accuracies of all models are comparable to the experimental variability of the <italic>Salmonella typhimurium</italic> mutagenicity bioassay (80&#x2013;85% according to (<xref ref-type="bibr" rid="B24">Piegorsch and Zeiger, 1991</xref>). All of these models have balanced sensitivity (true positive rate) and specificity (true negative rate) and provide highly significant concordance with experimental data (as determined by McNemar&#x2019;s Test). This is a clear indication that <italic>in silico</italic> predictions can be as reliable as the bioassays. Given that the variability of experimental data is similar to model variability it is impossible to decide which model gives the most accurate predictions, as models with higher accuracies might just approximate experimental errors better than more robust models.</p>
<p>Our results do not support the assumption that local models are superior to global models for classification purposes. For regression models (lowest observed effect level) we have found however that local models may outperform global models (<xref ref-type="bibr" rid="B12">Helma et&#x20;al, 2018</xref>) with accuracies similar to experimental variability.</p>
<p>As all investigated algorithms give similar accuracies the selection will depend more on practical considerations than on intrinsic properties. Nearest neighbor algorithms like lazar have the practical advantage that the rationales for individual predictions can be presented in a straightforward manner that is understandable without a background in statistics or machine learning (a screenshot of the mutagenicity prediction for 12,21-Dihydroxy-4-methyl-4,8-secosenecinonan-8,11,16-trione is depicted in <xref ref-type="fig" rid="F8">Figure&#x20;8</xref>). This allows a critical examination of individual predictions and prevents blind trust in models that are intransparent to users with a toxicological background.</p>
<fig id="F8" position="float">
<label>FIGURE 8</label>
<caption>
<p>Lazar screenshot of 12,21-Dihydroxy-4-methyl-4,8-secosenecinonan-8,11,16-trione mutagenicity prediction.</p>
</caption>
<graphic xlink:href="fphar-12-708050-g008.tif"/>
</fig>
</sec>
<sec id="s4-3">
<title>4.3 Descriptors</title>
<p>This study uses two types of descriptors for the characterization of chemical structures:</p>
<p>
<italic>MolPrint2D</italic> fingerprints (MP2D, (<xref ref-type="bibr" rid="B2">Bender et&#x20;al, 2004</xref>) use atom environments (i.e.,&#x20;connected atom types for all atoms in a molecule) as molecular representation, which resembles basically the chemical concept of functional groups. MP2D descriptors are used to determine chemical similarities in the default lazar settings, and previous experiments have shown, that they give more accurate results than predefined fingerprints (e.g., MACCS, FP2-4).</p>
<p>
<italic>Chemistry Development Kit</italic> (CDK, <xref ref-type="bibr" rid="B31">Willighagen et&#x20;al., 2017</xref>) descriptors were calculated with the PaDEL graphical interface (<xref ref-type="bibr" rid="B34">Yap, 2011</xref>). They include 1D and 2D topological descriptors as well as physical-chemical properties.</p>
<p>All investigated algorithms obtained models within the experimental variability for both types of descriptors (<xref ref-type="table" rid="T1">Table&#x20;1</xref>, <xref ref-type="table" rid="T2">Table&#x20;2</xref>, <xref ref-type="fig" rid="F2">Figure&#x20;2</xref>).</p>
<p>Given that similar predictive accuracies are obtainable from both types of descriptors the choice depends once more on practical considerations:</p>
<p>MolPrint2D fragments can be calculated very efficiently for every well defined chemical structure with OpenBabel (<xref ref-type="bibr" rid="B23">O&#x2019;Boyle et&#x20;al, 2011</xref>). CDK descriptor calculations are in contrast much more resource intensive and may fail for a significant number of compounds (from 8,290).</p>
<p>MolPrint2D fragments are generated dynamically from chemical structures and can be used to determine if a compound contains structural features that are absent in training data. This feature can be used to determine applicability domains. CDK descriptors contain in contrast a predefined set of descriptors with unknown toxicological relevance.</p>
<p>MolPrint2D fingerprints can be represented very efficiently as sets of features that are present in a given compound which makes similarity calculations very efficient. Due to the large number of substructures present in training compounds, they lead however to large and sparsely populated datasets, if they have to be expanded to a binary matrix (e.g., as input for tensorflow models). CDK descriptors contain in contrast in every case matrices with 1,442 columns which can cause substantial computational overhead.</p>
</sec>
<sec id="s4-4">
<title>4.4 Pyrrolizidine Alkaloid Mutagenicity Predictions</title>
<sec id="s4-4-1">
<title>4.4.1 Algorithms and Descriptors</title>
<p>
<xref ref-type="fig" rid="F7">Figure&#x20;7</xref> shows a clear differentiation between the different pyrrolizidine alkaloid groups. Nevertheless differences between predictions from different algorithms and descriptors (<xref ref-type="table" rid="T3">Table&#x20;3</xref>) were not expected based on crossvalidation results.</p>
<p>In order to investigate, if any of the investigated models show systematic errors in the vicinity of pyrrolizidine-alkaloids we have performed a detailled t-SNE analysis of all models (see <xref ref-type="fig" rid="F5">Figure&#x20;5</xref> and <xref ref-type="fig" rid="F6">Figure&#x20;6</xref> for two examples, all visualisations can be found at <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper/tree/fures">https://git.in-silico.ch/mutagenicity-paper/tree/figures</ext-link>).</p>
<p>None of the models showed obvious deviations from their expected behavior, so the reason for the disagreement between some of the models remains unclear at the moment. It is however possible that some systematic errors are covered up by converting high dimensional spaces to two coordinates and are thus invisible in t-SNE visualisations.</p>
<p>Only two compounds from the PA dataset (Senecivernine and Retronecine) are part of the training set. Both are non-mutagenic and were predicted as non-mutagenic by all models (instances have been removed from the training set for unbiased predictions). Despite the exact concordance, we cannot draw any general conclusions about model performance based on two examples with a single outcome.</p>
</sec>
<sec id="s4-4-2">
<title>4.4.2 Necic Acid</title>
<p>The rank order of the necic acid is comparable in all models. PAs from the monoester type had the lowest genotoxic probability, followed by PAs from the open-ring diester type. PAs with macrocyclic diesters had the highest genotoxic probability. The result fits well with current state of knowledge: in general, PAs, which have a macrocyclic diesters as necic acid, are considered to be more mutagenic than those with an open-ring diester or monoester (<xref ref-type="bibr" rid="B5">EFSA, 2011</xref>; <xref ref-type="bibr" rid="B8">Fu et&#x20;al, 2004</xref>). As pointed out above, open diesters and macrocyclic PAs have a reduced detoxification due to steric hinderance of the respective esterases (<xref ref-type="bibr" rid="B25">Ruan et&#x20;al, 2014</xref>). This was also confirmed by more recent studies, confirming that macrocyclic- and open-diesters are more genotoxic <italic>in&#x20;vitro</italic> than monoesters (<xref ref-type="bibr" rid="B1">Allemang et&#x20;al, 2018</xref>; <xref ref-type="bibr" rid="B19">Louisse et&#x20;al, 2019</xref>; <xref ref-type="bibr" rid="B9">Hadi et&#x20;al, 2021</xref>).</p>
</sec>
<sec id="s4-4-3">
<title>4.4.3 Necine Base</title>
<p>In the rank order of necine base PAs, platynecine is the least mutagenic, followed by retronecine, and otonecine. Saturated PAs of the platynecine-type are generally accepted to be less or non-mutagenic and have been shown in <italic>in&#x20;vitro</italic> experiments to form no DNA-adducts (<xref ref-type="bibr" rid="B32">Xia et&#x20;al, 2013</xref>). In literature, otonecine-type PAs were shown to be more mutagenic than those of the retronecine-type (<xref ref-type="bibr" rid="B17">Li et&#x20;al, 2013</xref>).</p>
</sec>
<sec id="s4-4-4">
<title>4.4.4 Modifications of Necine Base</title>
<p>The group-specific results reflect the expected relationship between the groups: the low mutagenic probability of <italic>N</italic>-oxides and the high probability of dehydropyrrolizidines (DHP) (<xref ref-type="bibr" rid="B3">Chen et&#x20;al., 2010</xref>). However, <italic>N</italic>-oxides may be <italic>in vivo</italic> converted back to their parent mutagenic/tumorigenic parent PA (<xref ref-type="bibr" rid="B33">Yan et&#x20;al, 2008</xref>), on the other hand they are highly water soluble and generally considered as detoxification products, which are <italic>in vivo</italic> quickly renally eliminated (<xref ref-type="bibr" rid="B3">Chen et&#x20;al., 2010</xref>).</p>
<p>DHP are regarded as the toxic principle in the metabolism of PAs, and are known to produce protein- and DNA-adducts (<xref ref-type="bibr" rid="B3">Chen et&#x20;al., 2010</xref>). None of our investigated models did meet this expectation and all of them predicted the majority of DHP as non-mutagenic. However, the following issues need to be considered: On the one hand, all DHP were outside of the stricter applicability domain of MP2D lazar. This indicates that they are structurally very different than the training data and might be out of the applicability domain of all models based on this training set. In addition, DHP has two unsaturated double bounds in its necine base, making it highly reactive. DHP and other comparable molecules have a very short lifespan <italic>in vivo</italic>, and usually cannot be used in <italic>in&#x20;vitro</italic> experiments.</p>
<p>Overall the low number of positive mutagenicity predictions was unexpected. PAs are generally considered to be genotoxic, and the mode of action is also known. Therefore, the fact that some models predict the majority of PAs as not mutagenic seems contradictory. To understand this result, the experimental basis of the training dataset has to be considered. The training dataset is based on the <italic>Salmonella typhimurium</italic> mutagenicity bioassay (Ames test). There are some studies, which show mutagenicity of PAs in the Ames test (<xref ref-type="bibr" rid="B3">Chen et&#x20;al., 2010</xref>). Also, <xref ref-type="bibr" rid="B26">Rubiolo et&#x20;al. (1992)</xref> examined several different PAs and several different extracts of PA-containing plants in the Ames test. They found that the Ames test was indeed able to detect mutagenicity of PAs, but in general, appeared to have a low sensitivity. The pre-incubation phase for metabolic activation of PAs by microsomal enzymes was the sensitivity-limiting step. This could very well mean that the low sensitivity of the Ames test for PAs is also reflected in the investigated models.</p>
<p>In summary, we found marked differences in the predicted genotoxic probability between the PA groups: most mutagenic appeared the otonecines and macrocyclic diesters, least mutagenic the platynecines and the mono- and diesters. These results are comparable with <italic>in&#x20;vitro</italic> measurements in hepatic HepaRG cells (<xref ref-type="bibr" rid="B19">Louisse et&#x20;al, 2019</xref>), where relative potencies (RP) were determined: for otonecines and cyclic diesters RP &#x3d; 1, for open diesters RP &#x3d; 0.1 and for monoesters RP &#x3d;&#x20;0.01.</p>
<p>Due to a lack of differential data, European authorities based their risk assessment in a worst-case approach on lasiocarpine, for which sufficient data on genotoxicity and carcinogenicity were available (<xref ref-type="bibr" rid="B13">HMPC, 2014</xref>; <xref ref-type="bibr" rid="B6">EMA, 2020</xref>). Our data further support a tiered risk assessment based on <italic>in silico</italic> and experimental data on the relative potency of individual PAs as already suggested by other authors (<xref ref-type="bibr" rid="B22">Merz and Schrenk, 2016</xref>; <xref ref-type="bibr" rid="B19">Louisse et&#x20;al, 2019</xref>; <xref ref-type="bibr" rid="B27">Rutz et&#x20;al, 2020</xref>).</p>
<p>The practical question how to choose model predictions in the absence of experimental data remains open. Tensorflow predictions do not include applicability domain estimations and the rationales for predictions cannot be traced by toxicologists. Transparent models like lazar may have an advantage in this context, because they present rationales for predictions (similar compounds with experimental data) which can be accepted or rejected by toxicologists and provide validated applicability domain estimations.</p>
</sec>
</sec>
</sec>
<sec id="s5">
<title>5 Conclusion</title>
<p>A new public <italic>Salmonella</italic> mutagenicity training dataset with 8,309 experimental results was created and used to train lazar and Tensorflow models with MolPrint2D and CDK descriptors. All investigated algorithm and descriptor combinations showed accuracies comparable to the interlaboratory variability of the Ames&#x20;test.</p>
<p>Pyrrolizidine alkaloid predictions showed a clear separation between different classes of PAs which were generally in accordance with the current toxicological knowledge about these compounds. Some of the models showed however a substantially lower number of mutagenicity predictions, despite similar crossvalidation results and we were unable to identify the reasons for this discrepancy within this investigation.</p>
<p>Our data show that large difference exist with regard to mutagenic probabilities between different pyrrolizidine subgroups. To adjust risk assessment of pyrrolizidine contamination, our data supports a tiered risk assessment based on <italic>in silico</italic> predictions and experimental data of individual pyrrolizidine alkaloids.</p>
</sec>
</body>
<back>
<sec id="s6">
<title>Data Availability Statement</title>
<p>Data, source code, results and supplementary material presented in this study is publicly available at the Git repository <ext-link ext-link-type="uri" xlink:href="https://git.in-silico.ch/mutagenicity-paper">https://git.in-silico.ch/mutagenicity-paper</ext-link>. Further inquiries can be directed to the corresponding authors.</p>
</sec>
<sec id="s7">
<title>Author Contributions</title>
<p>CH: Calculation of MP2D features, lazar models (validation and pyrrolizidine alkaloid predictions), comparison of results, manuscript text, figures and tables VS: Preliminary experiments, calculation of CDK descriptors, interpretation of pyrrolizidine alkaloid predictions, manuscript text JD: Project supervision and coordination, interpretation of pyrrolizidine alkaloid predictions, manuscript text PB: Random forest, support vector machine, linear regression and neural net models (validation and pyrrolizidine alkaloid predictions), manuscript&#x20;text.</p>
</sec>
<sec id="s8">
<title>Funding</title>
<p>This work was partially supported by an unconditional reasearch grant of Max Zeller S&#xf6;hne AG to CH.</p>
</sec>
<sec sec-type="COI-statement" id="s9">
<title>Conflict of Interest</title>
<p>Author CH was employed by and the owner of company In Silico Toxicology Gmbh. JD is employed by Max Zeller Soehne AG.</p>
<p>The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.</p>
</sec>
<ref-list>
<title>References</title>
<ref id="B1">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Allemang</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Mahony</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Lester</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Pfuhler</surname>
<given-names>S.</given-names>
</name>
</person-group> <year>2018</year>. &#x201c;<article-title>Relative Potency of Fifteen Pyrrolizidine Alkaloids to Induce DNA Damage as Measured by Micronucleus Induction in HepaRG Human Liver Cells</article-title>.&#x201d; <source>Food Chem. Toxicol.</source> <volume>121</volume>: <fpage>72</fpage>&#x2013;<lpage>81</lpage>. <pub-id pub-id-type="doi">10.1016/j.fct.2018.08.003</pub-id> </citation>
</ref>
<ref id="B2">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Bender</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Mussa</surname>
<given-names>H. Y.</given-names>
</name>
<name>
<surname>Glen</surname>
<given-names>R. C.</given-names>
</name>
<name>
<surname>Reiling</surname>
<given-names>S.</given-names>
</name>
</person-group> (<year>2004</year>). <article-title>Molecular Similarity Searching Using Atom Environments, Information-Based Feature Selection, and a Na&#xef;ve Bayesian Classifier</article-title>. <source>J.&#x20;Chem. Inf. Comput. Sci.</source> <volume>44</volume> (<issue>1</issue>), <fpage>170</fpage>&#x2013;<lpage>178</lpage>. <pub-id pub-id-type="doi">10.1021/ci034207y</pub-id> </citation>
</ref>
<ref id="B3">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Chen</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Mei</surname>
<given-names>N.</given-names>
</name>
<name>
<surname>Fu</surname>
<given-names>P. P.</given-names>
</name>
</person-group> <year>2010</year>. &#x201c;<article-title>Genotoxicity of Pyrrolizidine Alkaloids</article-title>.&#x201d; <source>J.&#x20;Appl. Toxicol.</source>, <volume>30</volume>, <fpage>183</fpage>&#x2013;<lpage>196</lpage>. <pub-id pub-id-type="doi">10.1002/jat.1504</pub-id> </citation>
</ref>
<ref id="B4">
<citation citation-type="journal">
<collab>EFSA</collab>. <year>2016</year>. &#x201c;<article-title>Guidance on the Establishment of the Residue Definition for Dietary Assessment: EFSA Panel on Plant Protect Products and Their Residues (PPR)</article-title>.&#x201d; <source>EFSA J.</source>, <volume>14</volume>. <fpage>1</fpage>&#x2013;<lpage>12</lpage>. <pub-id pub-id-type="doi">10.2903/j.efsa.2016.4549</pub-id> </citation>
</ref>
<ref id="B5">
<citation citation-type="journal">
<collab>EFSA</collab>. <year>2011</year>. &#x201c;<article-title>Scientific Opinion on Pyrrolizidine Alkaloids in Food and Feed</article-title>.&#x201d; <source>EFSA J.</source> <volume>9</volume>: <fpage>1</fpage>&#x2013;<lpage>134</lpage>. <pub-id pub-id-type="doi">10.2903/j.efsa.2011.2406</pub-id> </citation>
</ref>
<ref id="B6">
<citation citation-type="book">
<collab>EMA</collab> (<year>2020</year>). <source>Public Statement on the Use of Herbal Medicinal Products Containing Toxic, Unsaturated Pyrrolizidine Alkaloids (PAs) Including Recommendations Regarding Contamination of Herbal Medicinal Products with Pyrrolizidine Alkaloids</source>. <publisher-loc>Amsterdam</publisher-loc>: <publisher-name>Committee on Herbal Medicinal Products (HMPC), EMA/HMPC/893108/2011 Rev. 1</publisher-name>.</citation>
</ref>
<ref id="B7">
<citation citation-type="book">
<collab>European Chemical Agency</collab> (<year>2017</year>). <source>Guidance on Information Requirements and Chemical Safety Assessment</source>. <comment>Chapter R.7a: Endpoint Specific Guidance. European chemical agency guidance on information requirements</comment>. <publisher-loc>Helsinki</publisher-loc>: <publisher-name>ECHA</publisher-name>. <pub-id pub-id-type="doi">10.2823/337352</pub-id> </citation>
</ref>
<ref id="B8">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Fu</surname>
<given-names>P. P.</given-names>
</name>
<name>
<surname>Xia</surname>
<given-names>Q.</given-names>
</name>
<name>
<surname>Lin</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Chou</surname>
<given-names>M. W.</given-names>
</name>
</person-group> <year>2004</year>. <article-title>Pyrrolizidine Alkaloids-Ggenotoxicity, Metabolism Enzymes, Metabolic Activation, and Mechanisms</article-title>.&#x201d; <source>Drug Metab. Rev.</source> <volume>36</volume>: <fpage>1</fpage>&#x2013;<lpage>55</lpage>. <pub-id pub-id-type="doi">10.1081/dmr-120028426</pub-id> </citation>
</ref>
<ref id="B9">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Hadi</surname>
<given-names>N. S. A.</given-names>
</name>
<name>
<surname>Bankoglu</surname>
<given-names>E. E.</given-names>
</name>
<name>
<surname>Schott</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Leopoldsberger</surname>
<given-names>E.</given-names>
</name>
<name>
<surname>Ramge</surname>
<given-names>V.</given-names>
</name>
<name>
<surname>Kelber</surname>
<given-names>O.</given-names>
</name>
<etal/>
</person-group> <year>2021</year>. &#x201c;<article-title>Genotoxicity of Selected Pyrrolizidine Alkaloids in Human Hepatoma Cell Lines HepG2 and Huh6</article-title>.&#x201d; <source>Mutat. Research/Genetic Toxicol. Environ. Mutagenesis</source> <volume>861-862</volume>: <fpage>503305</fpage>. <pub-id pub-id-type="doi">10.1016/j.mrgentox.2020.503305</pub-id> </citation>
</ref>
<ref id="B10">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Hansen</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Mika</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Schroeter</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Sutter</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>ter Laak</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Steger-Hartmann</surname>
<given-names>T.</given-names>
</name>
<etal/>
</person-group> (<year>2009</year>). <article-title>Benchmark Data Set for In Silico Prediction of Ames Mutagenicity</article-title>. <source>J.&#x20;Chem. Inf. Model.</source> <volume>49</volume> (<issue>9</issue>), <fpage>2077</fpage>&#x2013;<lpage>2081</lpage>. <pub-id pub-id-type="doi">10.1021/ci900161g</pub-id> </citation>
</ref>
<ref id="B11">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Hartmann</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Witte</surname>
<given-names>L.</given-names>
</name>
</person-group> (<year>1995</year>). &#x201c;<article-title>Chemistry, Biology and Chemoecology of the Pyrrolizidine Alkaloids</article-title>,&#x201d; in <source>In <italic>Alkaloids: Chemical and Biological Perspectives</italic>
</source>. Editor <person-group person-group-type="author">
<name>
<surname>Pelletier</surname>
<given-names>S. W.</given-names>
</name>
</person-group> (<publisher-loc>London, New York</publisher-loc>: <publisher-name>Pergamon</publisher-name>), <fpage>155</fpage>&#x2013;<lpage>233</lpage>. <pub-id pub-id-type="doi">10.1016/b978-0-08-042089-9.50011-5</pub-id> </citation>
</ref>
<ref id="B12">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Helma</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Vorgrimmler</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Gebele</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>G&#xfc;tlein</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Engeli</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Zarn</surname>
<given-names>J.</given-names>
</name>
<etal/>
</person-group> (<year>2018</year>). <article-title>Modeling Chronic Toxicity: A Comparison of Experimental Variability with (Q)SAR/Read-Across Predictions</article-title>. <source>Front. Pharmacol.</source> <volume>9</volume>, <fpage>413</fpage>. <pub-id pub-id-type="doi">10.3389/fphar.2018.00413</pub-id> </citation>
</ref>
<ref id="B13">
<citation citation-type="book">
<collab>HMPC</collab> (<year>2014</year>). <source>Public Statement on the Use of Herbal Medicinal Products Containing Toxic, Unsaturated Pyrrolizidine Alkaloids (PAs)</source>. <publisher-loc>London</publisher-loc>: <publisher-name>Committee on Herbal Medicinal Products (HMPC)</publisher-name>. <comment>EMA/HMPC/893108/2011</comment>.</citation>
</ref>
<ref id="B14">
<citation citation-type="book">
<collab>ICH</collab> (<year>2017</year>). <source>ICH guideline M7(R1) on Assessment and Control of DNA Reactive (Mutagenic) Impurities in Pharmaceuticals to Limit Potential Carcinogenic Risk</source>. <publisher-loc>London</publisher-loc>: <publisher-name>European Medicines Agency, Committee for Human Medicinal Products</publisher-name>. <comment>EMA/CHMP/ICH/83812/2015</comment>.</citation>
</ref>
<ref id="B15">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Kazius</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>McGuire</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Bursi</surname>
<given-names>R.</given-names>
</name>
</person-group> (<year>2005</year>). <article-title>Derivation and Validation of Toxicophores for Mutagenicity Prediction</article-title>. <source>J.&#x20;Med. Chem.</source> <volume>48</volume>, <fpage>312</fpage>&#x2013;<lpage>320</lpage>. <pub-id pub-id-type="doi">10.1021/jm040835a</pub-id> </citation>
</ref>
<ref id="B16">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Langel</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Ober</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Pelser</surname>
<given-names>P. B.</given-names>
</name>
</person-group> (<year>2011</year>). <article-title>The Evolution of Pyrrolizidine Alkaloid Biosynthesis and Diversity in the Senecioneae</article-title>. <source>Phytochem. Rev.</source> <volume>10</volume>, <fpage>3</fpage>&#x2013;<lpage>74</lpage>. <pub-id pub-id-type="doi">10.1007/s11101-010-9184-y</pub-id> </citation>
</ref>
<ref id="B17">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Li</surname>
<given-names>Y. H.</given-names>
</name>
<name>
<surname>Kan</surname>
<given-names>W. L.</given-names>
</name>
<name>
<surname>Li</surname>
<given-names>N.</given-names>
</name>
<name>
<surname>Lin</surname>
<given-names>G.</given-names>
</name>
</person-group> <year>2013</year>. &#x201c;<article-title>Assessment of Pyrrolizidine Alkaloid-Induced Toxicity in an <italic>In Vitro</italic> Screening Model</article-title>.&#x201d; <source>J.&#x20;Ethnopharmacol</source> <volume>150</volume> (<issue>2</issue>): <fpage>560</fpage>&#x2013;<lpage>567</lpage>. <pub-id pub-id-type="doi">10.1016/j.jep.2013.09.010</pub-id> </citation>
</ref>
<ref id="B18">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Lin</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Cui</surname>
<given-names>Y. Y.</given-names>
</name>
<name>
<surname>Hawes</surname>
<given-names>E. M.</given-names>
</name>
</person-group> (<year>1998</year>). <article-title>Microsomal Formation of a Pyrrolic Alcohol Glutathione Conjugate of Clivorine. Firm Evidence for the Formation of a Pyrrolic Metabolite of an Otonecine-type Pyrrolizidine Alkaloid</article-title>. <source>Drug Metab. Dispos</source> <volume>26</volume> (<issue>2</issue>), <fpage>181</fpage>&#x2013;<lpage>184</lpage>. </citation>
</ref>
<ref id="B19">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Louisse</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Rijkers</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Stoopen</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Holleboom</surname>
<given-names>W. J.</given-names>
</name>
<name>
<surname>Delagrange</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Molthof</surname>
<given-names>E.</given-names>
</name>
<etal/>
</person-group> <year>2019</year>. &#x201c;<article-title>Determination of Genotoxic Potencies of Pyrrolizidine Alkaloids in HepaRG Cells Using the &#x3b3;H2AX Assay</article-title>.&#x201d; <source>Food Chem. Toxicol.</source> <volume>131</volume>: <fpage>110532</fpage>. <pub-id pub-id-type="doi">10.1016/j.fct.2019.05.040</pub-id> </citation>
</ref>
<ref id="B20">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Maaten</surname>
<given-names>L. J.&#x20;P. van. der.</given-names>
</name>
<name>
<surname>Hinton</surname>
<given-names>G. E.</given-names>
</name>
</person-group> (<year>2008</year>). <article-title>Visualizing Data Using T-SNE</article-title>. <source>J.&#x20;Machine Learn. Res.</source> <volume>9</volume>, <fpage>2579</fpage>&#x2013;<lpage>2605</lpage>. </citation>
</ref>
<ref id="B21">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Mattocks</surname>
<given-names>A. R.</given-names>
</name>
</person-group> (<year>1986</year>). <source>Chemistry and Toxicology of Pyrrolizidine Alkaloids</source>. <publisher-loc>London, Orlando</publisher-loc>: <publisher-name>Academic Press</publisher-name>.</citation>
</ref>
<ref id="B22">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Merz</surname>
<given-names>K. H.</given-names>
</name>
<name>
<surname>Schrenk</surname>
<given-names>D.</given-names>
</name>
</person-group> <year>2016</year>. &#x201c;<article-title>Interim Relative Potency Factors for the Toxicological Risk Assessment of Pyrrolizidine Alkaloids in Food and Herbal Medicines</article-title>.&#x201d; <source>Toxicol. Lett.</source> <volume>263</volume>: <fpage>44</fpage>&#x2013;<lpage>57</lpage>. <pub-id pub-id-type="doi">10.1016/j.toxlet.2016.05.002</pub-id> </citation>
</ref>
<ref id="B23">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>O&#x27;Boyle</surname>
<given-names>N. M.</given-names>
</name>
<name>
<surname>Banck</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>James</surname>
<given-names>C. A.</given-names>
</name>
<name>
<surname>Morley</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Vandermeersch</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Hutchison</surname>
<given-names>G. R.</given-names>
</name>
</person-group> <year>2011</year>. &#x201c;<article-title>Open Babel: An Open Chemical Toolbox</article-title>.&#x201d; <source>J.&#x20;Cheminform</source> <volume>3</volume> (<issue>1</issue>): <fpage>33</fpage>. <pub-id pub-id-type="doi">10.1186/1758-2946-3-33</pub-id> </citation>
</ref>
<ref id="B24">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Piegorsch</surname>
<given-names>W. W.</given-names>
</name>
<name>
<surname>Zeiger</surname>
<given-names>E.</given-names>
</name>
</person-group> (<year>1991</year>). &#x201c;<article-title>Measuring Intra-assay Agreement for the Ames Salmonella Assay</article-title>,&#x201d; in <source>In <italic>Statistical Methods In Toxicology, Lecture Notes In Medical Informatics</italic>
</source>. Editor <person-group person-group-type="editor">
<name>
<surname>Hotorn</surname>
<given-names>L.</given-names>
</name>
</person-group> (<publisher-name>Springer-Verlag</publisher-name>), <fpage>35</fpage>&#x2013;<lpage>41</lpage>. <pub-id pub-id-type="doi">10.1007/978-3-642-48736-1_5</pub-id> </citation>
</ref>
<ref id="B25">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Ruan</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Yang</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Fu</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Ye</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Lin</surname>
<given-names>G.</given-names>
</name>
</person-group> (<year>2014</year>). <article-title>Metabolic Activation of Pyrrolizidine Alkaloids: Insights into the Structural and Enzymatic Basis</article-title>. <source>Chem. Res. Toxicol.</source> <volume>27</volume>, <fpage>1030</fpage>&#x2013;<lpage>1039</lpage>. <pub-id pub-id-type="doi">10.1021/tx500071q</pub-id> </citation>
</ref>
<ref id="B26">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Rubiolo</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Pieters</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Calomme</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Bicchi</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Vlietinck</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Vanden Berghe</surname>
<given-names>D.</given-names>
</name>
</person-group> <year>1992</year>. &#x201c;<article-title>Mutagenicity of Pyrrolizidine Alkaloids in the Salmonella Typhimurium/Mammalian Microsome System</article-title>.&#x201d; <source>Mutat. Res.</source>, <volume>281</volume>: <fpage>143</fpage>&#x2013;<lpage>147</lpage>. <pub-id pub-id-type="doi">10.1016/0165-7992(92)90050-r</pub-id> </citation>
</ref>
<ref id="B27">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Rutz</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Gao</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>K&#xfc;pper</surname>
<given-names>J.&#x20;H.</given-names>
</name>
<name>
<surname>Schrenk</surname>
<given-names>D.</given-names>
</name>
</person-group> <year>2020</year>. &#x201c;<article-title>Structure-Dependent Genotoxic Potencies of Selected Pyrrolizidine Alkaloids in Metabolically Competent Hepg2 Cells</article-title>.&#x201d; <source>Arch. Toxicol.</source>, <volume>94</volume>: <fpage>4159</fpage>&#x2013;<lpage>4172</lpage>. <pub-id pub-id-type="doi">10.1007/s00204-020-02895-z</pub-id> </citation>
</ref>
<ref id="B28">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Sch&#xf6;ning</surname>
<given-names>V.</given-names>
</name>
<name>
<surname>Hammann</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Peinl</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Drewe</surname>
<given-names>J.</given-names>
</name>
</person-group> <year>2017</year>. <article-title>Editor&#x27;s Highlight: Identification of Any Structure-specific Hepatotoxic Potential of Different Pyrrolizidine Alkaloids Using Random Forests and Artificial Neural Networks</article-title>.&#x201d; <source>Toxicol. Sci.</source>, <volume>160</volume>: <fpage>361</fpage>&#x2013;<lpage>370</lpage>. <pub-id pub-id-type="doi">10.1093/toxsci/kfx187</pub-id> </citation>
</ref>
<ref id="B29">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Wang</surname>
<given-names>Y. P.</given-names>
</name>
<name>
<surname>Yan</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Fu</surname>
<given-names>P. P.</given-names>
</name>
<name>
<surname>Chou</surname>
<given-names>M. W.</given-names>
</name>
</person-group> (<year>2005</year>). <article-title>Human Liver Microsomal Reduction of Pyrrolizidine Alkaloid N-Oxides to Form the Corresponding Carcinogenic Parent Alkaloid</article-title>. <source>Toxicol. Lett.</source> <volume>155</volume> (<issue>3</issue>), <fpage>411</fpage>&#x2013;<lpage>420</lpage>. <pub-id pub-id-type="doi">10.1016/j.toxlet.2004.11.010</pub-id> </citation>
</ref>
<ref id="B30">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Weininger</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Weininger</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Weininger</surname>
<given-names>J.&#x20;L.</given-names>
</name>
</person-group> <year>1989</year>. &#x201c;<article-title>SMILES. 2. Algorithm for Generation of Unique Smiles Notation</article-title>.&#x201d; <source>J.&#x20;Chem. Inf. Comput. Sci.</source> <volume>29</volume>: <fpage>97</fpage>&#x2013;<lpage>101</lpage>. <pub-id pub-id-type="doi">10.1021/ci00062a008</pub-id> </citation>
</ref>
<ref id="B31">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Willighagen</surname>
<given-names>E. L.</given-names>
</name>
<name>
<surname>Mayfield</surname>
<given-names>J.&#x20;W.</given-names>
</name>
<name>
<surname>Alvarsson</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Berg</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Carlsson</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Jeliazkova</surname>
<given-names>N.</given-names>
</name>
<etal/>
</person-group> <year>2017</year>. &#x201c;<article-title>The Chemistry Development Kit (Cdk) V2.0: Atom Typing, Depiction, Molecular Formulas, and Substructure Searching</article-title>.&#x201d; <source>J.&#x20;Cheminform.</source>, <volume>9</volume>, <fpage>33</fpage>. <pub-id pub-id-type="doi">10.1186/s13321-017-0220-4</pub-id> </citation>
</ref>
<ref id="B32">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Xia</surname>
<given-names>Q.</given-names>
</name>
<name>
<surname>Zhao</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Von Tungeln</surname>
<given-names>L. S.</given-names>
</name>
<name>
<surname>Doerge</surname>
<given-names>D. R.</given-names>
</name>
<name>
<surname>Lin</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Cai</surname>
<given-names>L.</given-names>
</name>
<etal/>
</person-group> <year>2013</year>. &#x201c;<article-title>Pyrrolizidine Alkaloid-Derived DNA Adducts as a Common Biological Biomarker of Pyrrolizidine Alkaloid-Induced Tumorigenicity</article-title>.&#x201d; <source>Chem. Res. Toxicol.</source>, <volume>26</volume>: <fpage>1384</fpage>&#x2013;<lpage>1396</lpage>. <pub-id pub-id-type="doi">10.1021/tx400241c</pub-id> </citation>
</ref>
<ref id="B33">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Yan</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Xia</surname>
<given-names>Q.</given-names>
</name>
<name>
<surname>Chou</surname>
<given-names>M. W.</given-names>
</name>
<name>
<surname>Fu</surname>
<given-names>P. P.</given-names>
</name>
</person-group> <year>2008</year>. &#x201c;<article-title>Metabolic Activation of Retronecine and Retronecine N-Oxide - Formation of DHP-Derived DNA Adducts</article-title>.&#x201d; <source>Toxicol. Ind. Health</source>. <volume>24</volume>(<issue>3</issue>): <fpage>181</fpage>&#x2013;<lpage>188</lpage>. <pub-id pub-id-type="doi">10.1177/0748233708093727</pub-id> </citation>
</ref>
<ref id="B34">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Yap</surname>
<given-names>C. W.</given-names>
</name>
</person-group> <year>2011</year>. &#x201c;<article-title>PaDEL-descriptor: An Open Source Software to Calculate Molecular Descriptors and Fingerprints</article-title>.&#x201d; <source>J.&#x20;Comput. Chem.</source>, <volume>32</volume>: <fpage>1466</fpage>&#x2013;<lpage>1474</lpage>. <pub-id pub-id-type="doi">10.1002/jcc.21707</pub-id> </citation>
</ref>
</ref-list>
</back>
</article>