The datasets for the training and testing of the models were collected from different sources (online tools as well as publications). The data were cleaned by using an in-house system combining the molecular operating environment (MOE 2014.09) (Molecular Operating Environment, 2014) wash option and the Atkinson Standardiser (https://github.com/flatkinson/standardiser). This approach was used for some of the datasets and for others the cleaning procedure of already published papers were used (Pinto et al., 2012; Kotsampasakou et al., 2015). In general, duplicates were removed from the dataset, including pairs of stereoisomers. Further, if these compounds share the same class label, one of the compounds was kept. A detailed list of references as well as the number of compounds revealed after the preprocessing of the data is given in Table S1. In the sections Transporter Models and Hepatotoxicity Models, the data curation as well as the model generation for the specific endpoints is given. The datasets are available in the Supplementary Material (Data Sheet 2).

The web service allows for the prediction of interactions between a small molecule and eight different liver transporters (Figure 1, transporters marked in blue). The lack of publicly available data for the other transporters explains the absence of respective models in the Vienna LiverTox Workspace.

All the models predicting whether a compound will be a substrate of a transporter (BCRP, P-gp, BSEP, MRP2, and MRP3) were built upon a dataset correlating expression levels of 47 ABC-transporters with drug toxicity, which can then be used to infer transported vs. non-transported compounds (Szakács et al., 2004). For the transport inhibition models (BCRP, P-gp, BSEP, MRP3, MRP4, OATP1B1, and OATP1B3) the datasets were collected from literature and, if necessary, manually aggregated. In both cases, the models predict a binary outcome: the query compound is a substrate or not, or an inhibitor or not.

For chemistry encoding of the compounds, we used circular fingerprints or 2D molecular descriptors as implemented in RDKit version 2015.03.1 (https://www.rdkit.org/). Different machine learning algorithms were applied and the one giving the best cross-validation results was kept as final model. Especially for the transport prediction, a heavy class imbalancy (most of the drugs in the training set were non-substrates) was noted, which was handled by MetaCost (Domingos, 1999). The exact methodology and cross-validation performance for each individual transporter model is described in the documentation available at https://livertox.univie.ac.at, and an overview is given in Table 1. In some cases, external test sets were collected from Metrabase (Mak et al., 2015) or from recent publications (time-split evaluation).

Three models in the web service can be used to assess human liver damage potentially caused by a test compound: hyperbilirubinemia, cholestasis, and drug-induced liver injury (DILI).

For hyperbilirubinemia, 835 compounds were taken from Kotsampasakou et al. (2017a) and the modeling methodology was kept as in Kotsampasakou et al. (2017b): ECFP-like fingerprints were computed with RDKit (https://www.rdkit.org/), then a combination of feature selection, MetaCost, and support vector machines with RBF kernel was used for learning. The cholestasis model uses data from Kotsampasakou and Ecker (2017) and a combination of MetaCost and a tree algorithm to predict whether a compound is likely to cause cholestasis or not. Finally, the DILI model is based on a 966-compound dataset carefully compiled from literature (Kotsampasakou et al., 2017c). RDKit molecular descriptors and a random forest of 500 trees are used for modeling.

The Vienna LiverTox Workspace has been implemented as Python/php based web service. It consists of two parts, namely the backend and the frontend. The backend consists of a docker image which runs the machine learning models on an input SD-File. It consists of a Python Flask server (https://palletsprojects.com/p/flask/) which processes the requests from the frontend. Each request consists of one or more input molecules and a list of models to run the predictions on. The frontend, also a docker image, is based on the CakePHP framework (https://cakephp.org/) and is responsible for the user interface (UI), which sends the request to the backend and displays the results. The web service provides, after a login, the possibility to upload a SD-File of 10 compounds. The service can also be used without logging in but then it is only possible to draw and predict a single molecule. JSME (Bienfait and Ertl, 2013) is used as drawing tool. The web service runs on an Ubuntu Linux based server with two twelve-core Intel Xeon 64bit processors and 128 GB RAM, and is hosted at the University of Vienna by the Pharmacoinformatics Research Group.

The models use the RDKit library (https://www.rdkit.org/) (version 2015.03.1) for computing the descriptors and handling the chemistry aspects, while the Weka (Hall et al., 2009) (version 3.7.11) and scikit-learn (Pedregosa et al., 2011) (version 0.14.1) libraries are used to train and run the predictive models. The models also include a compound cleaning step, implemented with the Atkinson Standardiser (https://github.com/flatkinson/standardiser) (Figure 2).

The steps performed during model building and a test compound prediction are shown in Figure 2. In both cases, the compound is standardized and the molecular descriptors are calculated. In the case of the model generation, this allows the training of the model and its development (left workflow). For the prediction of a test compound, the descriptors are passed to the available model to predict its class affiliation (right workflow).

In general, the output of the model gives, in addition to the class prediction, the actual score. This score is a numerical value between 0 and 1, and roughly corresponds to a probability of being active (inhibitor, substrate, or toxic compound). Therefore, a value close to 1 indicates substrate/inhibitor/toxic properties, a value close to 0 annotates for non-substrates/non-inhibitors/non-toxic.

The Applicability Domain (AD) is used to validate the reliability of a given prediction model. It defines whether a dataset of interest is in or out of domain, meaning if it falls within the chemical space of the model or not. If it is out of Domain, the prediction cannot be regarded as reliable.

In our study, an Applicability Domain model, using the approach of Sahigara et al. (2013), was created for each transporter with the respective training set. RDKit descriptors were used as molecule representation. This approach combines the classical, widely used k-Nearest Neighbor (k-NN) method with a probability density function estimation. It uses three stages to determine the reliable space of a prediction model. First, a set of thresholds is defined depending on the diverse densities of the training set by considering the 15 k-nearest neighbors using Euclidean distances. This allows the AD to consider a dense and sparse training region (The threshold defines if a test sample can be reliable predicted). In a next step a decision rule is derived to filter out outlier molecules. Finally, the reliability of the AD is tested by looking at the model statistics and prediction errors. This feature is not yet implemented in the Web service, but will be available soon.

The performance of the models was estimated by calculating statistical performance metrics using a 10-fold cross-validation. The results are provided in Table 2. The overall accuracy, corresponding to the rate of correct predictions, ranges from 0.59 to 0.87. Also, the sensitivity of the models was calculated (0.57–0.85). This parameter gives the number of actual positives that are correctly identified and is expressed by the number of true positives divided by the number of positive predictions. Further, the number of actual negatives was determined by the number of true negatives divided by the number of negative predictions. The so-called specificity ranges from 0.56 to 0.90. To estimate a metric for the quality of the models, the Matthews correlation coefficient (MCC) and the Area under the Receiver Operating Characteristics curve (ROC AUC) were determined. The MCC is a number between −1 and 1 where 0 indicates a prediction equal to a random prediction and 1 indicates a perfect prediction, whereas −1 is a complete miss. The scores for our models are in the range of 0.20–0.76. The ROC AUC measures the ability of the model to distinguish between negatives and positives, while a higher value indicates a better performance. In the models provided on the Vienna LiverTox Workspace the ROC AUC ranges from 0.64 to 0.94. Furthermore, if data was available, the models were also validated with one or more external test sets. For more details see the documentation on the website.

In this section, we briefly detail how predictions can be generated for a given compound using the LiverTox web service. In first instance, the compound is drawn or its SMILES string is pasted in the Molecule Editor (Bienfait and Ertl, 2013). Then the models can be selected on the left panel, either one by one or all at the same time (Figure 3, left side).

By clicking on the SUBMIT button, the data is sent to the backend server where the predictions are running. Upon completion of the calculations, a table listing the different models and corresponding outputs will appear (Figure 3, right side). The second column in the results table corresponds to the binary classification, while the third column “Score” gives the actual output of the model, which corresponds to a probability of being active. For example, for BCRP inhibition and transport and for DILI, the output score is close to 0.5 (which is the threshold used to separate predicted actives from predicted inactives), which indicates an uncertainty of these three models for the particular query compound. Propafenone derivatives are frequently reported as inhibitors of P-gp, and indeed the P-gp inhibition model predicts this particular one to be an inhibitor with a high score (0.93).

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.