The landscape of drug discovery is a complex and dynamic field that necessitates strategies for identifying and optimizing molecules. Ranging from the broad exploration of chemical space to nuanced structural modifications, these strategies present unique advantages and challenges. These are exemplified in Figure 2. This section outlines key strategies commonly used in the field.

Molecules, being tangible entities that cannot be directly manipulated through computations, need methods to represent their properties and characteristics. These range from quantum mechanics-based electron distribution models to basic chemical formulas. The representations are communicated through various encodings, such as the structural formula, which visually outlines molecular topologies. Encodings for molecular structure generation are typically view atoms as indivisible units connected by covalent bonds.

String-based encodings like the Wiswesser Line Notation (42) and SMILES (43) represent molecular topology of organic molecules through character sequences. However, limitations in SMILES led to the development of DeepSMILES (44), ensuring syntactic correctness, and SELFIES (45), ensuring both syntactical validity and chemical space adherence.

Graph-based encodings depict molecules as nodes (atoms) linked by edges (bonds) (46), with 2D versions showing connections and 3D versions adding spatial information. While 3D encodings offer more detail, they also add complexity in algorithm training.

Feature-based encodings use molecular descriptors (47) or fingerprints (48) to detail molecules based on properties or experimental data, useful for comparison and property prediction but less effective for structure creation.

AI can define encodings by converting molecular structures into a continuous numerical space, which can be decoded back (8). Trained on extensive molecular datasets, they capture essential features and allow for the exploration of new molecular structures. Challenges in learning encodings include accurately representing chemical space and ensuring relevance to drug development, as molecules close in latent space might have divergent biological and chemical properties.

Access to extensive data is crucial to train algorithms. Key to this is the availability of public databases. Prominent among these are ChEMBL (49), PubChem (50), BindingDB (51), and DrugBank (52), which offer essential bioactivity data for drug discovery. ChemSpider (53) aggregates information on molecular properties and available compounds. CompTox (54) is a resource for toxicity and environmental hazards. GDB17 (55) lists organic compounds up to 17 atoms, while QM9 (56) provides a subset with quantum mechanically determined molecular conformations. ZINC (57) catalogs commercially available compounds, and Enamine’s REAL database lists synthesizable molecules.

The Protein Data Bank (PDB) (58) provides structural details of proteins and nucleic acids, crucial for understanding molecular interactions. Uniprot (59) offers protein sequences and functional annotations, which help target selection and validation. The Therapeutic Target Database (TTD) (60) focuses on well-studied therapeutic targets.

The field of de novo drug design has been significantly advanced by generative algorithms, each offering unique advantages and applications. A training dataset of molecular structures is essential for these techniques. These use a set of known molecules to train a generative algorithm which is then used to generate novel molecular structures. Key AI methods are summarized below, with a detailed discussion available in the comprehensive review by Martinelli (7).

Rule-based techniques, formulated before trained generative algorithms, provide a structured approach to exploring the chemical space. Combinatorial chemistry principles are used to construct a well-defined chemical space and a set of traversal rules for these techniques. Combinatorial chemistry involves the creation of molecules by combining molecular substructures or atoms based on human-defined rules. These methodologies differ according to the rules employed in the combinatorial process and the exploratory algorithms used to select molecules to analyze from these potentially extensive sets.

The chemical space can be defined in several ways:

The main strategies employed to explore the defined chemical space can be grouped in two categories:

Active Learning (AL) is a novel introduction to the field of de Novo Drug Design for the exploration of chemical space. AL operates on the principle that models can achieve higher accuracy with fewer data by intelligently selecting their training samples. Central to AL is the concept of uncertainty. AL addresses epistemic uncertainty, related to the lack of knowledge in certain areas of the modeled space by the model (88). For instance, an unfamiliar molecular structure can increase uncertainty in predictions due to insufficient knowledge in that area. AL reduces uncertainty by adding new data to the training set.

AL has traditionally been used to improve model predictions, but is now also used to explore the chemical space. It identifies and investigates molecules linked with high uncertainty, assumed to be those presenting less-explored molecular structures. By doing so, AL enables more efficient exploration with fewer experiments. These approaches employ AI techniques like Bayesian machine learning algorithms and Ensemble learning to estimate uncertainty. The process follows an iterative process, it identifies molecules with high uncertainty, tests them and adds the results to the training set. This results in the refinement of the model and the expansion of the training set, which is effectively the set of screened compounds. Predictions from the model are generally discarded, as the method is primarily used to explore and not predict. Unlike the methods previously outlined, this methodology is purely explorative and does not generate novel structures, the structures for this exploration can come from different sources, such as databases, combinatorial chemistry, or generative algorithms. Strategies that include molecules predicted to have desirable properties can be used to combine the exploration and optimization steps. In fact, it has been shown that purely exploratory approaches effectively explore the chemical space but do not identify many active compounds (89). A schematic representation of the process using AL is exemplified in Figure 3.

However, AL requires regular generation of new data, making it more suited for virtual rather than experimental screening. While these strategies do not improve the accuracy of the prediction, they optimize the quantity of tested compounds; making it possible to screen larger libraries. Despite this, it has been applied in the context of automated experimental testing (90).

The application of AL to drug discovery has been explored since the early 2000s, but most of these studies were retrospective in nature (91). The focus has only recently shifted towards identifying and evaluating specific AL techniques that could be effectively integrated into the drug discovery process (33), but it is still an emerging field with slow and limited adoption (7, 89). Three experimental high-throughput screening results datasets were used to demonstrate the potential of AL, showing a sixfold increase in Enrichment Factor over traditional virtual screening methods (89).

Rule-based filters serve as rapid evaluation systems to categorize molecules based on predetermined criteria related to properties like bioactivity or synthetic feasibility. These systems typically use two-dimensional molecular structures to pinpoint promising candidates and weed out potential “false positives”—molecules that initially appear viable but fail in later testing stages. Tools like Lipinski’s Rule of 5 (92) assess oral bioavailability, while the QED metric (93) estimates therapeutic potential. Structural filters, such as PAINS (94) and BRENK (95), identify molecules that could interfere with bioassays or prove unsuitable as lead candidates. Additionally, instruments like the Synthetic Accessibility Score (96) and automated retrosynthetic analysis (97–99) aid in eliminating hard-to-synthesize molecules. However, it’s important to recognize that while these methods are broadly useful, they are primarily tailored for known drugs and targets. This specialization can limit their applicability in assessing novel structures (100).

Ligand-based virtual screening, employed in drug discovery when a target’s structure is unknown, it is computationally simple and identifies ligands with similar binding characteristics. Its effectiveness hinges on the availability of known ligands interacting with the target, a limitation when such data is scarce. In de novo design, focusing on structurally similar compounds is less preferred and as such these methodologies are often not appropriate.

Key techniques in this approach include Quantitative Structure-Activity Relationship (QSAR), which correlates molecular structure with biological activity, aiding in predicting the behavior of new molecules in terms of activity and toxicity (62, 101, 102). Pharmacophore Modeling identifies essential molecular features for biological activity, using known ligands to create models for finding new molecules with similar characteristics (31, 103, 104). Structural Similarity assesses the resemblance between molecules, facilitating the discovery of new compounds similar to active ones, or exploring diverse molecular structures for further study (17, 48, 105).

Structure-based virtual screening is a technique in drug discovery that uses the 3D structures of both the drug molecule and its target. It can use either experimentally determined structures or accurate computational models of the target, made through methods like homology modeling or AI tools like AlphaFold (106, 107). This approach can be useful even without data on binding ligands.

Molecular Docking is commonly used in structure-based virtual screening to predict binding modes (108). This knowledge aids in evaluating the interaction’s quality and provides comparative insights based on other known ligands (109–111). Molecular simulations, typically conducted through Molecular Dynamics or Monte Carlo techniques, aim to predict the structural, thermodynamic, and kinetic attributes of the molecule-target interaction (112). These simulations can be computationally demanding, making them most effective when focused on a refined list of candidates. One common application is the estimation of the Free Energy of binding through methods like Alchemical Free Energy, Thermodynamic integration or Free Energy Perturbation (113).

The application of de novo techniques in oncological drug design has primarily focused on kinase targets, largely due to the extensive research and abundant data availability (124). As AI methodologies advance, there’s a growing expectation that these tools will be applied to lesser-studied targets. While these studies have yielded successful results, the molecules generated do not show significant structural or binding mode differences compared to known ligands. Consequently, de novo techniques have shown more success in drug optimization rather than in the discovery of entirely novel hits.

In 2021, Yu et al. (22) undertook a scaffold hopping campaign to identify novel structures for JAK1 inhibition. To enhance this process, they used a neural network with Graph-Based Variational Autoencoders, training the model on scaffolds derived from drug-like compounds and fine-tuning it with the structure and bioactivity of known kinase inhibitors. The objective was to hop through different scaffolds while retaining similar side chains, as these two were encoded separately into the model. With a known JAK1 inhibitor as the starting scaffold, they generated 30,000 molecular structures. These underwent a screening process considering physicochemical properties, structural alerts, and their alignment with established inhibitors. A QSAR model and molecular docking were used to prioritize 25 molecules. 7 of these were synthesized, all of which demonstrated JAK1 inhibitory potential in experimental trials. While the inhibitory activity of the identified molecules is significant, they did not conduct kinase screening tests to assess selectivity, and in consequence it is difficult to appreciate the real value of the compounds.

In 2022, Jang et al. focused on FLT3 (125), a critical kinase in hematopoiesis. When mutated in acute myeloid leukemia (AML), it is often associated with adverse outcomes. Their approach involved enhancing the FLT3 selectivity of a previously active molecule against breast cancer cells, which was predicted to interact with FLT3. A deep learning generative model was trained on drug-like SMILES structures and fine-tuned with known FLT3 binders. The model generated over 10,000 structures. These were filtered, and the resulting molecules were ranked based on their binding affinities derived from Alchemical Free Energy calculations. The most promising compound was synthesized and tested in cellular cultures and on the protein. It showed affinity for the mutated FLT3 variant and inhibited FLT3-mutated AML cell proliferation. The study, however, has limitations: it leaned heavily on molecular docking simulations without experimental assays for kinase selectivity, and did not test on healthy cell lines, leaving potential toxicity unexplored. Additionally, the molecules produced strongly resembled known FLT3 inhibitors, questioning the novelty of molecules designed through this approach. The findings are promising, but comprehensive experimental validation is required.

In their 2023 study, Zhu et al. engaged in a ‘hit-to-lead’ campaign targeting SIK2 (25). They utilized a proprietary AI system, with an existing inhibitor as the reference, to produce molecules with varied substituents. Each molecule, out of the 5,000 candidates, was docked onto the protein structure modelled by AlphaFold and evaluated on structural and protein interaction criteria. They were then grouped into 56 clusters based on structural similarities and hydrogen bond interactions. From each cluster, two molecules were synthesized and underwent in vitro and in vivo testing. One molecule stood out for its potent inhibitory activity and ideal ADMET properties.

While the potential of AI to drive de novo molecular design is significant, this field faces several challenges. AI methodologies have proven successful in targeting well-studied proteins, yet advancements in the design of molecules significantly different from known ligands remain limited. Available methods typically optimize molecules within established chemical spaces, raising questions about their applicability to genuine de novo drug design. The challenge is determining whether the current methodologies need refinement, or if the approach itself is unsuitable for this type of problem.

A key area of focus has been trained generative algorithms, which have brought new life to de novo drug design. Their ability to explore chemical spaces beyond their training set structures is still a concern (122). Efforts are ongoing to expand the range of molecular structures these algorithms can explore (7, 126). Yet, it remains challenging to assess their effectiveness in spanning chemical space. There are metrics for comparing molecule sets, but no methodology currently enables the enumeration of the chemical space accessible to generative algorithms. Such a methodology would be crucial for comparing the diversity and breadth of chemical spaces generated by these algorithms with those from combinatorial chemistry and existing chemical databases.

Data availability is also a challenge for trained generative algorithms. The bias towards well-explored molecular structures and targets in currently available datasets can limit their ability to generate diverse and innovative molecular designs for new targets (11).

Another significant limitation is the uncertainty regarding the quantity of molecules that need to be sampled from these algorithms to find a promising hit. These algorithms are expected to guide experimental design, but there are no comprehensive benchmarks of their effectiveness. Their limited application in large-scale hit discovery programs makes it difficult to assess their suitability for extensive de novo drug discovery and the resources needed. These methods have the potential to streamline the hit discovery process and reduce the number of necessary tests. However, this efficiency must be balanced against the higher synthetic efforts required for molecules generated through these algorithms, as opposed to those identified via high-throughput screening from commercial compound libraries. While advancements in synthesizability have been made, the necessity for custom synthesis will always persist. It is important to determine the extent to which these algorithms can reduce testing needs. Virtual screening can play a significant role in filtering compounds. This process predominantly depends on structure-based methods, as ligand-based methods are less effective outside well-characterized chemical spaces. The drawback of this reliance is the increased demand for computational resources, which exceeds that of traditional virtual screening methods. This raises questions about the circumstances under which these novel AI methodologies may offer advantages over established techniques. Determining the right balance between computational effort and the efficiency of hit discovery remains a challenge.

The synthesizability of molecules generated by AI algorithms remains a challenging issue (4, 7, 97, 124). Despite this, significant progress has been made towards addressing this problem. New algorithms that can generate molecules more amenable to synthesis and improved methods for evaluating synthesizability, are expected to mitigate this limitation in the near future.

Rule-based systems offer an advantage over trained generative techniques in that they do not require training data and allow the enumeration of the explorable chemical space. Despite their longstanding presence, these systems have been primarily used to optimize hits or leads, with limited application in full de novo design. The success of these methods depends on the formulation of effective rules for generation. These rules must not only ensure synthesizability, but also ensure the creation of a diverse chemical spaces. Recent progress in this area involves generating synthon spaces from a set of generic reactions, potentially leading to more easily synthesizable molecules. The limited availability of publicly available reactions sets poses limitations, and it is unclear how much these methods improve synthesizability over other approaches.

Active Learning (AL) shows promise in optimizing virtual screenings, but integrating experimental data with computational results is a problem yet to be resolved. A methodology allowing such integration would enable a more targeted search process.

Another focus area is multi-objective optimization, which is crucial for developing effective and safe drugs. The ligands that bind effectively to the target must also have desirable physicochemical properties, selectivity, and an acceptable ADMET profile. Advances are being made in this field, using multi-objective optimization techniques from other disciplines, raising hopes for a solution in the near future.

Finally, the widespread application of these methodologies is hampered by their limited diffusion and ease of use. De novo design requires an integrated approach combining computational science, chemistry, and biology. Many models lack this interdisciplinary aspect and are therefore computationally sophisticated but less practical from a chemical or biological standpoint. This gap restricts their utility, especially for experimental chemists who require user-friendly tools. There are some commercial software with intuitive interfaces, but their use is still limited by the need for extensive computational knowledge and high costs (127, 128).

Methodologies for de novo drug design utilizing AI have been developed, yet their application in large-scale campaigns remains limited. The effectiveness of these methods compared to traditional approaches is still an open question. Therefore, a critical development area involves devising metrics to compare these AI methodologies against conventional methods and using de novo design in larger drug development campaigns.

Advancements in broadening the chemical space accessible to trained generative algorithms are being done (126). Future progress will be key in assessing whether these algorithms can create novel molecules distinct from their training data. This will clarify if such limitations are inherent to these methods or if they can be surpassed for more innovative molecular design. The capability of these algorithms to generate novel structures will significantly indicate their flexibility and influence in drug discovery.

Active Learning (AL) emerges as a potent tool in AI-based de novo drug design. Primarily used in compound screening, AL’s potential extends to optimizing identified hits—an area ripe for exploration. These methods could potentially replace evolutionary algorithms in rule-based systems, emphasizing the generation of unique or underexplored molecules. Currently, ensemble methods dominate AL, but integrating Bayesian models, renowned for their efficacy (88), could significantly enhance the exploration of chemical spaces.

Transitioning to the next phase of development, hybrid systems that merge exploratory and trained methodologies present a promising path for comprehensive chemical space exploration. Rule-based systems efficiently probe molecular structures akin to identified hits. In contrast, trained generative algorithms have proven effective in refining structures within explored chemical spaces. A cohesive de novo drug generation pipeline might first employ AL for rapid hit discovery, followed by Rule-Based systems for initial optimization. Subsequently, trained generative algorithms could refine these hits further, offering a layered approach composed of broad initial scans followed by focused lead tuning. This integrated strategy could substantially advance de novo drug design capabilities.

Furthermore, retrosynthetic analysis tools are advancing, particularly in addressing synthesizability challenges. Improvements in these tools may lessen the emphasis on generating only synthetically feasible molecules, paving the way for more creative and expansive molecular design.