Structural variants (SVs) are defined as DNA rearrangements ≥50 bp and include copy number variants (CNVs; deletions and duplications) as well as insertions, inversions, translocations, and more complex combinations of these described events (Alkan et al., 2011; Sudmant et al., 2015). Although single nucleotide variants (SNVs) were initially thought to contribute the majority of genomic variation in humans (Sachidanandam et al., 2001; Zou et al., 2020), SVs can extend to well over megabases of sequence, accounting for more varying base pairs than any other class of sequence variants (Ho et al., 2020).

Several studies have implicated SVs in human health, with associated phenotypes ranging from cognitive neurological disorders (Rovelet-Lecrux et al., 2006; Pytte et al., 2020) to obesity (Walters et al., 2013) and cancer (Li et al., 2020; Aganezov et al., 2020), among others (Weischenfeldt et al., 2013).

Despite the importance of SVs, they have been largely understudied compared to SNVs because of dominant short-read sequencing technologies hindering their identification, especially in low-complexity regions, which are known to be SV hotspots (Mills et al., 2011). Indeed, it has been shown that from a computational perspective, repeats create ambiguities in short-read alignment and assembly which, in turn, introduces errors in calling genetic variants (Treangen and Salzberg, 2011; Mantere et al., 2019).

Long-read sequencing from Pacific Biosciences and Oxford Nanopore Technologies (ONT) has emerged in recent years (Chaisson et al., 2015; Jain et al., 2016) and proved invaluable in identifying previously intractable DNA sequences (Li and Freudenberg, 2014; Bolognini et al., 2020) and close gaps in the human genome assemblies and unraveling otherwise undetected SVs at population-scale (Beyter et al., 2020; Wu et al., 2021).

The idea of sequencing DNA fragments using a protein nanopore dates back to the 1980s and culminated in the ONT MinION device being released in June 2014 (Deamer et al., 2016). A single MinION flowcell has 512 sensors collecting measurements from 2048 nanopores and currently allows us to sequence a full human genome at 3-4X coverage with read lengths up to ∼800 kbp (Jain et al., 2018). While low-coverage data can be used to detect CNVs at array resolution (Magi et al., 2019), higher throughput facilitates the resolution of the full SV spectrum with base-pair resolution and can be achieved by combining multiple MinION runs (Cretu Stancu et al., 2017) or by sequencing through the high-performance PromethION platform (De Coster et al., 2019).

Thanks to the efforts of the Human Genome Structural Variation (Chaisson et al., 2019) and Genome in a Bottle (GIAB) (Zook et al., 2020) consortia, high-coverage nanopore sequencing data have been released to the research community together with high-quality SV callsets that enable an accurate estimation of precision and recall of SV calling methods. Moreover, with many other studies to follow, such as the All of Us research program and the Human Pangenome project (De Coster et al., 2021), a throughout benchmark of available strategies for the identification and characterization of SVs from nanopore data is greatly needed.

In this article, we present an evaluation of current long-read SV calling pipelines applied to nanopore sequencing data. Specifically, we focus on germline SVs identified by read alignment–based approaches and evaluate each SV caller’s ability to detect genomic breakpoints of different SV types and size ranges and the effects of read alignment, sequencing coverage, variant allele depth, and integration of multiple call sets on SV detection and genotyping. A scalable workflow for SV calling based on the popular workflow language Snakemake (Köster and Rahmann, 2012) is available at https://github.com/davidebolo1993/EViNCe and can be used to reproduce findings described in this article and adapted to future nanopore sequencing datasets.

The evaluation workflow used in this work is outlined in short below. Additional details are provided in the accompanying Supplementary Material.

We benchmarked 5 SV calling methods, namely, Sniffles (Sedlazeck et al., 2018), SVIM (Heller and Vingron, 2019), cuteSV (Jiang et al., 2020), npInv (Shao et al., 2018), and pbsv (https://github.com/PacificBiosciences/pbsv), using real ONT PromethION data released by the GIAB consortium for the NA24385 Ashkenazim individual (Shafin et al., 2019) and synthetic ONT data generated using the SV simulator VISOR (Bolognini et al., 2019), aligned to the GRCh37 and GRCh38 versions of the human reference genome, respectively, using the long-read aligners minimap2 (Li, 2018), NGMLR (Sedlazeck et al., 2018), lra (Ren and Chaisson, 2021), and pbmm2 (https://github.com/PacificBiosciences/pbmm2) (Supplementary Datasheet S1).

By randomly down-sampling the original alignments and filtering the generated SV callsets on different numbers of reads supporting a reported SV, we evaluated the influence of various depths of coverage and variant allele depths on SV callers’ ability to detect genomic breakpoints and identify their genotype.

Precision and recall of the SV callsets generated by combining the different long-read aligners and SV callers were calculated using truvari (https://github.com/spiralgenetics/truvari) against the truth SV callsets from GIAB and VISOR. In accordance with similar studies (Gong et al., 2020), the following criteria were used to pick out true-positive calls: 1) the genomic position of the breakpoints identified for a candidate SV must be within a predefined reference distance (500 bp) from at least one SV in the truth callset, 2) the SV type reported for the candidate SV must match the SV type of the SV in the truth callset, and 3) for genotyping, the genotype of the candidate SV must match the genotype of the SV in the truth callset. Candidate SVs absent (and, for genotyping, also those not having a matching genotype) from the truth callset were considered false positives, and vice versa for false negatives.

While short-read sequencing has been considered the gold standard for the majority of sequencing projects for years (Roberts et al., 2021), such data have biases in whole-genome sequencing studies due to the uneven coverage of regions with high/low GC and difficulty of mapping short reads in low-complexity regions. Long-read sequencing has already proved invaluable in overcoming these limitations, improving on short reads for the resolution of SVs in comparative and clinical studies (Sanchis-Juan et al., 2018).

In this article, we provided a succinct yet comprehensive evaluation of long-read SV calling pipelines applied to ONT data. In particular, we focused on germline SVs, and as such, our findings are likely not reproducible in different contexts, such as somatic variant calling, for which alternative strategies exist (Shiraishi et al., 2020).

We tested four general-purpose SV callers (Sniffles, SVIM, cuteSV, and pbsv) and a tool tailored specifically to inversions (npInv) across four long-read aligners (minimap2, NGMLR, lra, and pbmm2) using both real and simulated ONT data. In particular, we used the ultra-long ONT reads released by the GIAB consortium for the NA24385 Ashkenazim individual, for which a truth set of deletions and insertions based on the integration of multiple technologies is available, and synthetic long reads generated using the SV simulator VISOR (SI00001) to complement SVs missing in the real dataset (inversions, duplications, and translocations). Also, although the NA24385 truth set from GIAB is assumed to be sufficiently complete, which is supported by the fact that the majority of the SVs identified by the different SV calling pipelines is shared with the ground truth, a consistent number of deletions and insertions identified by multiple SV callers are absent from the truth callset, suggesting that at least part of them could have been missed in the ground truth.

We first calculated the precision, recall, and F-score of the different SV calling pipelines after filtering for high-quality variants (“PASS” SVs not falling in decoy contigs and supported by at least 10 reads—which is the default for SV callers like Sniffles and cuteSV) and evaluated the impact of each SV type and various SV sizes on the SV callers’ performances. In accordance with prior evaluations (De Coster et al., 2019; Zhou et al., 2019), we observed the highest precision at SV calling with Sniffles following NGMLR alignment in both the NA24385 (∼0.96) and SI00001 (∼0.99) datasets but at a cost of low recall, with most of the false-negative SVs being shorter than 500 bp in the real dataset. However, cuteSV, SVIM, and pbsv all performed better than Sniffles in terms of the F-score across the different aligners, and Sniffles also hit the lowest F-score values at SV genotyping in both datasets. When taking into account the individual SV types, cuteSV, SVIM, and pbsv had the best performances for the detection of deletions, and cuteSV after NGMLR (NA24385 and SI0001) and minimap2 (SI00001) hit the highest F-score for the detection of insertions and duplications together with SVIM after NGMLR. For inversions, SVIM (after minimap2), Sniffles, npInv (after minimap2 or NGMLR), and pbsv all hit an F-score of ∼ 0.9 or higher, while SVIM after minimap2 and pbsv performed better than the other aligner–SV caller combinations for the detection of translocation breakpoints. Notably, none of the SV callers tested were able to identify high-quality duplications or translocations after lra alignment, especially in the SI00001 dataset where they are known to occur. Manual investigation of the variant files generated by the different SV callers before filtering revealed that most duplications and translocations had few supporting reads ( < 3 in most cases) and were not flagged as “PASS.” As a consequence, being more permissive with the filters used could improve on the detection of these SV categories in datasets aligned with lra.

When further evaluating the influence of genome coverage on the SV caller’s performances, we concluded that adding more than 15X–20X produced only little increment in sensitivity but was associated with a marked decrease in precision for Sniffles and SVIM. On the other hand, a slight increase in precision could be reached by combining multiple callsets and the consensus from cuteSV, Sniffles, and SVIM after NGMLR hit the highest precision values in our tests. Last, we highlighted that having at least 5–10 reads supporting a called SV represents a good trade-off to optimize precision and recall.

Given the results presented in this article, we recommend using cuteSV for the initial assessment of the data as it achieves substantial precision and recall at both SV calling and genotyping, even when analyzing low-coverage data. Because the choice of the pipeline could depend on the need of the users to retrieve SV calls with either high precision or high recall, we conclude that Sniffles should be preferred when looking for high precision, while cuteSV or SVIM should be preferred when high recall is required. However, due to low F-score values at SV genotyping, we do not recommend using Sniffles for an accurate estimation of the zygosity of SV calls, as it often misclassified or missed heterozygous variants in our tests. Combining SV calls from cuteSV, Sniffles, and SVIM can be helpful in further reducing the final false positive rate.