Histopathology-Based Deep-Learning Predicts Atherosclerotic Lesions in Intravascular Imaging

Background: Optical coherence tomography is a powerful modality to assess atherosclerotic lesions, but detecting lesions in high-resolution OCT is challenging and requires expert knowledge. Deep-learning algorithms can be used to automatically identify atherosclerotic lesions, facilitating identification of patients at risk. We trained a deep-learning algorithm (DeepAD) with co-registered, annotated histopathology to predict atherosclerotic lesions in optical coherence tomography (OCT). Methods: Two datasets were used for training DeepAD: (i) a histopathology data set from 7 autopsy cases with 62 OCT frames and co-registered histopathology for high quality manual annotation and (ii) a clinical data set from 51 patients with 222 OCT frames in which manual annotations were based on clinical expertise only. A U-net based deep convolutional neural network (CNN) ensemble was employed as an atherosclerotic lesion prediction algorithm. Results were analyzed using intersection over union (IOU) for segmentation. Results: DeepAD showed good performance regarding the prediction of atherosclerotic lesions, with a median IOU of 0.68 ± 0.18 for segmentation of atherosclerotic lesions. Detection of calcified lesions yielded an IOU = 0.34. When training the algorithm without histopathology-based annotations, a performance drop of >0.25 IOU was observed. The practical application of DeepAD was evaluated retrospectively in a clinical cohort (n = 11 cases), showing high sensitivity as well as specificity and similar performance when compared to manual expert analysis. Conclusion: Automated detection of atherosclerotic lesions in OCT is improved using a histopathology-based deep-learning algorithm, allowing accurate detection in the clinical setting. An automated decision-support tool based on DeepAD could help in risk prediction and guide interventional treatment decisions.


INTRODUCTION
Coronary artery disease is the leading cause of death worldwide, accounting for the majority of acute coronary syndromes and sudden cardiac deaths. Identification of atherosclerotic tissue might allow stratification of patients at risk for future coronary events (1)(2)(3). Invasive assessment of the coronary arteries using high-resolution intravascular imaging has emerged as an important tool for identifying atherosclerotic lesions (4), as the close proximity of the imaging catheter allows a more precise and high-resolution visualization of the vascular tissue compared to non-invasive modalities (5). The main advantage of optical coherence tomography (OCT) in comparison to other imaging modalities such as intravascular ultrasound (IVUS) is its superb resolution of 10-20 µm with a tissue penetration of approximately 2-3 mm, which enables clear visualization of atherosclerotic components such as lipid tissue with foam cell infiltration or calcifications (5,6). However, assessment and evaluation of OCT is currently based on clinical expertise from skilled interventionalists and pitfalls in plaque characterization can lead to misclassification (7,8). Additionally, each volume of OCT images obtained in one measurement, also called a pullback, generates a vast amount of data, precluding from analog assessment of single frames. Due to the high clinical relevance and the cumbersome manual review, automated or semi-automated approaches for preliminary evaluation of OCT images is of great interest for the interventional community (9)(10)(11). For intravascular imaging, several groups demonstrated the possibility of automated plaque characterization using semiautomated approaches such as quantification of various optical signal properties (e.g. light attenuation) (12)(13)(14)(15). The feasibility of using deep-learning algorithms for detection and visualization of atherosclerotic plaque components was also demonstrated recently (9,11,(16)(17)(18)(19). In this work, we developed DeepAD, a deep-learning algorithm trained on data with histopathologybased annotations from autopsy specimens as well as clinical OCTs for prediction of atherosclerotic lesions, and evaluated it in dedicated real-world cases (Figure 1).

Histopathology and ex vivo OCT Imaging
Autopsy samples of coronary arteries were acquired from the department of pathology at Klinikum Rechts der Isar in accordance with federal state law and after approval from relatives (ethical approval number: 291/18 S). Coronary arteries were then cut by an experienced pathologist according to the standard AHA classification. OCT imaging of the single segments was then performed as previously described (15) guidewire over which the OCT catheter (2.7-F, St. Jude Medical, St. Paul, Minnesota) was advanced. An imaging pullback was performed from the distal to the proximal arterial segments while simultaneously flushing the vessel with contrast to improve imaging quality (pullback speed 5 mm/s = 120 frames/s). Afterwards, arteries were cut at 3 mm intervals and stained with haematoxylin-eosin (H.E.) and Movat-Pentachrome. The presence of different atherosclerotic plaque types and plaque components was evaluated by an expert in cardiovascular pathology (MJ) according to the classification by Virmani et al. (20), (see Table 1). Co-registration of OCT frames and histological sections of human autopsy samples was achieved as previously described (21) by first, considering only the proximal, middle or distal part of the OCT pullback for the respective slide and second, visual alignment of anatomical landmarks (calcifications, side branches, lumen contour) within each part of the segment.

Clinical Data Set
OCT pullbacks from the OCT database at the German Heart Center Munich (including all patients undergoing intravascular imaging with clinical indication for OCT during coronary angiography) were screened. OCT imaging was performed according to current recommendations (22) with commercially available OCT systems (Dragonfly DF-OCT-catheter combined with the C7-XRTM imaging system; LightLab Imaging Inc., Westford MA, USA). Pullbacks were assessed for the presence of atherosclerotic plaque features as previously described (23) and included fibrous, lipid and calcified plaques (see Table 2). A total of 222 frames from 51 patients were used for analysis. In case of stented vessels, proximal and distal regions outside the stent were used. For baseline characteristics, (see Table 3).

Manual Annotation of OCT Frames
OCT images were transferred to an offline working station and transformed into 8bit RGB images. Labeling of OCT frames was achieved using the freeware tool LabelMe (24). In each frame, the suspected plaque area was identified. Plaque area was measured based on histopathology and by tracing the leading edge in OCT images using a polygonal line tool, respectively. Plaque area was defined as the presence of lipid accumulation, necrotic core with or without foam cell infiltration and/or calcification (Figure 2). To avoid pitfalls as reported in (7), we included the guidewire artifact into the label "background".

DeepAD
DeepAD is an ensemble of semantic segmentation networks performing pixelwise classification with respect to atherosclerotic lesions. In addition to histopathology-based OCT annotations, we employed so-called weakly supervised training by using OCT images annotated without histopathology. While histopathologybased annotations represent the gold standard, annotations based on OCT images contain valuable information for learning to segment atherosclerotic lesions, however, remain limited in resolution and diagnosis, hence providing a weak supervisory signal to the training (25). We trained and selected several models in so-called ensembles, an effective method to avoid overfitting  on small data sets (26). All the models in the ensemble were created with the same network architecture. The differences in the models were the weight initializations of the networks, training batch sampling as well as optimization parameters. As no random seeds were set when training the models, these parameters were assigned different values at the start of each  Values are mean ± SD or n/N (%).
training round. This is a commonly used strategy to achieve different models as the network training of Deep Learning algorithms is stochastic and typically converges to different local minima (27). At inference time, all model predictions were averaged to obtain the final prediction for a given pixel. For further details on network architecture, hyperparameters, exact ensembling, (see Supplementary Material).

Evaluation on Unseen Patients
To leverage all the available data, DeepAD was trained several times using "leave one patient out" cross-validation. Specifically, for each training round, all histopathology-annotated OCT images from one patient were held out for testing. The remaining data was divided into a 80 % training and 20 % validation split, with no patient overlap between groups. The model was then trained with the training set and tested on the validation set. For each model in the ensemble, the model weights that performed best on the validation set were then tested on the held-out patient. This process was then repeated until each histopathology-annotated patient had been held out for testing once. The OCT images annotated without histopathology were never included in the test set.

Segmentation
Segmentation describes the task of linking certain regions or pixels within an image to a specific class label (here: lumen, lesion, other). We evaluated the segmentation performance using the intersection over union (IOU, also called Jaccard index), a common metric for evaluation of the quality of segmentation algorithms (28). The IOU is calculated as the overlap between the manual annotation (i.e. the "ground truth") with the prediction from the algorithm divided by the union (see Figure 2A): IOU can range from 0 (no overlap between ground truth and prediction) to 1 (complete overlap between ground truth and prediction; Figure 2A).

a-Line Classification
In contrast to segmentation (i.e. the pixel-wise classification), a-line classification computes a binary output regarding the presence of a certain manually annotated class or label in an aline (e.g. a lesion). In our work, one a-line was defined as one angular direction originating from the center of the lumen. For each OCT frame, 360 a-lines represented the whole 360 • vessel circumference. Classification was evaluated on a binary level, with a-line being positive if >3 pixels along the a-line contained atherosclerotic lesion and negative if no pixels along the a-line contained atherosclerotic tissue, as defined in previous works (11).

Histopathology and Clinical Data set Characteristics
In the histopathology data set, a total of 62 lesions from 7 patients were identified according to the classification by Virmani et al. (20) and manually co-registered with OCT (see Table 1 for details regarding the different plaque types) as previously described (21). For the clinical data set, a total of 222 OCT frames were identified as suitable for analysis (see Table 2 for details on the different plaque types). For examples from both datasets (see Figure 3). The histopathology-based annotations as well as the clinical annotations were annotated independently by two clinical and histopathology experts. The estimated interobserver variability for manual annotation of OCT frames with and without histopathology-based annotations was measured as the IOU between two independent expert observers. The median interobserver variability was 0.58 without and 0.71 with histopathology. Thus, the agreement was considerably higher when histopathology-based annotations were available. We also measured the alignment between annotations on the same scans with and without histopathology, being 0.61 across the two annotators. This showed that a difference in results was observed when annotating with or without histopathology.

DeepAD Accurately Predicts Atherosclerotic Tissue in Unseen Patients
DeepAD predicted atherosclerotic lesions with a median IOU of 0.68 ± 0.18 across all test patients (Figure 4A). Good and moderate performing examples ( Figure 4B) illustrate that the DeepADs is highly accurate in the localization of atherosclerotic lesions across the lumen circumference as well as the difficulty of correctly estimating the extent of this lesion beyond the lumen border (see e.g. Figure 4B, moderate example). Figure 4B (lower row) illustrates a low performing prediction from DeepAD. Here DeepAD falsely predicts a lesion below the lumen which is not supported by the manual histopathologybased annotation. Overall, DeepAD predicts lesions with an IOU of 0.03 lower than the median IOU agreement between two observers, showing close to on par segmentations with expert annotators. To evaluate the importance of using histopathology-based OCT annotations, all OCT images in the histopathology data set were additionally annotated without histopathology by two independent observers. The algorithm was then trained with a dataset of the exact same size, but without histopathology-based annotations. The influence and importance of using histopathology-based annotation in the creation of the DeepAD algorithm was measured as the discrepancy of these models' performance with respect to the histopathology-based annotations. When the algorithm was trained using only annotations without histopathology, the median performance on the test set dropped by more than 0.25 IOU from 0.68 ± 0.18 to 0.42 ± 0.18 when predicting atherosclerotic lesion tissue ( Figure 4A). This corresponds to a performance drop of around 33 %. In Figure 4B the predictions from DeepADs good, moderate and low performing cases are also shown for the algorithm trained without histopathology. In this case the segmentation algorithm yields false positive predictions (Figure 4B good performing example) and less extensive segmentation beyond the lumen border ( Figure 4B median example). In the low performing example in Figure 4B both algorithms output similar false positive tissue prediction.

Prediction of Calcification by DeepAD
Prediction of calcifications in our data set yielded an average IOU = 0.34. Figure 5 shows representative examples of good, intermediate and low performance. In most cases calcification was correctly localized by DeepAD but incompletely predicted (i.e. not all of the calcification was predicted by DeepAD).

Application of DeepAD in Clinical Cohort
To illustrate the use of the DeepAD in real-world clinical practice, atherosclerotic lesion prediction was demonstrated in 11 cases of patients who presented to the German Heart Center and underwent coronary angiography and intravascular imaging with OCT. Performance of the algorithm was retrospectively compared against manual analysis of each pullback by an expert regarding the presence or absence of atherosclerotic lesions as well as correct spatial prediction of the DeepAD on a framelevel. 3D-rendering of the complete pullback allowed quick and intuitive visualization of regions with atherosclerotic lesions in red, (see Figure 6A). A total of 3284 frames were analyzed and high agreement was seen between automated prediction by DeepAD and manual analysis (mean agreement 88%, 2898 of 3284 frames), with good sensitivity, specificity and accuracy (86.8, 82.9 and 85.8%, see Table 4). DeepAD tended to slightly underestimate the presence of atherosclerotic lesions (66 vs. 71% by manual analysis). Representative frames of false-positive predictions are shown in Supplementary Figure 2. For examples on detection of calcification, (see Figure 6B).

DISCUSSION
The goal of our study was the development of a fully-automated, deep-learning algorithm trained on histopathology and clinical data for the prediction of atherosclerotic lesions in intravascular OCT. The main findings of our work are: I DeepAD is able to predict atherosclerotic lesions in intravascular OCT images when trained on data from histopathology and clinical imaging, with an IOU = 0.68 (±0.18). II Although prediction of calcification was lower (IOU = 0.34), calcified lesions were localized correctly in most cases (circumferential and spotty). III Using histopathology-based annotations helped in training the algorithm over using only clinical annotations. IV Clinical applicability of DeepAD was demonstrated in a small clinical cohort of 11 cases.  Table 4). Case 9 mostly has fibrolipidic lesions without any calcifications while case 10 shows heavily calcified lesions (note that DeepAD is able to detect and differentiate circumferential and spotty calcifications). Case 11 shows almost no atherosclerotic lesions. Application of deep-learning technology has the ability to transform the biomedical field in the 21th century. Due to significant advancements in access and storage of big data, and the availability of adequate processing power, use of algorithms embedded in artificial intelligence is likely to have a considerable impact on clinical practice toward more standardized and personalized patient care (29). The need for promoting artificial intelligence in the field of intravascular imaging is supported by the fact that the consequenes of advanced and undetected coronary artery disease are still detrimental, with cardiovascular disease being the leading cause of death worldwide and acute coronary syndromes or stroke accounting for 85% of these deaths (30). This hypothesis is supported by two largescale studies: using virtual-histopathology IVUS (VH-IVUS) in 697 patients presenting with ACS, Stone et al. showed that lesions characterized as thin-cap fibroatheroma, although being angiographically mild, were responsible for the majority of nonculprit coronary events after 3 years (4). Similar, Prati et al. found that the presence of four high-risk plaque features identified by OCT (minimal lumen area <3.5 mm 2 fibrous cap thickness <75 mm, lipid arc circumferential ex-tension >180 • and infiltration with macrophages) was associated with a higher risk of coronary events in 1,003 patients (31). Although all clinically used intracoronary imaging modalities (IVUS, OCT and NIRS) possess inherent limitations, the availability and application of high-resolution intracoronary imaging will likely increase with time (8). Most recently, NIRS (using near-infrared spectroscopy to visualize lipid-rich plaques) has been shown to detect patients at a higher risk for subsequent coronary events (32), being the first intracoronary imaging modality with the ability to provide predictive assessment of atherosclerotic coronary lesions. In our work, DeepAD predicted atherosclerotic lesions in general with high accuracy. While the identification of vulnerable plaques might be desirable and could be regarded as the "holy grail" in cardiovascular imaging, it may be argued that the concept of a single "vulnerable" plaque leading to coronary events might be overly simplistic and that rather the patient himself should be considered vulnerable. In this regard, detection of atherosclerosis burden might help in stratification of patients as the presence of atherosclerosis correlates with cardiovascular events (33,34). For example, prospective studies have shown that identification of coronary artery calcification can be used for risk prediction in patients with suspected coronary artery disease and is therefore utilized in clinical risk scores (35). This seems contradictory at first as it is known from studies on vascular biology that calcification per se is an important mechanism of plaque stabilization: while macrophages are undergoing apoptosis, microcalcifications are occurring as a consequence of cell death (36). Eventually, the whole necrotic core might calcify over time, leading to stabilization and passivation of high-risk plaque regions (36). Nevertheless, calcification is closely correlating with overall atherosclerotic burden and therefore predicts mortality in patients with suspected coronary artery disease (37). Hence, similarities to our approach of using deep learning for detection of atherosclerotic burden with OCT are obvious: DeepAD enables identification of coronary atherosclerosis, which can be visualized using OCT but often pose a challenge in daily clinical practice for inexperienced clinician.
When detecting calcification, performance of DeepAD yielded an IOU = 0.34 (±0.07). While this is definitely inferior to the detection of atherosclerotic lesions in general (IOU = 0.68), we demonstrate that in most cases, calcification is correctly localized by DeepAD but incompletely predicted (i.e. not the complete extent of the calcification is predicted by DeepAD). For the interventionalist, this limitation might be neglectable as the sheer presence of calcification (even though not completely detected) is highly informative. This in turn might influence interventional treatment strategy (use of highpressure or cutting-balloon, pre-or post-dilatation etc.) to deliver an optimal procedural result and avoid e.g. incomplete stent expansion.
With respect to previous work, only one study used coregistered histopathology images with OCT images for plaque segmentation (16). While our work is relying on a deep-learning approach to segment atherosclerotic lesions, the work by He at al. used ex vivo carotid plaque tissue samples and different, carefully handcrafted features with a decision tree. In our approach, CNN learns all filters automatically without any human supervision, hence a decision tree is not needed this way. Besides, deeplearning methods that do not use histopathology images have a limitation on the annotation quality: When segmenting without the help of histology images, only features which are clearly visible in the OCT image will be annotated and tissue that reaches further out will probably be missed. With the help of histopathology, these regions can be annotated more accurately. This ideally enables DeepAD to predict even hard-to-detect features that likely would have been missed if trained on clinical OCTs only.
Ultimately, DeepAD should be regarded as a decisionsupporting tool which can aid in the clinical setting by providing a more standardized characterization of OCT with the potential to quickly visualize atherosclerotic burden in the cath lab. Whether this may be used for risk prediction has to be validated in dedicated future studies. In conclusion, our work highlights the value of using artificial intelligence in the field of intracoronary imaging. By providing an easily applicable tool based on a deep-learning algorithm trained with histopathology-annotated data as well as with clinical expertise, DeepAD can ultimately improve the diagnostic performance of intracoronary optical coherence tomography in detecting atherosclerotic lesions.

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be made available by the authors, upon reasonable request.

ETHICS STATEMENT
The studies involving human participants were reviewed and approved by Ethical Committee of TU München, Ethical Approval Number: 291/18 S. The patients/participants provided their written informed consent to participate in this study.

AUTHOR CONTRIBUTIONS
All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.