Development and Prospective Validation of an Ultrasound Prediction Model for the Differential Diagnosis of Benign and Malignant Subpleural Pulmonary Lesions: A Large Ambispective Cohort Study

Objective To develop and prospective validate an ultrasound (US) prediction model to differentiate between benign and malignant subpleural pulmonary lesions (SPLs). Methods This study was conducted retrospectively from July 2017 to December 2018 (development cohort [DC], n = 592) and prospectively from January to April 2019 (validation cohort [VC], n = 220). A total of 18 parameters of B-mode US and contrast-enhanced US (CEUS) were acquired. Based on the DC, a model was developed using binary logistic regression. Then its discrimination and calibration were verified internally in the DC and externally in the VC, and its diagnostic performance was compared with those of the existing US diagnostic criteria in the two cohorts. The reference criteria were from the comprehensive diagnosis of clinical-radiological-pathological made by two senior respiratory physicians. Results The model was eventually constructed with 6 parameters: the angle between lesion border and thoracic wall, basic intensity, lung-lesion arrival time difference, ratio of arrival time difference, vascular sign, and non-enhancing region type. In both internal and external validation, the model provided excellent discrimination of benign and malignant SPLs (C-statistic: 0.974 and 0.980 respectively), which is higher than that of “lesion-lung AT difference ≥ 2.5 s” (C-statistic: 0.842 and 0.777 respectively, P <0.001) and “AT ≥ 10 s” (C-statistic: 0.688 and 0.641 respectively, P <0.001) and the calibration curves of the model showed good agreement between actual and predictive malignancy probabilities. As for the diagnosis performance, the sensitivity and specificity of the model [sensitivity: 94.82% (DC) and 92.86% (VC); specificity: 92.42% (DC) and 92.59% (VC)] were higher than those of “lesion-lung AT difference ≥ 2.5 s” [sensitivity: 88.11% (DC) and 80.36% (VC); specificity: 80.30% (DC) and 75.00% (VC)] and “AT ≥ 10 s” [sensitivity: 64.94% (DC) and 61.61% (VC); specificity: 72.73% (DC) and 66.67% (VC)]. Conclusion The prediction model integrating multiple parameters of B-mode US and CEUS can accurately predict the malignancy probability, so as to effectively differentiate between benign and malignant SPLs, and has better diagnostic performance than the existing US diagnostic criteria. Clinical Trial Registration www.chictr.org.cn, identifier ChiCTR1800019828.


INTRODUCTION
It is of great importance to differentiate benign and malignant pulmonary lesions since lung cancer is one of the neoplasms with the highest morbidity and mortality in the world (1,2). Computed tomography (CT), the first-line diagnostic method for lung disease, may fails to diagnose some atypical lesions immediately (3). A definitive diagnosis often requires long-term follow-up or invasive diagnostics, which leads to increased radiation exposure and a high risk of severe complications (4,5). Given these problems, new noninvasive methods are being explored to enable more precise diagnosis.
Compared with CT, ultrasound (US) has the advantage of real time, non-radiation, and bedside availability, and has been used in assessing lung disease since the 19th century (6). To date, both B-mode US (B-US) and contrast-enhanced US (CEUS) have been proven to be valuable in the differential diagnosis of subpleural pulmonary lesions (SPLs) (7)(8)(9). Reportedly, qualitative parameters such as degree and homogeneity of enhancement, perfusion pattern, and vascular sign; and quantitative parameters such as arrival time (AT), lesion-lung AT difference, wash-in rate (WIR) and wash-out rate (WOR) were the potentially useful parameters (8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19). Among them, AT is the most frequently assessed parameter and is recommended by the guidelines proposed by European Federation of Societies for Ultrasound in Medicine and Biology (EFSUMB) (8). It can differentiate benign and malignant SPLs by the difference in the time taken for US contrast agent (UCA) to arrive at the lesion from injection (10)(11)(12)(13)(14). However, this parameter is affected by multiple physiological and external factors, such as the length of vessels, cardiac function, and the injection velocity of UCA (20). Therefore, lesion-lung AT difference is proposed, which can exclude the influence of individual factors to a certain extent, and provides better diagnostic performance (13).
Multiple studies have confirmed the effectiveness of US in the diagnosis of SPLs. In B-US images, wedge-shaped, ill-defined margins and aerated bronchus signs are considered as the characteristics of benign lesions, while malignant lesions tended to present spherical and well-defined margins. As for CEUS, AT >7.5s and lesion-lung AT difference >2.5s are recommended as the criteria for predicting malignancy. Nevertheless, they are still limited to the exploration of small samples or single indicators, and the level of evidence is low (8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19). Therefore, the value of lung US requires a larger and more robust study design (9).
We aimed to conduct a large cohort study to explore more valuable parameters of B-US and CEUS and to construct a model for differentiating benign and malignant SPLs with improved accuracy.

MATERIALS AND METHODS
This retrospective and prospective cohort study was in accordance with the ethical standards formulated in the Helsinki Declaration and approved by the Institutional Review Board (No. K18-197Y). Moreover, our study protocol followed the statement of Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) and registered in the Chinese Clinical Trial Registry (No. ChiCTR1800019828). All patients signed written informed consent for CEUS, and the patients recruited prospectively also signed separate written informed consent to participate in this clinical trial.

Patients
We retrospectively collected the medical data of consecutive patients who underwent both B-US and CEUS of lung between July 2017 and December 2018 in a leading pulmonary hospital, and used it as a development cohort (DC) for model establishment and internal validation. Then, we prospectively collected same patient data from January to April 2019 as an external validation cohort (VC) for the model.
The inclusion criteria were (a) the lesion was found for the first time and localized beneath the pleura, using X-ray, CT or US; (b) the lesion was not blocked by artifacts of gas or bones and can be clearly imaged by US. The exclusion criteria were (a) hypersensitivity reaction to UCA (SonoVue; Bracco SpA, Milan, Italy); (b) severe cardiovascular disease; (c) a definite diagnosis had not been successful.

Image Acquisition and Analysis
The process of image acquisition and analysis was unified in the DC and VC. Two radiologists performed this process together (K.B. and Y.Z., with 4 and 5 years of experience in lung US, respectively) using the LOGIQ E9 US System (General Electric Healthcare, Chicago, IL, USA) configured with a 1-6 MHz convex probe. For discordant assessments, a third senior radiologist (Y.W., with 18 years of experience of lung US) was consulted on the cases and would make the final decision. All radiologists had received standardized training before participating in this study. Radiologists were required to locate the probe parallel to the intercostal space and on the largest section of the lesion to obtain B-US and CUES images, and once the probe is positioned, it cannot be moved. The specific method of image acquisition and analysis and the definition of US parameters are given in Figure 1 and Table 1.
B-mode US (B-US) parameters were recorded in the largest section of the lesion with general instrument settings: (1) transverse diameter, (2) longitudinal diameter, and (3) the angle between lesion border and thoracic wall.
In contrast-enhanced mode, the mechanical index was set at 0.1, and the gain was adjusted to show the surface of air-filled lungs only (20 dB). Then 1.5 ml of UCA was injected into the median cubital vein within 2s via a 20-gauge needle, followed by an immediate flush with 5 mL of normal saline and the dynamic clip was recorded for 3 minutes (8,20).
Qualitative CEUS parameters were obtained by observing dynamic clips frame by frame: (a) perfusion pattern, (b) degree of enhancement, (c) homogeneity, (d) vascular sign, and (e) nonenhancing region type.
Quantitative CEUS parameters were measured and calculated according to the time-intensity curve (TIC). We used the TIC images and parameters of subpleural pulmonary lesions. In this series of figures, the diagrams of the patterns and corresponding US images are displayed in pairs (except for A-C). In the diagrams of the patterns, pink represents the thoracic wall, green represents the lesion, yellow represents the enhanced area of air-filled lung tissues in contrast-enhanced US (CEUS) mode, light blue represents the non-enhancing air-filled lung tissues, black represents the non-enhancing region of a lesion, white arrow (in C) represents the emission and reception of the US signal, and red line represents the vascular pattern.

Reference Standards
The diagnosis of all lesions was made by two senior respiratory physicians. Histopathology is in priority, when histopathology could not make a definite diagnosis, microbial evidence, imaging findings, clinical symptoms, and treatment effects were required to make a comprehensive analysis. In addition, all cases were followed up for at least 12 months. Respiratory physicians, Pathologists, laboratorian and patients were blinded to our image analysis results.

Statistical Analyses
SPSS V.20.0 (SPSS Inc., Chicago, IL, USA) and R software V.3.6.0 (Institute for Statistics and Mathematics, Vienna, VIC, Austria) were used for statistical analysis. Categorical data were expressed as absolute numbers and percentages, while continuous data were shown as medians and interquartile range (IQR) or mean ± standard deviation. P values of <0.05 were considered to reflect statistical significance. The prediction model was developed based on the data of the DC. A univariate analysis was firstly performed to select US parameters showing significant differences between benign and malignant lesions as candidate variables. Then, a multivariate analysis using binary logistic regression (forward stepwise method) was performed to screen the candidate variables and establish a prediction model. Finally, the model was used to calculate the malignancy probability of each case to plot the ROC curve and determine the cutoff value when Youden's index reached its maximum.
The data of the DC and VC were used for internal and external validation of the model respectively. The C-statistic was used to assess the discrimination and the calibration curve based on 1000 bootstrap re-samples was used to evaluate the agreement between actual and predictive malignancy probabilities. Finally, the diagnostic ability of the model and the existing US diagnostic criteria for malignant lesions: "lesion-lung AT difference ≥ 2.5 s" (13) and "AT ≥ 10 s" (14) was compared.

Patient Demographic and Clinical Characteristics
A total of 837 consecutive patients were enrolled in our study, including 592 in the DC and 220 in the VC; while 19 and 6 patients were excluded from each respective cohort ( Figure 2). The demographic and clinical characteristics of both cohorts were comparable ( Table 2).

Univariate Analysis of US Parameters
Using data from the DC, 14 out of 18 US parameters showed statistical differences between benign and malignant lesions and were selected as candidate variables. The details are as follows ( Table 3).
All of 3 B-US parameters showed significant statistical differences between benign and malignant lesions. The

US Parameters
Definition Recorded content

Transverse diameter
The maximum diameter of the lesion parallel to the thoracic wall. value, cm Longitudinal diameter The maximum diameter of the lesion perpendicular to the thoracic wall. value, cm Angle between lesion border and thoracic wall As long as one angle is obtuse, the parameter was classified as obtuse.
obtuse or acute

Perfusion pattern
The way UCA enters the lesion. For a lesion manifesting overlapping patterns, the one with the widest involvement is recorded.
hilum-to-pleura, periphery-to-center or part -to-whole Degree of enhancement The enhanced degree of air-filled lung tissues is defined as hyper-enhancement and the enhanced degree of thoracic wall muscle is defined as hypo-enhancement.
hyper-, iso-or hypo-enhancement Homogeneity Distribution uniformity of UCA in the lesion. homogeneous or heterogeneous Vascular sign The morphological character of the earliest enhanced blood vessels in the lesion. negative, tree-like or curly hair-like Non-enhancing region type The morphological characteristic of non-enhancing region.

Multivariate Analysis and Model Establishment
Among the 14 candidate variables, the following 6 were selected for modeling (Table 4) The US prediction model is shown in Figures 3A, B in the form of nomogram and formula, which can be used to predict the malignancy probability of a lesion. The detailed method of application was explained in the figure legend.
The cutoff value obtained from the ROC curve was 44.59%, that was, when the malignancy probability was ≥ 45.59%, the lesion was judged as malignant ( Figure 3C). The applications of the model in benign and malignant SPLs are shown in Figure 4 and Figure 5 (the corresponding dynamic clips are Supplementary Videos 1, 2).

Model Validation
In terms of discrimination, the C-statistics of the model were 0.974 (95% confidence interval [CI]: 0.957-0.985) and 0.980 (95% CI 0.951-0.994) in the internal (DC) and external (VC) verification cohorts, respectively, which is significantly higher than those of "lesion-lung AT difference ≥ 2. As for calibration, the calibration curves showed good agreement between actual and predictive malignancy probabilities. Although the apparent probability of the external validation cohort showed a slight deviation, the bias-corrected probability improved the final result ( Figures 6A, B).

DISCUSSION
US has been used in the differential diagnosis of benign and malignant SPLs for decades. In terms of B-US, the lesion shape, internal echo, boundary definition, bronchial inflation sign, relationship with pleura and adjacent organs are valuable (6)(7)(8). As for CEUS, the perfusion mode, degree of enhancement, homogeneity, microvascular characteristics, and especially AT are recommended indicators (8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19). However, studies that proposed the above indicators also has limitations, including small sample size, inconsistent conclusions from different researchers, strong subjectivity, and no multi factor analysis (9). In this large-scale and multiparameter study, we constructed a new US prediction model for the differential S.E., standard error; S.b, standardized regression coefficient; df, degree of freedom; OR, odds ratio; CI, confidence interval; BI, basic intensity; Lesion-lung AT difference, the AT difference between lesion and air-filled lung tissues; Ratio of AT difference, the ratio of "AT difference between lesion and air-filled lung tissues" to "AT difference between thoracic wall and air-filled lung tissues"; Neg, negative.
diagnosis of benign and malignant SPLs and obtained good discrimination and calibration. To our best knowledge, this is the first model based on both B-US and CEUS parameters for diagnosis of SPLs, which can provide radiologists with more accurate diagnostic information than B-US and the existing CEUS diagnostic criteria. In our model, the time-related quantitative CEUS parameters played a dominant role on the prediction of malignancy probability. Similar to liver, lung also has a dual blood supply comprised of the pulmonary arteries and the bronchial arteries (19,21,22). When a malignant tumor develops, the blood supply from the bronchial arteries will markedly increase and gradually replaces the supply from the pulmonary artery, becoming the main source of blood for the tumor (19,21,22). And this characteristic can be identified by AT, because the arterial phases of these two types of blood vessels are different (19). It is controversial that different investigators have proposed different cutoff values for AT. Sartori et al. (17) suggested AT > 7.5 s, the arterial phase of a lesion was delayed, indicating a great possibility of malignancy, while Caremani et al. (14) recommended AT > 10 s as the cutoff value. Furthermore, multiple internal and external factors may influence the value of AT (20). Thus, quantitative observations of AT alone as a diagnostic criterion to determine etiology of SPLs is at high risk of misjudgment. To deal with these issues, Bai et al. (13) took the AT of lung tissue as the baseline, calculated the AT difference between of lesion and the baseline, and finally obtained lesion-lung AT difference, which was proven to provide better diagnostic performance.
Our study evaluated both AT and lesion-lung AT difference, and the results showed that their discriminations were inferior to that of our model. We attribute the excellent performance of the model to our newly created parameter, ratio of AT difference, which uses air-filled lung tissues and the thoracic wall as reference, and judges the blood supply source of the lesion by comparing the AT differences of air-filled lung tissues, lesion,  The ROC curve of the model. The larger the area under the curve, the stronger the ability to differentiate between benign and malignant lesions. The best cutoff value and the diagnostic performance are shown in detail. BI, basic intensity; AT, arrival time; Lesion-lung AT difference, the AT difference between lesion and air-filled lung tissues; Ratio of AT difference; the ratio of "AT difference between lesion and air-filled lung tissues" to "AT difference between thoracic wall and air-filled lung tissues". and thoracic wall. This parameter represents the ratio of "the time taken by the blood from the pulmonary circulation to the lesion" to "the time taken by the blood from the pulmonary circulation to the systemic circulation". The smaller the ratio, the more likely the blood supply of the lesion is from the pulmonary circulation. The larger the ratio, the more likely the blood supply of the lesion is from the systemic circulation. This parameter has greater clinical utility because individual differences are excluded to the extent possible. For example, in some patients, ATs of lung, lesion, and thoracic wall are very close, and the lesion-lung AT difference may be far shorter than the cutoff value, so all of these lesions will be classified as benign. But when ratio of AT difference is applied, they can be accurately differentiated because of the invariant proportional relationship. Different from the above point of view, several studies have shown varying types of arterial supply for some special types of lesions, including mixed arterial supply and even dominant pulmonary arterial supply (22,23). It seems that it is not rigorous to diagnose only by the source of blood supply. Fittingly, our model also included qualitative CEUS parameters, including vascular sign and non-enhancing region type. For vascular sign, Wang et al. (15) believed that the main features of benign lesions were branching, pointed, patchy and rim-like, while those of malignant lesions were vascular, cottonlike and dead-wood like. However, Caremani et al. (14) described spots, points, and ring-enhancement as the features of malignant lesions, and linear hyperechoic images as the features of benign lesions. These examples clearly demonstrate that complicated subjective classification systems will lead to inconsistent conclusions among researchers. In our study, only distinct tree-like and curly hair-like vascular signs were recorded. The tree-like sign is a feature of the benign lesions, which corresponds to a pulmonary artery with a normal structure, and the curly-hair-like sign is a feature of the malignant lesions, which corresponds to tumor neovascularization with a disordered structure and tortuous form. Compared with the existing studies (14,15), our classification method is simpler, more specific, and more accurate.
In term of non-enhancing regions, the main feature of malignant lesions is irregular with ragged edges while those of benign lesions are often regular with smooth edges. Tuberculosis, in particular, often presents as a single affected area or as multiple small patches of non-enhancing region early in the course of the disease, which gradually expands and merges into large patches of non-enhancing region as the disease progresses. The nonenhancing region features that we have described are consistent with those of other imaging studies (24)(25)(26).
The qualitative parameters in our prediction model are easy to implement with excellent reproducibility, and played an important auxiliary role in the prediction of malignancy probability.
In addition to CEUS parameters, only one B-US parameter, the angle between lesion border and thoracic wall, was included in the final model, which reflects the morphological characteristics of the lesions. In contrast to CT, US cannot easily display the lobes, spinous processes, or burrs on the edge of a lesion because the boundary between the lesion and air-filled lung tissues is not clear in US images (27,28). However, the boundary between the lesion and thoracic wall is clear and the angle can be easily measured. In particular, pulmonary tuberculosis lesions with hyperplasia as the main feature often showed obtuse angles, which resulted in a large proportion of obtuse angle lesions in the benign lesion group in this study (25,29). But in general hospitals that are not responsible for tuberculosis diagnosis and treatment, the resulting bias will be minimal.
In addition to the above variables included in the model, some CEUS parameters are also different between benign and malignant lesions, such as perfusion pattern, PI and WOR, which are consistent with previous literatures (8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19). Based on the principle of simplifying the model as much as possible to avoid over fitting, our model only included 6 indicators to synthesize the information of B-US and CEUS. In summary, the method is simple and has strong clinical feasibility.

LIMITATIONS
The primary limitation of this study is that it was conducted at a single center, so there is a possible bias caused by a  nonrepresentative distribution of disease types. However, our center is a leading pulmonary hospital. Patients came from all over the country for our study may effectively reduce the bias.
The second limitation is that more features of B-US and color doppler flow imaging were not included in this study, because the gas artifacts and respiratory movement might affect their stability and accuracy. However, these indicators, especially B-US parameters, may still be of certain value in the diagnosis.
Considering that these limitations, we plan to conduct a prospective multicenter study with an increased number of US parameters in order to improve our model further.

CONCLUSION
Our model, synthesizing multiple parameters of B-US and CEUS, could contribute to improved performance in the differential diagnosis of benign and malignant SPLs, compared with the existing CEUS diagnostic criteria. It is simple, nonradioactive and has great potential for clinical use.

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

ETHICS STATEMENT
The studies involving human participants were reviewed and approved by the Institutional Review Board of Shanghai pulmonary hospital (No. K18-197Y). The patients/participants provided their written informed consent to participate in this study.

AUTHOR CONTRIBUTIONS
YW, LF and KB had the idea for and designed the study. They had full access to all data in the study and take responsibility for the integrity of the data analysis. KB and YW wrote the first full draft of the report. KB, YZ and YW contributed to image analysis. YZ, M-jS, H-wC, YC, H-mZ, C-hT and JY contributed to critical revision of the report. D-mX and X-fY contributed to the statistical analysis. All authors contributed to the article and approved the submitted version.