Development and validation of a dynamic nomogram based on conventional ultrasound and contrast-enhanced ultrasound for stratifying the risk of central lymph node metastasis in papillary thyroid carcinoma preoperatively

Purpose The aim of this study was to develop and validate a dynamic nomogram by combining conventional ultrasound (US) and contrast-enhanced US (CEUS) to preoperatively evaluate the probability of central lymph node metastases (CLNMs) for patients with papillary thyroid carcinoma (PTC). Methods A total of 216 patients with PTC confirmed pathologically were included in this retrospective and prospective study, and they were divided into the training and validation cohorts, respectively. Each cohort was divided into the CLNM (+) and CLNM (−) groups. The least absolute shrinkage and selection operator (LASSO) regression method was applied to select the most useful predictive features for CLNM in the training cohort, and these features were incorporated into a multivariate logistic regression analysis to develop the nomogram. The nomogram’s discrimination, calibration, and clinical usefulness were assessed in the training and validation cohorts. Results In the training and validation cohorts, the dynamic nomogram (https://clnmpredictionmodel.shinyapps.io/PTCCLNM/) had an area under the receiver operator characteristic curve (AUC) of 0.844 (95% CI, 0.755–0.905) and 0.827 (95% CI, 0.747–0.906), respectively. The Hosmer–Lemeshow test and calibration curve showed that the nomogram had good calibration (p = 0.385, p = 0.285). Decision curve analysis (DCA) showed that the nomogram has more predictive value of CLNM than US or CEUS features alone in a wide range of high-risk threshold. A Nomo-score of 0.428 as the cutoff value had a good performance to stratify high-risk and low-risk groups. Conclusion A dynamic nomogram combining US and CEUS features can be applied to risk stratification of CLNM in patients with PTC in clinical practice.


Introduction
Papillary thyroid carcinoma (PTC) is the most common type of thyroid cancer, accounting for 80%-90% of all thyroid carcinomas (1,2). PTC is a lymphotropic tumor, and 20%-90% of patients with PTC develop cervical lymph node metastasis (LNM), and approximately 70% of these cases involve central lymph node metastasis (CLNM) (3,4). For patients with PTC, CLNM is an important risk factor for distant metastasis or tumor recurrence and an indicator for surgical strategy (5)(6)(7)(8); in these cases, central compartment lymph node dissection (CLND) is recommended (9). However, preoperative identification of CLNM has been a challenge. Approximately 45% of PTC patients with clinically negative central ventricular LNs (cN0) were reported to have CLNM pathologically confirmed after surgery (10). Whether prophylactic CLND should be performed in cN0 PTC patients is still under debate, possibly raising risks of nerve injury and hypoparathyroidism (11). Therefore, accurate and noninvasive preoperative prediction of CLNM has been of increasing importance in clinical practice to optimize treatment decisions.
Conventional ultrasound (US) is the first-line modality for evaluating thyroid nodules and cervical lymph nodes (9). However, US is limited in detecting CLNM because of interference of the thyroid gland and adjacent organs. As reported, just 30.0%-53.2% of cases with CLNM could be detected by US (2,12,13). In recent years, some US-based imaging modalities were proposed to enhance the capability of identifying CLNM. A radiomics nomogram based on the shear wave elastography (SWE) was established by Jiang et al. (7). However, SWE has not been widely used in clinical practice.
Contrast-enhanced US (CEUS) is an imaging modality that reveals tumor microvascular perfusion through the accumulation of contrast agent microbubbles in blood vessels (14). CEUS has been widely applied to distinguish benign and malignant thyroid nodules (15,16). Several studies reported that CEUS may be a potential tool to predict CLNM in patients with PTC (14,17). However, most studies focused on CEUS features alone and failed to provide a feasible and generalizable prediction model.
In this study, we aimed to develop and validate a nomogram by combining US and CEUS features to facilitate preoperative risk stratification for individualizing treatment decision.

Patients
This two-way cohort study was approved by the Ethics Committee of our hospital. The requirement for individual consent for retrospective data was waived. All patients with prospective data signed informed consent.
Patients who underwent total or partial (lobectomy or near-total thyroidectomy) thyroidectomy for PTC between January 2017 and December 2019 were retrospectively evaluated from the institutional database. From February 2021 to October 2021, we prospectively recruited PTC patients who were diagnosed by ultrasound-guided fine needle aspiration and prepared to receive surgery in our institution. Patients were enrolled according to the following inclusion criteria (1): solitary PTC was determined pathologically; (2) CLND was performed; and (3) US and CEUS were performed preoperatively. The exclusion criteria were as follows: (1) maximum tumor diameter was <5 mm (CEUS is limited by respiratory motion and volume effect); (2) US or CEUS image was incomplete or unclear; (3) skip metastases were found; and (4) history of secondary malignancy.
A total of 216 patients with pathologically confirmed PTC were included. The patients were divided into the training and validation cohorts, with a patient ratio in the training to validation cohorts of 1:1. The training cohort consisted of 108 patients (33 male and 75 female patients; mean age, 43.72 ± 12.44 years), and the validation cohort enrolled 108 patients (20 male and 88 female patients; mean age, 44.4 ± 11.48 years) who were randomly selected from the prospective data. Each cohort was divided into CLNM (+) and CLNM (−) groups according to the pathology results. Figure 1 shows the patient selection.

US and CEUS image acquisition
Training cohort images were acquired with an EPIQ5 (Philips Ultrasound, Inc., Bothell, Washington, USA) system equipped with a 5-to 12-MHz linear probe, and validation cohort images were acquired with a Resona7 (Shenzhen Mindray Bio-Medical Electronics Co., Ltd., Shenzhen, China) system equipped with a 5to 12-MHz linear probe. US and CEUS images for individuals were obtained from the same instruments. SonoVue (Bracco, SpA, Milan, Italy) was used for all patients as a contrast agent with a mechanical index of 0.08 and a volume of 1.2-2.4 ml (2.4 ml in the validation cohort) when CEUS was performed. The US features of the lesion were carefully evaluated, including size, location, margin, echogenicity, aspect ratio, calcification, and contact with the capsule. The longitudinal plane with the clear lesion was selected for CEUS. The contrast agent was injected intravenously as a bolus, followed by a 5-ml saline flush (0.9% sodium chloride). All CEUS images were observed for 3 min and stored on the hard disk for further analysis.
The US and CEUS features were independently analyzed by two experienced radiologists (with more than 10 years of experience in thyroid imaging) blinded to pathological outcomes. If the radiologists disagreed, a consensus was obtained by discussion.

Conventional US image analysis
US features were characterized as follows. Size refers to the maximum tumor diameter. The location was classified into the left lobe, right lobe, and isthmus. The margin was divided into regular or irregular. Echogenicity was classified into hypoechoic, isoechoic, or hyperechoic relative to surrounding thyroid parenchyma. The aspect ratio was classified as ≤1 or >1. Contact with the capsule was described as yes or no according to whether more than 20% of the tumor was touching the thyroid capsule or an absence of echogenicity of the thyroid capsule on US.
The CEUS quantitative parameters were obtained by QLAB or Resona7 system software. The region of interest (ROI) was first outlined along the outer margin of the tumor, defined as ROI1. ROI2 was copied from ROI1 and outlined in the surrounding thyroid parenchyma at the same tumor depth. Two time-intensity curves (TIC) were obtained. The analysis time was the first 60 s of the dynamic CEUS images. TIC parameters included the following: (1) wash in slope (WIS), (2) time to peak (TP), (3) peak intensity (PI), and (4) area under the curve (AUC). These values were measured three times for each tumor and averaged as P ROI1 and P ROI2 . The ratio of P ROI1 to P ROI2 (P ROI1 / ROI2 ) was used for comparison to reduce the potential effect of differences from the imaging system, image analysis software, and contrast agent doses.

Feature selection and model construction
The least absolute shrinkage and selection operator (LASSO) regression with penalty parameter tuning conducted by 10-fold cross-validation was applied in the training cohort to select useful predictive features for CLNM. Univariate and multivariate logistic analyses were performed to identify the risk factors of CLNM. The prediction model was established by combining LASSO and multivariate logistic regression analysis and presented in the form of a nomogram.

Evaluation and validation of the model
The prediction model's performance was evaluated by receiver operating characteristic (ROC) curves in the training and validation cohorts. AUC was calculated to assess the discrimination performance of the prediction model in the training and validation cohorts. Calibration of the nomogram was evaluated using the calibration curve and Hosmer-Lemeshow test.

Clinical utility of the prediction model
To estimate the predictive value of the prediction model, decision curve analysis (DCA) was performed by quantifying the net benefits at different threshold probabilities in the validation cohort.
The Nomo-score, that is, the nomogram-predicted probability, was calculated in each patient. The cutoff value of the Nomo-score was obtained through the maximum Youden index, and patients were classified as high risk and low risk using this value. Flowchart of patient enrollment in the study.

Statistical analysis
Statistical analysis was conducted with SPSS Statistics version 26.0 (IBM Corp.), R software version 4.1.0 (The R Foundation for Statistical Computing), and GraphPad Prism 9.0. Quantitative data were presented as mean ± standard deviation, and Mann-Whitney U tests were used for comparison. Categorical data were presented as numbers and percentages, and the chi-square test was used for comparison. The Delong test was used in ROC. The two-sided p < 0.05 was considered as significant difference.

Patient characteristics
Clinical characteristics, US, and CEUS features of all patients are summarized in Table 1. Except for margin, aspect ratio, and time of wash-in, there were no differences in characteristics between the two cohorts (p > 0.05). The rates of CLNM were 44.4% (48/108) and 49.1% (53/108) in the training and validation cohorts, respectively, with no significant difference found between the cohorts (p = 0.495).

Correlation between clinical characteristics and CLNM
As shown in Table 2, sex, age, tumor size, enhancement intensity, and homogeneity of enhancement were significantly different between CLNM (+) and CLNM (−) subgroups of patients with PTC in both the training and validation cohorts (p < 0.05). The ROC curve showed that the best cutoff values of age and tumor size were 42 years with an AUC of 0.675 and 0.95 cm with an AUC of 0.709, respectively.
Contact with the capsule was associated with CLNM only in the training cohort (p = 0.033). Differences of WIS (p = 0.001) and PI (p = 0.010) between CLNM (+) and CLNM (−) groups were only observed in the validation cohort. There was no significant difference in other clinical characteristics. There were no differences of US and CEUS features between CLNM (+) and CLNM (−) patients with PTC (p > 0.05).

Feature selection and model construction
In the training cohort, LASSO regression analysis was performed to select the useful predictive features for CLNM, including sex, age (≤42 years), size, and enhancement intensity (Figure 2). Multivariate logistic regression analysis revealed that sex, age (≤ 42 years), size, enhancement intensity, and homogeneity of enhancement were independent risk factors for CLNM (Table 3). All these risk factors were incorporated into the prediction model  (Figure 3). This prediction model is displayed as a dynamic nomogram (https://clnmpredictionmodel.shinyapps.io/ PTCCLNM/).

Evaluation and validation of the prediction models
The ROC curves of the prediction model and the single ultrasonic features in the training and validation cohorts are shown in Figure 3. There was good discrimination of the prediction model in the training (AUC: 0.844, 0.773-0.915) or validation (AUC: 0.827, 0.747-0.906) cohorts ( Table 4). The AUC value of the prediction model was higher compared with any single US or CEUS feature (p < 0.05) (Figure 4).
The calibration curve and Hosmer-Lemeshow test showed that the prediction model had good concordance in the training (p = 0.385) and validation (p = 0.285) cohorts ( Figure 5).

Clinical use
The DCA results of the prediction and clinical models are presented in Figure 6. Based on the DCA results, the prediction model has a higher clinical net benefit rate than the US or CEUS features alone in predicting CLNM when the threshold probability is between 7% and 82%.
The cutoff value of the Nomo-score for the diagnosis of CLNM was ≥0.428. We divided patients into low-risk (100 patients) and high-risk groups (116 patients) using this cutoff value. Patients in the high-risk group were more likely to have CLNM (p < 0.001). CLNM (+) and CLNM (−) were discriminated well with a cutoff value of 0.428 in both the training and validation cohorts (Figure 7).

Discussion
In this study, we developed and validated a prediction model by combining US and CEUS features for evaluating the risk of CLNM in PTC patients. The prediction model presented as a dynamic nomogram was more convenient for clinical use, and it showed good diagnostic performance in both training and validation cohorts. A cutoff value derived from Nomo-score can be used for CLNM risk stratification in patients with PTC.
Only half of CLNM cases can be accurately detected by the conventional US (18-20). Some studies reported that the US features of PTC, such as calcification, taller than wide, and contact with the capsule, are related to CLNM (21,22). However, these US features were not correlated with the CLNM in our study, likely owing to different definitions of US features among prior studies. For example, Tian et al. (23) reported that microcalcification within PTC was the strongest predictor for CLNM, and the type of microcalcification depended on its size. In contrast, we classified the microcalcification pattern as present or absent. Previous studies showed that larger tumor size was associated with an increased risk of CLNM (24). In our study, the binary variable of tumor size according to 0.95 cm was not related to CLNM in the LASSO logistic regression and multivariate logistic regression (p = 0.175). Considering that tumor size was associated with CLNM in both the training and validation cohorts, we incorporated tumor size into the prediction model as a continuous variable, and multivariate regression revealed a strong correlation with CLNM (OR: 4.118, p = 0.009).
Previous studies have shown that some US features were valuable for predicting CLNM in PTC patients, but the results were not consistent. In addition, it is difficult to predict CLNM with US alone. Therefore, we explored the value of multimodal US in the diagnosis of CLNM.
In this study, regarding the CEUS features of PTC, heterogeneous enhancement and iso-or hyperenhancement were associated with CLNM, which is consistent with some previous studies. PTC may destroy neovascular tissue when tumorous infiltration and metastasis occur, and it may cause perfusion defects as presented in heterogeneous enhancement (25). Of note, iso-or hyperenhancement is the strongest risk factor in the prediction model. Angiogenesis plays an important role in the process of tumor invasion and underlies the development, growth, and metastasis of tumor. Hyperenhancement indicates an abundant blood supply in the tumor microenvironment, which is associated with an increased risk for CLNM. Similar findings were     ROC curves of the prediction model and the single US and CEUS features for predicting CLNM (A) in the training cohort and (B) in the validation cohort. Nomogram of the prediction model to assess the risk of CLNM in patients with PTC. found in high-grade breast tumors (26). As previously reported, some CEUS quantitative parameters can be applied to predict the risk of CLNM in PTC patients. We also analyzed quantitative parameters such as WIS, TP, PI, and AUC. In the training cohort, the PI and WIS were higher in patients with PTC who had CLNM than in patients without CLNM, but the difference was not statistically significant (p > 0.05). Tao et al. (14) reported that PI was an independent risk factor of CLNM. Considering PI value is likely related to tumor microvessel density, further showing angiogenesis' important role in the development and metastases of tumors (27). However, limited sample in this study leads to the cautious interpretation of the results; thus, further study is required.
In the present study, younger age and male were related to a higher risk of CLNM in PTC patients, consistent with other studies (28,29). Ning et al. (30) suggested that younger age may indicate a higher biological aggressiveness of tumor. Our study concluded that PTC patients who are less than 42 years old are prone to have CLNM, with an age cutoff value close to the value suggested by Tian et al. (≤40 years old). Accumulating evidence has shown an association between being male and a poor prognosis of PTC, but the results have been inconsistent (31)(32)(33).
We used LASSO and multivariate logistic regression in this study to select features. Our prediction model combining US, CEUS features, and clinical factors showed better diagnostic efficiency compared with the single ultrasonic imaging features in both training and validation cohorts. According to the DCA curve, the application of the prediction model could benefit patients more than a treat-none or treat-all strategy when the threshold probability was between 7% and 82%. We also established a risk stratification criterion based on the Nomo-score and showed that patients with a Nomo-score ≥ 0.428 were likely to have a higher incidence of CLNM. Therefore, this prediction model can be used to evaluate individuals preoperatively, and CLND was recommended for patients with a Nomo-score ≥ 0.428.
Our study has several limitations. First, the training cohort in this study was retrospective, and some bias inevitably exists. Second, the data might be affected by the different machines and probes used in the training and validation cohorts, but a ratio was used to reduce FIGURE 6 Decision curves of the prediction model and the single US and CEUS features in predicting CLNM for papillary thyroid carcinoma.

FIGURE 5
Calibration curves of the nomogram in the training (A) and validation (B) cohorts.
this effect. Third, this study needs to be further validated using a study with a larger sample size. Finally, all data were obtained from a single institution; thus, external validation in multicenter clinical trials is warranted.
In conclusion, a web-based dynamic nomogram based on US and CEUS features was constructed and showed a good performance in predicting the CLNM risk in PTC patients. This may be instrumental in refining surgery strategy in PTC patients.

Data availability statement
The original contributions presented in the study are included in the article/supplementary material. Further inquiries can be directed to the corresponding author.