Machine learning and Shapley Additive exPlanations to predict metastasis of lymph nodes posterior to the recurrent laryngeal nerve in cN0 papillary thyroid carcinoma

Abstract

Objective:

Prophylactic dissection of lymph nodes posterior to the recurrent laryngeal nerve (LN-prRLN) in clinically node-negative (cN0) papillary thyroid carcinoma (PTC) remains controversial due to the inability to preoperatively assess LN-prRLN metastasis.

Materials and methods:

This study aims to construct and validate an interpretable predictive model for LN-prRLN metastasis in cN0 PTC using machine learning (ML) method. Data were collected from hospital A and divided into training and testing sets (7:3). Additional data from the hospital B were used as validation set. Nine ML models, including XGBoost, were developed. Predictive performance was evaluated using ROC curves, decision curve analysis (DCA), calibration curves, and precision-recall curves. The best model was compared to a traditional logistic regression-based nomogram using learning curves and the method of Probability-based Ranking Model Approach (PMRA). SHapley Additive exPlanations (SHAP) were used to interpret the top ten predictive features and create a web-based calculator.

Results:

A total of 2033 patients were included. XGBoost outperformed other models with AUCs of 0.859, and 0.885 for the testing, and validation sets, respectively, compared to the nomogram (0.814, 0.836). SHAP-based visualizations identified the top ten predictive features: ipsilateral paratracheal lymph node metastasis rate, number of total central lymph node metastases, total central lymph node metastasis rate, number of ipsilateral paratracheal lymph node metastases, pretracheal lymph node metastasis rate, ipsilateral paratracheal lymph node metastasis, unclear tumor border, size, and age ≤39 years. These features were used to develop a web-based calculator.

Conclusion:

ML is a reliable tool for predicting LN-prRLN metastasis in cN0 PTC patients. The SHAP method provides insights into the XGBoost model, and the resultant web-based calculator is a clinically useful tool to assist in the surgical planning for LN-prRLN dissection.

1 Introduction

Papillary thyroid carcinoma (PTC) is the most common subtype of differentiated thyroid carcinoma (DTC), accounting for approximately 85% of thyroid cancers (1–4). Despite its indolent nature, lymph node metastasis (LNM) occurs in 20-90% of PTC patients, leading to local recurrence and distant metastasis (2, 5, 6). Anatomically, the right recurrent laryngeal nerve (RLN) branches from the first segment of the subclavian artery and divides the right paratracheal lymph nodes into LN-prRLN and anterior RLN lymph nodes (LN-arRLN) (6–8).

Prophylactic central lymph node dissection (CLND) is recommended for high-risk PTC patients (T3 or T4) (2, 9, 10), But is controversial for cN0 patients due to the low LN-prRLN metastasis rate (2.7%-22.4%) (11–13). Extensive dissection increases the risk of complications such as RLN injury, parathyroid damage, or chyle leakage (14–16).However, occult LN-prRLN metastasis, the situation of central compartment without metastasis, but with LN-prRLN metastasis, has been reported in 2.3%-5.86% of cases (17–19). It may cause recurrence, and reoperation can lead to severe complications like permanent vocal cord paralysis or hypoparathyroidism (5, 20). Accurately predicting LN-prRLN metastasis could optimize surgical planning, minimizing unnecessary dissections and associated complications (8, 21, 22).

Although ultrasound is commonly used for examination (4, 12, 23, 24), its accuracy is limited (44.4%-65.8%) due to the deep anatomical location of LN-prRLN and the presence of micrometastases in cN0 patients (25). Multimodal prediction models incorporating clinical, ultrasound, and intraoperative frozen pathology data have improved accuracy but often rely on linear models such as logistic regression or LASSO (11, 21, 26), which may have selection bias and overfitting issues (19, 21, 27), could not allowed clinicians to make preliminary assessments of the likelihood of LN-prRLN metastasis prior to surgery (25).

Machine learning (ML) has shown promise in medical applications, addressing the limitations of linear models by handling large (28–30), diverse datasets through various algorithms (29, 31, 32). This study develops a new ML-based scoring system to predict LN-prRLN metastasis in cN0 PTC patients, comparing traditional models with various ML models, visualizing predictive results, and creating a web-based calculator for clinical decision support.

2 Materials and methods

2.1 Patients

This retrospective study was approved by the Ethics Committee of the First Affiliated Hospital of Chongqing Medical University (Approval No. 2020-181). All methods of this study were performed in accordance with the principles outlined in the Declaration of Helsinki. Clinical records of 4185 PTC patients treated at the First Affiliated Hospital of Chongqing Medical University (Hospital A) from 2016 to 2020 were retrospectively analyzed. Additionally, clinical and pathological data of 651 PTC patients treated at the Women and Children’s Hospital of Chongqing Medical University (Hospital B) from 2018 to 2020 were collected. Inclusion criteria were: age above 18 years; diagnosis of thyroid cancer; complete case data and regional lymph node analysis; no history of previous neck surgery or radiation; no preoperative diagnosis of cN1. Exclusion criteria included: previous neck surgery or radiation; age below 18 years; other types of thyroid malignancies; incomplete case data or lack of regional lymph node analysis; preoperative diagnosis of cN1 (1, 2). Strict adherence to inclusion and exclusion criteria resulted in 1714 PTC patients in the medical cohort and 319 PTC patients in the surgical cohort. Data preprocessing steps for the training, testing and validation sets included: 1. Handling missing values using mode imputation. 2. Categorizing age and tumor diameter into categorical variables using optimal cutoff values determined by ROC curve analysis.

Standardizing and normalizing numerical data to improve data quality and reduce feature disparities. All patients were divided into LN-prRLN metastasis group (LN-prRLN positive group) and non-metastasis group (LN-prRLN negative group) based on postoperative pathological examination results. The medical cohort was randomly divided into training and testing sets (internal validation set) in a 7:3 ratio. The entire surgical cohort was used as the validation set (external validation set). Overall balance of each dataset was verified using chi-square tests (Figure 1; Supplementary Material 1).

Figure 1

2.2 Routine feature variables selection

Clinical Features: Age, sex, BMI. The optimal cutoff value for age determined by ROC curve analysis was 39 years. BMI was categorized according to Chinese and WHO standards: BMI < 18.5: underweight; 18. 5 ≤ BMI < 24: normal weight; BMI ≥ 24: overweight. Ultrasound Features: Collected ultrasound image parameters included tumor boundary (extra-thyroidal extension, irregular/lobulated, smooth/no boundary), aspect ratio (≤1, >1), composition (cystic/spongiform, mixed cystic and solid, solid), internal echo pattern (anechoic, hyperechoic/isoechoic, hypoechoic, very hypoechoic), internal echo homogeneity (homogeneous, heterogeneous), echogenic foci (none/comet-tail artifact, coarse calcifications, peripheral calcifications, microcalcifications), peripheral blood flow (none, abundant), internal tumor vascularization (none, abundant), tumor size defined as the maximum diameter, and location (upper, middle, lower, multiple sites, isthmus). All parameters were measured twice by three radiologists with 10, 15, and 17 years of experience, respectively, and the results were averaged. Cases were re-tested a week later for validation. Pathological features: Extrathyroidal extension (ETE) was defined as tumor invasion beyond the thyroid capsule, including gross ETE (invasion into prethyroid muscles) and maximal ETE (invasion into the trachea, larynx, or recurrent laryngeal nerve). Hashimoto’s thyroiditis was diagnosed based on any of the following criteria: (i) thyroid peroxidase antibody levels >50 IU/mL, (ii) diffuse heterogeneity on ultrasound, (iii) diffuse lymphocytic infiltration on histopathology. Multifocality was confirmed via ultrasound and intraoperative frozen section. Tumor staging was based on intraoperative frozen pathology results (Table 1).

2.3 Surgical procedure and intraoperative frozen section variables

LN-prRLN were identified according to standardized anatomical landmarks as previously described (13, 18). The recurrent laryngeal nerve was traced from its laryngeal entry point caudally to its vagal origin (33, 34). LN-prRLN were defined as lymph nodes in the tissue plane immediately posterior to the recurrent laryngeal nerve, bounded anteriorly by the nerve, posteriorly by the prevertebral fascia, medially by the tracheoesophageal groove, and laterally by the carotid sheath (35, 36). During dissection, LN-prRLN were carefully separated from adjacent nodal stations using meticulous technique while preserving nerve integrity. Each specimen was individually harvested and placed in separate labeled containers indicating “Lymph nodes posterior to recurrent laryngeal nerve “ with detailed anatomical descriptions. All specimens were submitted to pathology with standardized labeling protocols, and a quality assurance protocol requiring cross-verification between surgical notes and pathological reports ensured 100% concordance in nodal station identification.

Surgical procedures: For right-sided PTC patients with tumors <4.0 cm, we routinely performed lobectomy and isthmectomy with ipsilateral CLND. If intraoperative frozen section examination detected LNM, total thyroidectomy was performed. For bilateral PTC, we performed total thyroidectomy with bilateral CLND. In the right paratracheal region, LN-arRLN were labeled as ipsilateral paratracheal lymph nodes to distinguish them from LN-prRLN. Total central lymph nodes (TCLN), excluding LN-prRLN, referred to prelaryngeal, pretracheal, and paratracheal lymph nodes collectively. The surgical procedure has been previously described in our studies (19, 28, 30). surgical specimens were categorized into four subgroups: prelaryngeal, pretracheal, paratracheal, and LN-prRLN, sequentially dissected and labeled, then sent for immediate intraoperative frozen section examination. Postoperative histopathological examination was independently conducted by three pathologists. Intraoperative frozen pathology provided 14 variables of LNM status: prelaryngeal LNM, pretracheal LNM, ipsilateral paratracheal LNM, and TCLNM were binary variables; prelaryngeal, pretracheal, ipsilateral paratracheal, and TCLNM numbers were continuous variables including micro metastases and macro metastases; ratios of prelaryngeal, pretracheal, ipsilateral paratracheal, and TCLNM were calculated as the number of metastatic lymph nodes divided by the total number of lymph nodes dissected in each region, also as continuous variables without cut-off value determination due to non-normal distribution.

2.4 Construction of traditional nomogram prediction model

Univariate analysis was conducted to screen feature variables, excluding those with poor correlation (Table 1). Feature variables with P-values < 0.05 in univariate analysis were included in multivariate logistic regression analysis, and a forest plot was generated (Table 2, Figures 2A, B). Independent risk factors identified through multivariate analysis were incorporated into the nomogram prediction model, drawn using the training set (Figure 2C). The goodness-of-fit was assessed using the Hosmer-Lemeshow method, comparing observed and predicted values, and displayed in a calibration curve (Figure 2D). The evaluation metrics of the traditional nomogram were subsequently compared with those of the ML prediction models.

Figure 2

2.5 Establishment, optimization, and screening of ML prediction models

The architecture of the proposed ML models included four steps: 1. Feature variable selection; 2. Selection and training of ML prediction models; 3. Evaluation and tuning of nine prediction models; 4. Optimization and validation of models using testing and validation sets to identify the optimal prediction model.

A total of 31 feature variables were included. Feature selection methods varied for different prediction models (Supplementary Material 2). For logistic regression, variables were selected through univariate and multivariate analysis. Support vector machines used recursive feature elimination. Decision trees, random forests, gradient boosting, and extreme gradient boosting (XGBoost) used tree-based feature importance. Nearest neighbor algorithms applied recursive feature elimination with cross-validation. Gaussian Naive Bayes used correlation coefficient-based univariate feature selection. Neural networks employed L1 regularization-based model feature selection. The most relevant or important feature variables were incorporated into prediction models.

Supervised learning was used for all ML prediction models, trained on the training set. Model tuning was performed using 10-fold cross-validation and grid search methods to optimize parameters and hyperparameters. Appropriate evaluation metrics were selected to assess model performance on training, testing and validation datasets. Evaluation metrics included ROC curves with AUC values for sensitivity and specificity trade-offs, decision curve analysis (DCA) curves for model utility assessment, calibration curves, and Brier scores for consistency between predicted probabilities and actual LNM rates. Precision-recall curves and area under precision-recall (AUPR) values assessed model performance at various prediction thresholds. Generalization ability and performance were evaluated using the testing set (internal validation) and validation set (external validation). Models were further tuned based on evaluation results, adjusting hyperparameters and adding regularization. Evaluation metrics and curves were recalculated for testing and validation sets to identify the optimal prediction model. Comparative ROC and DCA curves were plotted to assess the prediction value and utility of the optimal prediction model against traditional models (Figure 3).

Figure 3

2.6 Visualization and clinical application of the optimal model

Based on comprehensive evaluation metrics and assessment curves, the optimal prediction model and its parameter settings were identified: detailed parameter search range included ‘eta’: 0.027825594022071243, ‘n_estimators’: 333, ‘gamma’: 0.89, ‘max_depth’: 3, ‘min_child_weight’: 0, ‘colsample_bytree’: 0.3, ‘colsample_bylevel’: 0.0, ‘subsample’: 0.11111111111, ‘reg_lambda’: 0.2, ‘reg_alpha’: 0.Shapley Additive Explanations (SHAP) values for each feature variable in the prediction model were calculated, and a histogram of the top ten contributing feature variables was created. A web-based calculator was developed to visualize and apply the model in clinical settings.

2.7 Statistical analysis

Chi-square tests were used for analyzing binary, unordered multicategorical, and ordered multicategorical count data. Binary logistic regression analysis was employed to identify independent risk factors in metastasis of LN-prRLN. A nomogram prediction model was constructed based on statistically significant indicators from binary logistic regression analysis. Statistical analysis was conducted using R version 4.3.2 in R Studio (R Project for Statistical Computing). The “pROC” package was used for calculating optimal cut-off values for age and tumor diameter among categorical variables. The “foreign” and “rms” packages were used to create the nomogram prediction model and calibration curves. Python (version 3.11.5; Python Software Foundation, Wilmington, DE, USA) was used for variable selection, model training, and evaluation. The scikit-learn Python library (version 0.24) and the XGBoost package (version 1.7.3) were used to create and tune ML models. Performance metrics for evaluating classification performance were also derived using the scikit-learn Python library (version 0.24). The design code for this study using Python software can be obtained on GitHub (https://github.com/ZJ573693/ML-for-LN-prRLN).

3 Results

3.1 Basic characteristics

This study included 1200 patients (70%) in the training set, average age 42.5 ± 11.8 years (314 males, 886 females). The testing set comprised 514 patients (30%), average age 42.8 ± 11.5 years (150 males, 364 females). The validation set included 319 patients, average age 41.3 ± 11.3 years. Detailed ultrasound and clinicopathological characteristics are in Supplementary Material 1. Consistency analyses for continuous variables showed overall data balance (p>0. 05).

3.2 Establishment of traditional nomogram prediction model

Univariate analysis identified age ≤ 39 years, male gender, irregular tumor border, abundant tumor vascularization, tumor size > 10 mm, multifocality, ETE, T staging as T3-T4, prelaryngeal LNM, pretracheal LNM, IPLNM, and TCLNM as associated with LN-prRLN metastasis (Table 1, Figure 2A). Multivariate analysis confirmed irregular tumor border, T staging as T3-T4, prelaryngeal LNM, pretracheal LNM, IPLNM, and TCLNM as independent risk factors (Table 2, Figures 2A, B). These were included in the nomogram prediction model, showing good consistency (Figures 2C, D). ROC values for the training, testing and validation sets were 0.865, 0.814, and 0.836, respectively (Figures 3A, B).

3.3 Selection, optimization, and comparison of optimal ML model

All clinical, ultrasound, and intraoperative frozen pathology variables were used in nine ML algorithms. XGBoost was the optimal model for assessing LN-prRLN metastasis preoperatively and intraoperatively, with AUC values of 0.938, 0.885, and 0.859, respectively (Figures 4A1-A3). Decision curve analysis demonstrated superior net benefit of the XGBoost model across various threshold probabilities (Figures 4B1-B3).Calibration curves and Brier scores indicated low error rates (Figures 4C1-C3). Precision-recall curves showed high precision and accuracy for XGBoost (Figures 4D1-D3). Compared to the traditional model, XGBoost had higher AUC values and better clinical utility (Table 3; Figures 3C, D).

Figure 4

3.4 Comparison of the optimal ML prediction model (XGBoost) and the traditional prediction model (nomogram prediction)

ROC curve analysis revealed that the XGBoost model had higher AUC values across all datasets: 0.938 (training set), 0.859 (testing set), and 0.885 (validation set), compared to the nomogram’s 0.865, 0.814, and 0.836, respectively (Figures 3A, C). The DCA curves further confirmed the superior clinical utility of the XGBoost model, as its curve consistently outperformed the nomogram’s curve (Figures 3B, D). The learning curve also showed higher mean AUC values for the XGBoost model, especially in the training and testing sets (Figures 5A, B). Based on the PMRA for predicting LN-prRLN metastasis, the XGBoost model had the highest probability of winning. Another eight models showed significant differences from the XGBoost model, though their probabilities of winning were all below 0.5 (Table 4 and Figures 5C, D).

Figure 5

3.5 Visualization and application of XGBoost model using SHAP

SHAP values for all patients in the training set were visualized in bar charts (Figure 6A). The classification bar chart showed the distribution of LN-prRLN metastasis cases (red) and non-metastasis cases (blue) (Figure 6B). The top ten parameters were IPLNMR, TCNLNM, TCLNM, TCLNMR, IPNLNM, pretracheal LNM ratio, IPLNM, tumor border, size, and age. SHAP summary plots reflected the relationship between parameters and predicted probabilities (Figure 6C). Higher SHAP values for certain LNM indicated higher LN-prRLN metastasis likelihood; conversely, higher values for age > 39 years and tumor diameter < 10 mm indicated lower likelihood. The tumor border had a positive effect, where a clear border indicated a lower likelihood of metastasis. SHAP interaction and decision plots demonstrated parameter effects on predictions (Figures 6D, 7).

Figure 6

Figure 7

3.6 Creating a web calculator for feature contribution visualization using SHAP

The contribution calculation heatmap and bar chart visually presented each feature’s contribution to the model output (Figures 8A, B). The top ten ranked feature variables were IPLNMR (SHAP value of 0.26, contribution ratio of 16.846%), TCNLNM (SHAP value of 0.24, 15.601%), TCLNM (SHAP value of 0.23, 15.104%), TCLNMR (SHAP value of 0.18, 11.722%), IPNLNM (SHAP value of 0.16, 10.624%), pretracheal LNM ratio (SHAP value of 0.13, 8.804%), IPLNM (SHAP value of 0.1, 6.856%), unclear tumor border (SHAP value of 0.09, 6.058%), size (SHAP value of 0.07, 4.427%), and age ≤39 years (SHAP value of 0.06, 3.958%), with a gradual decrease in the positive contribution to LN-prRLN metastasis (Figure 8C). The top ten feature variables were selected to create a web calculator, with contribution ratios shown (Figure 8D). The web calculator can be accessed at:(http://121.41.36.155:9003/static/html/index.html).

Figure 8

4 Discussion

The rate of metastasis in LN-prRLN varies between 6% and 34.1% in literature (3, 4, 8, 27), with cN0 PTC patients showing rates up to 22.4% (14, 25, 37). There remains controversy regarding the prophylactic dissection of LN-prRLN (37, 38), focusing on balancing surgical benefits and avoiding complications. Preoperative prediction of LN-prRLN metastasis in cN0 PTC is critical for precise management. Accurate lymph node assessment is essential for selective lymph node dissection (12, 39). Therefore, a reliable and rapid lymph node evaluation tool is necessary. In this multicenter retrospective study, we developed an integrated model combining clinical, ultrasound, and intraoperative frozen pathology assessments to predict LN-prRLN metastasis in cN0 PTC patients. This model helps determine the necessity and timing of LN-prRLN dissection.

Several studies have explored factors influencing LN-prRLN metastasis, mostly focusing on cN1 patients (3, 14, 21, 39). For instance, Zou M et al. identified age < 45 years (p = 0.005; OR 2.155), male gender (p = 0.043; OR 1.657), tumor diameter > 1.0 cm (p = 0.042; OR 1.702), microcalcifications (p = 0.022; OR 1.980), and lateral LNM on ultrasound (p = 0.001; OR 2.578) as independent risk factors for LN-prRLN metastasis (17);Bae SY et al. found tumor size>1.0 cm (p= 0.009, OR = 2.654) and CLNM (p< 0.001, OR = 5.005) significantly associated with LN-prRLN metastasis (18). Chang H et al. suggested that CLNM (excluding LN-prRLN metastasis, p<0.001,OR=14.715) and LLNM (p=0.016,OR=4.383) significantly predict LN-prRLN metastasis (13). In these studies (14, 23, 26), cN1 patients were analyzed alongside cN0 patients, although cN0 patients also have a high rate of occult LNM. Our study found a 14.177% rate of LN-prRLN metastasis in cN0 PTC, similar to previous studies (4, 25). Prophylactic LN-prRLN dissection requires more accurate predictive models and characteristic variables for precision.

Tumor size is a crucial factor influencing LN-prRLN metastasis, with a cutoff value typically around 1.0 cm. Qi GF et al. showed that a maximum tumor diameter > 1.0 cm (p < 0.001,OR: 5.729) is an independent risk factor for LN-prRLN metastasis (27). Xiao X et al. found a tumor diameter > 1.0 cm (p<0.001, OR = 2.897) to be an independent risk factor (37). Our study identified a tumor size cutoff of 10 mm for LN-prRLN metastasis, consistent with other studies (8, 9, 18). In SHAP analysis, size > 10 mm ranked ninth, contributing 4.427%. Indistinct tumor boundaries were the only significant ultrasound feature (SHAP value 0.09), consistent with another research (27). Lower tumor location and microcalcifications also significantly influence LN-prRLN metastasis, albeit with smaller contributions, aligning with previous studies (8, 11). Age is a recognized factor affecting LNM (8, 22), with our ROC analysis setting the optimal cutoff at 39 years, possibly due to early-stage detection through routine examinations.

Most published studies analyze postoperative pathology or clinical data, lacking methods to guide intraoperative decisions (39, 40). Early-stage patients need suitable indicators for preoperative and intraoperative LN-prRLN status assessment to avoid unnecessary lymph node dissection and nerve injury (41, 42). Intraoperative frozen section provides immediate pathological results, aiding surgical decisions and avoiding unnecessary reoperations (43–45). Intraoperative frozen section accuracy for thyroid cancer CLNM is typically 80%-90% (46, 47). Our study’s intraoperative frozen pathology identified variables contributing 85.557% to LN-prRLN metastasis prediction. The most significant factor was IPLNM, which is the primary independent factor in traditional models (P<0.001, OR = 3.605). ML-based SHAP analysis of the optimal model revealed that the SHAP value for the proportion of IPLNM was highest at 0.26, constituting 16.846% (Figure 8C). Clustering analysis through SHAP visualization identified IPLNM status (presence, number, and proportion) as the primary category influencing LN-prRLN metastasis. Consistent with previous studies, this research provides more accurate quantification of feature variables. For instance, studies by Shen B et al. identified paratracheal LNM as an independent risk factor for LN-prRLN (p<0.001, OR = 5.357) (1), while Yang H et al. highlighted LN-arRLN as an independent risk factor (p<0. 001, OR = 4. 386) (11). Similar findings were reported in studies (14, 37, 48). The second most influential factor on LN-prRLN metastasis in this study is TCLNM, excluding LN-prRLN metastasis, which ranks second in traditional prediction models (P = 0.013, OR = 2.633) (Figures 2A, B). SHAP analysis clustering identified TCNLNM with the highest SHAP value of 0.24, accounting for 15.601%. Consistent with prior research, studies by Bae SY found significant correlations between TCLNM (P = 0.001, OR = 5.005) and LN-prRLN metastasis (18). Kim D et al. ‘s study showed TCLNM (p<0.001,OR=5.203) was LN-prRLN metastasis (10). Furthermore, Ling Y et al. interpreted the role of intraoperative frozen sections, highlighting the number of metastatic lymph nodes during LN-prRLN as a potential predictive factor (P = 0.014, OR = 1.320). The false negative rate was quite low (6.2%), suggesting that cN0 patients may not require LN-prRLN resection when lymph node biopsy does not show metastasis. Number of prelaryngeal, pretracheal, and paratracheal LNM are risk factors for LN-prRLN metastasis (25). Our previous study also confirmed that number of prelaryngeal, pretracheal, and paratracheal LNM (p<0.001, OR = 21.078) were independent predictors of LN-prRLN metastasis (19). This study further elucidates paratracheal LNM as the third category of risk factors influencing LN-prRLN metastasis, possibly related to thyroid lymphatic drainage anatomically through the lower thyroid veins, distributing lymph from the thyroid to the paratracheal area, then to the lateral neck nodes, and finally returning to the lymphatic duct or thoracic duct, anatomically referred to as the descending pathway (7, 15).

Of greater significance in this study is the comparison between traditional prediction models and artificial intelligence ML features. Traditional prediction models play an important role in predicting LNM to some extent, but they have limitations. In this study, the predictive value of traditional prediction models was relatively high, with a training set AUC of 0.865 (Figure 3A), similar to previous reports (14, 18, 21). However, XGBoost selected through ML showed higher predictive value with a training set AUC of 0.938, and good performance in the testing and validation sets with AUCs of 0.859 and 0.885, respectively. DCA curves, calibration curves, and precision-recall curves demonstrated the model’s clinical applicability, high accuracy, and low error rate. The XGBoost model enables precise prediction of LN-prRLN metastasis in cN0 PTC patients and successfully identifies key parameters influencing LN-prRLN metastasis prediction. SHAP provides reasonable visual explanations for predictions, encompassing both positive and negative impacts. In this study, we not only demonstrated the overall feature importance for predicting LN-prRLN metastasis but also developed a network calculator using SHAP visualizations of contribution degrees. This tool offers practical utility for clinicians to predict LN-prRLN metastasis based on preoperative and intraoperative feature variables, guiding surgical approaches and precise surgical planning.

Limitations of this study warrant discussion. Firstly, to mitigate the issue of a relatively small sample size, we subjected the constructed XGBoost model to ten-fold cross-validation to demonstrate sufficient accuracy. Secondly, this study employed a retrospective design. However, to enhance model credibility, we conducted internal and external validations, achieving high AUC values. Additionally, we compared XGBoost evaluation results with those of traditional ML model, further proving its superior performance. Thirdly, during the construction of the XGBoost model and the creation of the network calculator, manual input of relevant parameters was required. When clinicians record image features, some unknown or overlooked aspects may lead to the loss of hidden relationships. To minimize this phenomenon, we conducted consistent data analysis to ensure model accuracy. Fourthly, our model was developed using single-center data, which may introduce potential biases related to institutional practices and patient demographics. Although external validation was performed using 319 patients from another hospital in the same city, this validation remains geographically limited and may not fully represent broader clinical populations across different regions. In conclusion, SHAP provides reasonable visual explanations for the XGBoost model, applicable independently for predicting LN-prRLN metastasis in cN0 PTC patients, including both positive and negative impacts, significantly enhancing clinician confidence in the clinical application of the XGBoost model. Future studies are expected to validate the XGBoost model through prospective studies with broad samples from multiple centers, widely applying it in clinical practice to guide personalized treatment of LN clearance in PTC patients.

5 Conclusion

We developed and internally and externally validated nine ML prediction models using preoperative and intraoperative frozen pathological features. These models were compared with traditional prediction models, and the XGBoost algorithm emerged as the optimal predictive model. Through SHAP visualization applications, we identified the top ten feature variables influencing LN-prRLN metastasis in cN0 PTC patients and calculated their contributions using SHAP analysis. Cluster analysis demonstrated the importance of intraoperative frozen pathological examination, which can be utilized for personalized prediction of LN-prRLN metastasis in cN0-PTC patients. ML-based prediction models accurately identify whether patients are at risk of LN-prRLN metastasis. The online network calculator created based on SHAP analysis of contribution percentages serves as an easy-to-use tool for clinicians to make precise surgical decisions. In the future, we aim to integrate imaging and molecular data to further optimize the model’s performance in the field of personalized precision medicine. Additionally, more multicenter studies are needed to further validate our findings.

References

• 1

  Wang H Li Q Tian T Liu B Tian R . Improving the risk prediction of the 2015 ATA recurrence risk stratification in papillary thyroid cancer. J Clin Endocrinol Metab. (2025) 110:534–41. doi: 10.1210/clinem/dgae465

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2

  Ringel MD Sosa JA Baloch Z Bischoff L Bloom G Brent GA et al . 2025 american thyroid association management guidelines for adult patients with differentiated thyroid cancer. Thyroid. (2025) 35:841–985. doi: 10.1177/10507256251363120

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3

  Gong Y Zuo Z Tang K Xu Y Zhang R Peng Q et al . Multimodal predictive factors of metastasis in lymph nodes posterior to the right recurrent laryngeal nerve in papillary thyroid carcinoma. Front Endocrinol (Lausanne). (2023) 14:1187825. doi: 10.3389/fendo.2023.1187825

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4

  Shen B Zhou C Xu C Yang B Wu X Fu X et al . Ultrasound-based radiomics for predicting metastasis in the lymph nodes posterior to the right recurrent laryngeal nerve in patients with papillary thyroid cancer. Curr Med Imaging. (2023) 31. doi: 10.2174/0115734056257332231024112410

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5

  Wang Y Wang D Chen L Guo K Sun T . A comparison of two operation methods revealed the risk factors and the necessity of LN-prRLN dissection in papillary thyroid carcinoma: A retrospective cohort study in FUSCC. Int J Endocrinol. (2020) 2020:7162793. doi: 10.1155/2020/7162793

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6

  Newman SK Harries V Wang L McGill M Ganly I Girshman J et al . Invasion of a recurrent laryngeal nerve from small well-differentiated papillary thyroid cancers: patient selection implications for active surveillance. Thyroid. (2022) 32:164–9. doi: 10.1089/thy.2021.0310

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 7

  Bellier A Pardo Vargas JS Cassiba J Desbrest P Guigui A Chaffanjon P . Anatomical variations in distal portion of the thoracic duct-A systematic review. Clin Anat. (2020) 33:99–107. doi: 10.1002/ca.23476

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8

  Shao J Wang X Yu H Ding W Xu B Ma D et al . Preoperative prediction of metastatic lymph nodes posterior to the right recurrent laryngeal nerve in cN0 papillary thyroid carcinoma. Cancer Manage Res. (2024) 16:421–9. doi: 10.2147/cmar.s454607

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9

  Du W Fang Q Zhang X Dai L . Metastasis of cN0 papillary thyroid carcinoma of the isthmus to the lymph node posterior to the right recurrent laryngeal nerve. Front Endocrinol (Lausanne). (2021) 12:677986. doi: 10.3389/fendo.2021.677986

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10

  Kim D Kwon HK Shin SC Sung ES Jeon YK Kim BH et al . Right posterior paratracheal lymph nodes metastasis is one of the predictive factors in right-sided papillary thyroid carcinoma. Surgery. (2019) 166:1154–9. doi: 10.1016/j.surg.2019.06.024

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11

  Yang H Tao L . Lymph node posterior to the right recurrent laryngeal nerve metastasis in right lobe T1a papillary thyroid carcinoma: A retrospective cohort study. Cancer Control. (2023) 30:10732748221149819. doi: 10.1177/10732748221149819

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12

  Zhou T Wu F Zhao L Jiang K Luo D . A meta-analysis of risk factors for lymph node posterior to the right recurrent laryngeal nerve metastasis in papillary thyroid carcinoma. Gland Surg. (2021) 10:1841–51. doi: 10.21037/gs-21-177

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13

  Chang H Yoo RN Kim SM Kim BW Lee YS Lee SC et al . The clinical significance of the right para-oesophageal lymph nodes in papillary thyroid cancer. Yonsei Med J. (2015) 56:1632–7. doi: 10.3349/ymj.2015.56.6.1632

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14

  Zhou M Duan Y Ye B Wang Y Li H Wu Y et al . Pattern and predictive factors of metastasis in lymph nodes posterior to the right recurrent laryngeal nerve in papillary thyroid carcinoma. Front Endocrinol (Lausanne). (2022) 13:914946. doi: 10.3389/fendo.2022.914946

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15

  Sharma AK Sahli ZT Mathur A . Bilateral chylothorax following reoperative central neck dissection for metastatic papillary thyroid cancer. BMJ Case Rep. (2018) 2018:bcr-2018-224916. doi: 10.1136/bcr-2018-224916

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16

  Zhou J Li D Xiao Q Zhuang Y Yang T Xue S et al . Bilateral chylothorax following papillary thyroid carcinoma with cervical lymph node dissection: Case report and comprehensive review of the literature. Med (Baltimore). (2024) 103:e40371. doi: 10.1097/MD.0000000000040371

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 17

  Zou M Wang YH Dong YF Lai XJ Li JC . Clinical and sonographic features for the preoperative prediction of lymph nodes posterior to the right recurrent laryngeal nerve metastasis in patients with papillary thyroid carcinoma. J Endocrinol Invest. (2020) 43:1511–7. doi: 10.1007/s40618-020-01238-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18

  Bae SY Yang JH Choi MY Choe JH Kim JH Kim JS . Right paraesophageal lymph node dissection in papillary thyroid carcinoma. Ann Surg Oncol. (2012) 19:996–1000. doi: 10.1245/s10434-011-2144-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 19

  Zhou J Li D Xiao Q Zhuang Y Yang T Xue S et al . Individualized prediction of metastatic involvement of lymph nodes posterior to the right recurrent laryngeal nerve in papillary thyroid carcinoma. Onco Targets Ther. (2019) 12:9077–84. doi: 10.2147/OTT.S220926

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20

  Li J He G Tong Y Tao L Xie L Gao L et al . A novel scoring system for the risk of papillary thyroid cancer metastases in lymph nodes posterior to the right of the recurrent laryngeal nerve. Endocr Pract Jan. (2021) 27:15–20. doi: 10.4158/EP-2020-0129

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21

  Lu KN Zhang Y Da JY Zhou TH Zhao LQ Peng Y et al . A novel scoring system for predicting the metastases of posterior right recurrent laryngeal nerve lymph node involvement in patients with papillary thyroid carcinoma by preoperative ultrasound. Front Endocrinol (Lausanne). (2021) 12:738138. doi: 10.3389/fendo.2021.738138

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22

  Zhu F Shen Y Zhu L Chen L Li F Xie X et al . Impact of thyroid carcinoma invasion of recurrent laryngeal nerve on cervical lymph node metastasis. Endocrine. (2024) 86:654–63. doi: 10.1007/s12020-024-03879-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23

  Li C Xiang J Wang Y . Risk factors for predicting lymph nodes posterior to right recurrent laryngeal nerve (LN-prRLN) metastasis in thyroid papillary carcinoma: A meta-analysis. Int J Endocrinol. (2019) 2019:7064328. doi: 10.1155/2019/7064328

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24

  Tessler FN Middleton WD Grant EG Hoang JK Berland LL Teefey SA et al . ACR thyroid imaging, reporting and data system (TI-RADS): white paper of the ACR TI-RADS committee. J Am Coll Radiol. (2017) 14:587–95. doi: 10.1016/j.jacr.2017.01.046

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 25

  Ling Y Zhang L Li K Zhao Y Zhao J Jia L et al . Carbon nanoparticle-guided intraoperative lymph node biopsy predicts the status of lymph nodes posterior to right recurrent laryngeal nerve in cN0 papillary thyroid carcinoma. Gland Surg. (2021) 10:1554–63. doi: 10.21037/gs-20-920

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 26

  Hou J Shan H Zhang Y Fan Y Wu B . Risk factors of metastasis to the lymph nodes posterior to the right recurrent laryngeal nerve in papillary thyroid carcinoma. Eur Arch Otorhinolaryngol. (2020) 277:881–6. doi: 10.1007/s00405-019-05748-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27

  Lu KN ZhaQi GF Feng JW Wu WX Ye J Hong LZ et al . Prediction model of lymph node metastasis posterior to the right recurrent laryngeal nerve. Eur Arch Otorhinolaryngol. (2023) 280:3429–35. doi: 10.1007/s00405-023-07946-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 28

  Zhou J Li D Ren J Huang C Yang S Chen M et al . Machine learning: A multicentre study on predicting lateral lymph node metastasis in cN0 papillary thyroid carcinoma. J Clin Endocrinol Metab. (2025) 110:2762–76. doi: 10.1210/clinem/dgaf070

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 29

  Han H Sun H Zhou C Wei L Xu L Shen D et al . Development and validation of a machine learning model for central compartmental lymph node metastasis in solitary papillary thyroid microcarcinoma via ultrasound imaging features and clinical parameters. BMC Med Imaging. (2025) 25:228. doi: 10.1186/s12880-025-01757-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 30

  Chun L Wang D He L Li D Fu Z Xue S et al . Explainable machine learning model for predicting paratracheal lymph node metastasis in cN0 papillary thyroid cancer. Sci Rep. (2024) 14:22361. doi: 10.1038/s41598-024-73837-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 31

  Liu X Li H Zhang L Gao Q Wang Y . Development and validation of a multidimensional machine learning-based nomogram for predicting central lymph node metastasis in papillary thyroid microcarcinoma. Gland Surg. (2025) 14:344–57. doi: 10.21037/gs-2024-508

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 32

  Wang H Zhang C Li Q Tian T Huang R Qiu J et al . Development and validation of prediction models for papillary thyroid cancer structural recurrence using machine learning approaches. BMC Cancer. (2024) 24:427. doi: 10.1186/s12885-024-12146-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 33

  Wu CW Chai YJ Dionigi G Chiang FY Liu X Sun H et al . Recurrent laryngeal nerve safety parameters of the Harmonic Focus during thyroid surgery: Porcine model using continuous monitoring. Laryngoscope. (2015) 125:2838–45. doi: 10.1002/lary.25412

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 34

  Dralle H Sekulla C Haerting J Timmermann W Neumann HJ Kruse E et al . Risk factors of paralysis and functional outcome after recurrent laryngeal nerve monitoring in thyroid surgery. Surgery. (2004) 136:1310–22. doi: 10.1016/j.surg.2004.07.018

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 35

  Garas G Okabayashi K Ashrafian H Shetty K Palazzo F Tolley N et al . Which hemostatic device in thyroid surgery? A network meta-analysis of surgical technologies. Thyroid. (2013) 23:1138–50. doi: 10.1089/thy.2012.0588

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 36

  Pacilli M Tartaglia N Gerundo A Pavone G Fersini A Ambrosi A . Energy based vessel sealing devices in thyroid surgery: A systematic review to clarify the relationship with recurrent laryngeal nerve injuries. Med (Kaunas). (2020) 56:651. doi: 10.3390/medicina56120651

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 37

  Xiao X Wu Y Zou L Chen Y Zhang C . Value of dissection of lymph nodes posterior to the right recurrent laryngeal nerve in patients with cN(0) papillary thyroid carcinoma. Gland Surg. (2022) 11:1204–11. doi: 10.21037/gs-22-337

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 38

  Wang B Zhu CR Liu H Yao XM Wu J . Relationship between pretracheal and/or prelaryngeal lymph node metastasis and paratracheal and lateral lymph node metastasis of papillary thyroid carcinoma: A meta-analysis. Front Oncol. (2022) 12:950047. doi: 10.3389/fonc.2022.950047

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 39

  Liu H Yao XY Zheng ZQ Chen SJ . Analysis of risk factors for posterior laryngeal recurrent nerve metastasis in micropapillary thyroid carcinoma. Endocr J. (2022) 69:131–7. doi: 10.1507/endocrj.EJ21-0073

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 40

  Lim YS Lee JC Lee YS Lee BJ Wang SG Son SM et al . Lateral cervical lymph node metastases from papillary thyroid carcinoma: predictive factors of nodal metastasis. Surgery. (2011) 150:116–21. doi: 10.1016/j.surg.2011.02.003

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 41

  Kim KE Kim EK Yoon JH Han KH Moon HJ Kwak JY . Preoperative prediction of central lymph node metastasis in thyroid papillary microcarcinoma using clinicopathologic and sonographic features. World J Surg. (2013) 37:385–91. doi: 10.1007/s00268-012-1826-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 42

  Lee DW Ji YB Sung ES Park JS Lee YJ Park DW et al . Roles of ultrasonography and computed tomography in the surgical management of cervical lymph node metastases in papillary thyroid carcinoma. Eur J Surg Oncol. (2013) 39:191–6. doi: 10.1016/j.ejso.2012.07.119

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 43

  Lv X Lu JJ Song SM Hou YR Hu YJ Yan Y et al . Prediction of lymph node metastasis in patients with papillary thyroid cancer based on radiomics analysis and intraoperative frozen section analysis: A retrospective study. Clin Otolaryngol. (2024) 49:462–74. doi: 10.1111/coa.14162

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 44

  Kim MJ Kim HJ Park CS Kim BW . Frozen section analysis of central lymph nodes in papillary thyroid cancer: the significance in determining the extent of surgery. Gland Surg. (2022) 11:640–50. doi: 10.21037/gs-22-15

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 45

  Shen B Zhang Z Shi X Cao C Zhang Z Hu Z et al . Real-time intraoperative glioma diagnosis using fluorescence imaging and deep convolutional neural networks. Eur J Nucl Med Mol Imaging. (2021) 48:3482–92. doi: 10.1007/s00259-021-05326-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 46

  Wang Y Zhang S Zhang M Zhang G Chen Z Wang X et al . Prediction of lateral lymph node metastasis with short diameter less than 8 mm in papillary thyroid carcinoma based on radiomics. Cancer Imaging. (2024) 24:155. doi: 10.1186/s40644-024-00803-7

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 47

  Peng Y Zhang ZT Wang TT Wang Y Li CH Zuo MJ et al . Prediction model of pathologic central lymph node negativity in cN0 papillary thyroid carcinoma. Front Oncol. (2021) 11:727984. doi: 10.3389/fonc.2021.727984

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 48

  Liu WQ Yang JY Wang XH Cai W Li F . Analysis of factors influencing cervical lymph node metastasis of papillary thyroid carcinoma at each lateral level. BMC Surg. (2022) 22:228. doi: 10.1186/s12893-022-01678-w

  • Pubmed Abstract
  • CrossRef
  • Google Scholar