Shuai Cui1†
Qifan Liu1†
Hailong Wang1
Husha Li1
Wei Li1
Chenlong Li2
Leilei Bi3
Yang Mu4
Wenjing Guo1
Jundong Yao1*
Zhoulong Zhang1- 1Department of Ultrasound, The First Affiliated Hospital, College of Clinical Medicine of Henan University of Science and Technology, Luoyang, China
- 2Department of Ultrasound, General Hospital of Pingmei Shenma Medical Group, Pingdingshan, China
- 3Department of Ultrasound, Xiangyang Hospital of Traditional Chinese Medicine, Xiangyang, China
- 4Department of Ultrasound, The Fourth Affiliated Hospital of Xinjiang Medical University, Urumqi, China
Objective: To establish a combined model based on ultrasound radiomics combined with multimodal ultrasound and evaluate its value in diagnosing benign and malignant nodules classified as Chinese-Thyroid Imaging Report and Data System (C-TIRADS) 4A.
Methods: Prospective collection of data from 446 patients with thyroid nodules classified as C-TIRADS 4A between December 2023 and August 2024. Based on the enrollment timeline, patients were divided into a training set (n=312) and a test set (n=134) in a 7:3 ratio. Using clinical information, multimodal ultrasound features, and radiomics features, a radiomics model was constructed using the Random Forest (RF) machine learning algorithm. Logistic regression was employed to develop the multimodal ultrasound model and the combined model. The predictive efficiency and accuracy of these models were evaluated using Receiver Operating Characteristic (ROC) curves, calibration curves, and Decision Curve Analysis (DCA). The diagnostic efficacy of junior physicians assisted by the ultrasound radiomics model was compared with that of senior physicians. DeLong’s test was performed to compare the diagnostic performance of the models.
Results: Multivariate analysis revealed that age (≤51 years), Sound Touch Elastography mean stiffness (STE Mean), orientation (vertical), margin (blurred), and margin (irregular) were independent risk factors for papillary thyroid carcinoma, and the multimodal ultrasound model was established. Based on 17 ultrasound radiomics features, a radiomics model was constructed using the RF machine learning algorithm. The combined model was developed by combining the two aforementioned models. In the training set, the areas under the curve (AUC) of the multimodal ultrasound model, ultrasound radiomics model, and combined model were 0.852, 0.940 and 0.956, respectively. In the test set, the AUC were 0.804, 0.832 and 0.863, respectively. DeLong’s test showed that the combined model performed best in the training set, and in the test set, the combined model outperformed the multimodal ultrasound model but showed no significant difference compared to the radiomics model. DCA indicated that the combined model achieved higher net benefits within a specific threshold probability range (0.15-0.90).
Conclusion: The combined model exhibits robust diagnostic capability in distinguishing benign from malignant thyroid nodules classified as C-TIRADS 4A.
1 Introduction
Thyroid cancer is one of the fastest-growing malignancies in terms of incidence worldwide. Over the past three decades, its global incidence has risen significantly, currently ranking as the seventh most common cancer globally (1, 2). Papillary thyroid cancer (PTC) accounts for more than 80% of all thyroid cancer cases (3). Although its prognosis is relatively favorable, the issue of overtreatment during diagnosis and management has become increasingly prominent.
Ultrasound examination has become the preferred imaging tool for risk assessment of thyroid nodules due to its non-invasive, real-time, and reproducible advantages. The establishment of the Chinese-Thyroid Imaging Report and Data System (C-TIRADS) has provided an important framework for standardizing the risk assessment of thyroid nodule malignancy (4). However, the malignancy risk of C-TIRADS 4A nodules spans a relatively wide range (2%-10%) (5, 6), leading to a significant number of patients undergoing unnecessary fine-needle aspiration biopsies or surgeries due to diagnostic uncertainty. This not only increases healthcare costs but may also cause patient anxiety. Therefore, there is an urgent need for a more precise differentiation method to optimize the management strategy for C-TIRADS 4A nodules.
Currently, the traditional C-TIRADS primarily relies on the morphological features of grayscale ultrasound (such as margins, echogenicity, calcifications, etc.) for risk stratification. However, the introduction of multimodal ultrasound techniques, including elastography, Color Doppler Flow Imaging (CDFI), and contrast-enhanced ultrasound, has provided multidimensional information for assessing nodule stiffness, vascular characteristics, and microcirculation. For example, Gong et al. (7) demonstrated that combining grayscale ultrasound with contrast-enhanced ultrasound improved the diagnostic the areas under the curve (AUC) from 0.844 to 0.897. Additionally, Nattabi et al. (8) further showed that nodule stiffness measured by Shear Wave Elastography (SWE) is significantly positively correlated with malignancy risk. Nevertheless, these techniques still heavily depend on physician experience, and there are notable differences in diagnostic consistency among physicians of varying experience levels, with the learning curve for junior physicians being particularly challenging.
The emergence of radiomics offers a novel approach to addressing the aforementioned challenges. This technology extracts high-throughput deep imaging features (such as texture, morphology, heterogeneity, etc.) and combines them with machine learning algorithms to construct objective quantitative models. It has already demonstrated remarkable potential in the differential diagnosis of tumors such as breast cancer and liver cancer (9–11). In the field of thyroid imaging, the application of ultrasound radiomics is still in its exploratory stages. However, preliminary studies suggest that it can effectively capture malignant features that are difficult to identify using traditional methods, such as microstructural heterogeneity within nodule (12). Nevertheless, the complementary nature of multimodal ultrasound parameters (e.g., vascular characteristics, elasticity modulus) and radiomics features has not yet been fully explored, and there is a lack of validation based on multicenter data.
This study aims to develop a machine learning model that integrates multimodal ultrasound features and ultrasound radiomics for the differentiation of benign and malignant thyroid nodules classified as C-TIRADS 4A. Through prospective multicenter data validation, the diagnostic performance of the model and its potential to assist junior physicians will be evaluated. The goal is to reduce unnecessary invasive procedures, shorten the learning curve for junior physicians, and enhance their diagnostic capabilities, thereby optimizing clinical decision-making processes.
2 Materials and methods
2.1 Study population
Prospectively collect data from patients with thyroid nodules classified as C-TIRADS 4A by ultrasound in multicenter medical institutions from December 2023 to August 2024. The inclusion criteria were as follows: ① age ≥ 18 years; ② ultrasound diagnosis of single or multiple C-TIRADS 4A nodules; ③ undergoing fine-needle aspiration biopsy (FNA) combined with BRAF V600E gene testing or surgical resection with definitive pathological results. The exclusion criteria were as follows: ① poor-quality ultrasound images (suboptimal image quality or insufficient resolution); ② history of neck surgery, cancer, or pregnancy; ③ incomplete clinical information; ④ non-PTC malignancies. A total of 446 patients were ultimately included, with 314 cases from The First Affiliated Hospital of Henan University of Science and Technology, 53 cases from General Hospital of Pingmei Shenma medical group, 48 cases from Xiangyang Hospital of Traditional Chinese Medicine, and 31 cases from The Fourth Affiliated Hospital of Xinjiang Medical University. Based on the enrollment timeline, patients were divided into a training set and a test set in a 7:3 ratio. Figure 1 illustrates the patient enrollment flowchart. This study received approval from the Ethics Committee of the First Affiliated Hospital of Henan University of Science and Technology (2024-03-K0160), All participants provided written informed consent, in compliance with regulations of the institution and the guidelines of the Declaration of Helsinki.
Figure 1. Flowchart of patient enrollment.
2.2 Ultrasound image acquisition
Prior to data collection, all ultrasound physician underwent centralized training on C-TIRADS 4A classification criteria, including grayscale ultrasound features (e.g., margins, echogenicity, calcifications) and multimodal parameters (e.g., elastography thresholds). All participating centers utilized identical ultrasound systems (Mindray Resona I9) equipped with linear array probes (frequency range: 7.5–12 MHz). Imaging parameters were standardized across centers, including gain (60–70 dB), depth (2.5-3.5 cm), dynamic range (50–60 dB), and mechanical index (MI: 0.8-1.0). The patient was positioned supine, with the neck fully extended to expose the thyroid area. Grayscale images, color Doppler images, and dynamic images of nodules were collected in both transverse and longitudinal sections. Elastography settings were uniformly configured to ensure consistent Young’s modulus calculations. Subsequently, three types of elastography images were acquired: Sound Touch Elastography (STE), Strain Elastography (SE), and Sound Touch Quantification (STQ). All images are stored in DICOM format.
2.3 Ultrasound image analysis
Ultrasound image feature was conducted by two doctors, Jundong Yao and Wei Li, both possessing 10 years of experience in ultrasonic diagnosis of thyroid diseases. They independently analyzed the images without access to the patients’ clinical information or pathological results. In cases of differing opinions, a third doctor, Zhoulong Zhang, who has over 30 years of experience in ultrasound diagnosis, made the final decision. This study evaluates several features of nodules, including maximum diameter of the nodule, multiplicity (solitary/multiple), margin (smooth/ill-defined/irregular), halo sign (present/absent), composition (cystic-solid/solid/predominantly solid), echogenicity (isoechoic/markedly hypoechoic/hypoechoic/hyperechoic/heterogeneous), echo texture (homogeneous/heterogeneous), orientation (vertical/parallel), calcification (absent/coarse calcification/microcalcification/indeterminate punctate echogenic foci/peripheral calcification), relationship to capsule (>2 mm/≤2 mm/extracapsular extension), and CDFI classification according to Alder (levels 0/I/II/III). The region of interest (ROI) for elastography is defined by manually tracing the contours of nodules using a specialized machine. The shell denotes a machine-generated boundary that extends 2 mm beyond the nodule following the tracing process. By outlining the nodule area, the system automatically calculates the STE mean stiffness (STE Mean), STE maximum stiffness (STE Max), STE minimum stiffness (STE Min), STE standard deviation (STE SD), SE mean stiffness (SE Mean), SE maximum stiffness (SE Max), SE minimum stiffness (SE Min), SE standard deviation (SE SD), STQ mean stiffness (STQ Mean), STQ maximum stiffness (STQ Max), STQ minimum stiffness (STQ Min), STQ standard deviation (STQ SD).
2.4 Pathological analysis
Ultrasound-guided fine-needle aspiration biopsy was performed by ultrasonologists who had completed standardized training and obtained qualification certificates. A 23G fine needle was utilized to efficiently extract samples from the thyroid lesion under ultrasound guidance. The cells were preserved in a liquid-based culture medium for subsequent genetic analysis. The test samples were then sent for pathological examination and genetic testing. A pathological diagnosis of PTC is deemed positive. The BRAF V600E mutation (a key genetic driver of PTC) is strongly associated with tumor aggressiveness, extrathyroidal extension, and lymph node metastasis (13). Its detection enhances diagnostic specificity for malignancy. If the BRAF V600E mutation is detected, it is recorded as positive; if not, it is noted as negative. All results were followed up for 6 months.
2.5 Model building
2.5.1 Construction of the multimodal ultrasound model
Univariate logistic regression analysis was conducted on the selected variables, which included gender, age, and ultrasound image characteristics such as STE Mean, STE Max, STE Min, STE SD, SE Mean, SE Max, SE Min, SE SD, STQ Mean, STQ Max, STQ Min, STQ SD, maximum diameter of the nodule, multiplicity, margin, halo sign, composition, echogenicity, echo texture, orientation, calcification, relationship to capsule and CDFI of the nodule. Variables with a p-value < 0.05 were considered risk factors. Subsequently, a multi-factor logistic regression analysis was performed, and independent risk factors were identified through stepwise forward logistic regression to construct a multimodal ultrasound model (Multi-model).
2.5.2 Construction of the ultrasound radiomics model
An ultrasound physician with 5 years of diagnostic experience (Husha Li) selected representative thyroid nodule images from each patient who met the inclusion criteria, utilizing the RadiAnt DICOM Viewer 2021.1 software (Medixant, Poznan, Poland), and saved the selected images in DICOM format (Figure 2A). Another ultrasound physician with 5 years of experience in ultrasound diagnosis, Hailong Wang, manually depicted the ROI images of the selected patients using the polygon mode in ITK-SNAP software (www.itksnap.org) without understanding the pathological results (Figures 2B, C). The DICOM images and segmented ROIs were subsequently imported into the radiomics software PyRadiomics for feature extraction, resulting in radiomics feature. The extracted radiomics features were normalized to conform to a N ~ (0, 1) distribution (Figure 2D). The Spearman correlation coefficient was employed to assess the correlation between features. For features exhibiting a correlation coefficient greater than 0.9, only one of the correlated features was retained, yielding radiomics feature selection. LASSO regression was applied for cross-validation and to determine the optimal penalty coefficient lambda. Features with a zero coefficient were excluded, and further dimensionality reduction was conducted to derive final radiomics features (Figures 2E, F). The features with non-zero coefficients were aggregated into a formula to compute the final radiomics score (see Supplementary Figure S1). The Random Forest (RF) algorithm was used to build the ultrasound radiomics model (Rad-model).
Figure 2. Flowchart of Ultrasound Radiomics Model Construction. CV stands for cross-validation, and MSE stands for mean square error. (A) Selection of representative thyroid nodule images. (B) Manual delineation of the ROI. (C) Example of a segmented ROI after manual delineation. (D) Normalization of the extracted radiomics features to conform to a standard normal distribution. (E) Application of LASSO regression for feature selection. (F) Final selection of radiomics features after dimensionality reduction.
2.5.3 Construction of the combined model
Based on the Multi-model and the Rad-model, a combined model (Com-model) was established by means of Logistic regression analysis and a nomogram was subsequently plotted.
2.6 Verification and clinical application of the nomogram
Receiver Operating Characteristic (ROC) curves were generated, and the AUC was calculated to assess the discriminatory performance of the model. Calibration curves were plotted, and the Hosmer-Lemeshow goodness-of-fit test was utilized to evaluate the calibration ability of the nomogram. The Brier score was computed to assess the overall performance of the model. Decision Curve Analysis (DCA) was constructed, and the clinical utility of the nomogram was estimated by calculating the net benefit across a range of threshold probabilities.
2.7 Rad-model assisted diagnosis
Fifty thyroid nodule images were randomly selected from enrolled patients and evaluated by 10 junior physicians (ultrasound diagnosis experience ≤3 years) and 10 senior physicians (ultrasound diagnosis experience >10 years) without knowledge of patient information. The 50 cases of thyroid nodules were assessed for benign and malignant conditions, as well as pathological results, and were re-evaluated one week later with the assistance of Rad-model prediction probability. The AUC was calculated to assess the diagnostic performance of both senior and junior physicians before and after the incorporation of ultrasound Rad-model assistance.
2.8 Statistical analyses
All statistical tests were conducted using SPSS 27.0, MedCalc (version 20.100), and R statistical software (version 4.0.2). We convert some continuous variables into categorical variables based on the ROC curve. Independent sample t-tests, chi-square tests, or Mann-Whitney U tests were employed to compare differences in age, nodule size, and multimodal ultrasound imaging characteristics between the training set and the test set. Logistic regression was utilized to construct Multi-model and Com-model, while the RF was adopted to establish Rad-model. ROC curves for the three models were generated using MedCalc. ROC analysis was performed, employing the Youden index to identify the optimal cut-off value for calculating the AUC, sensitivity, positive predictive value (PPV), negative predictive value (NPV) and accuracy. Internal validation of the models was performed using Bootstrap with 1000 resamplings. DeLong’s test was performed to compare the diagnostic performance of the models. A difference was considered statistically significant when P < 0.05. We utilized the “rms” and “pec” packages to build nomograms and calibration curves, the “caret” package for bootstrap validation, and the “ramda” and “ggDCA” packages to draw clinical decision curves for analysis using R statistical software.
3 Results
3.1 Characteristics of patients
This study included a total of 446 patients out of 490 patients with thyroid nodules. The set comprised 98 males and 348 females, with an average age of 50.09 ± 12.30 years. Based on the order of enrollment, the patients were divided into a training set (n=312) and a test set (n=134) in a ratio of 7:3. Within the training set, there were 172 benign cases and 140 malignant cases, while the test set contained 79 benign cases and 55 malignant cases. The maximum diameter of nodules in the training and test set was 10.10 ± 7.60 mm and 10.72 ± 7.30 mm. The baseline characteristics of the patients are presented in Table 1. There was no statistically significant difference between the two groups (P > 0.05).
Table 1. Baseline characteristics of patients in the training set and test set.
3.2 Construction and performance of the model
3.2.1 The Multi-model
Univariate logistic regression analyses showed that age (≤51 years) (OR: 2.863, 95% CI: 1.796-4.563), gender (female) (OR: 0.496, 95% CI: 0.283-0.869), STE Mean (OR: 1.046, 95% CI: 1.024-1.069), STE Max (OR: 1.012, 95% CI: 1.004-1.021), STE Min (OR: 1.025, 95% CI: 1.001-1.050), STE SD (OR: 1.064, 95% CI: 1.015-1.115), SE Mean (OR: 0.058, 95% CI: 0.006-0.596), STQ Mean (OR: 1.026, 95% CI: 1.011-1.041), STQ Max (OR: 1.011, 95% CI: 1.003-1.019), STQ SD (OR: 1.064, 95% CI: 1.026-1.103), maximum diameter (≤7.4 mm) (OR: 1.851, 95% CI: 1.177-2.903), margin (ill-defined) (OR: 9.704, 95% CI: 4.584-20.593), margin (irregular) (OR: 13.382, 95% CI: 7.254-24.684), halo sign (present) (OR: 0.224, 95% CI: 0.075-0.670), composition (solid) (OR: 9.792, 95% CI: 2.236-42.882), echogenicity (markedly hypoechoic) (OR: 3.219, 95% CI: 1.861-5.569), echogenicity (hypoechoic) (OR: 4.809, 95% CI: 2.261-10.229), orientation (vertical) (OR: 7.875, 95% CI: 4.697-13.203), calcification (coarse calcification) (OR: 0.218, 95% CI: 0.085-0.558), calcification (microcalcification) (OR: 0.204, 95% CI: 0.119-0.350) is a risk factor for PTC.
Multivariate logistic regression analysis showed that age (≤51 years) (OR: 2.752, 95% CI: 1.546-4.900), STE Mean (OR: 1.036, 95% CI: 1.010-1.062), margin (ill-defined) (OR: 6.187, 95% CI: 2.700-14.178), and margin (irregular) (OR: 7.011, 95% CI: 3.545-13.865), orientation (vertical) (OR: 3.515, 95% CI: 1.926-6.415) were independent risk factors for PTC (Table 2). The AUC of the Multi-model in the training set was 0.852 (95% CI: 0.808-0.890), sensitivity was 82.14%, specificity was 79.65%, PPV was 76.67%, and NPV was 84.57%. In the test set, the AUC of the Multi-model was 0.804 (95% CI: 0.726-0.867), sensitivity was 72.73%, specificity was 79.75%, PPV was 71.43%, and NPV was 80.77% (Figure 3 and Table 3).
Table 2. Results based on univariate and multivariate logistic regression analysis of the training set.
Figure 3. ROC Curves of the Com-Model in the Training (A) and Test Set (B).
Table 3. Predictive performance of the three models in the training set and test set.
3.2.2 The Rad-model
A total of 1,562 features related to radiomics were extracted initially. After applying dimensionality reduction, 17 features were selected (Figure 4), and the Rad-model was developed using the RF algorithm. The AUC of the Rad-model in the training set was 0.940 (95% CI: 0.907-0.963), sensitivity was 88.57%, specificity was 84.30%, PPV was 82.12%, and NPV was 90.06%. In the test set, the AUC of the Rad-model was 0.832 (95% CI: 0.758-0.891), sensitivity was 72.73%, specificity was 86.08%, PPV was 78.43%, and NPV was 81.93% (Figure 3 and Table 3).
Figure 4. Coefficients of the 17 selected features (see Supplementary Figure S2).
3.2.3 The Com-model
The Com-model was developed by integrating the Multi-model and the Rad-model. The AUC of the Com-model in the training set was 0.956 (95% CI: 0.926-0.976), sensitivity was 90.00%, specificity was 97.79%, PPV was 96.92%, and NPV was 92.31%. Internal validation was performed using 1000 bootstrap resampling iterations, resulting in an average AUC of 0.954 for the Com-model. In the test set, the AUC of the Com-model was 0.863 (95% CI: 0.793-0.917), sensitivity was 70.91%, specificity was 89.87%, PPV was 82.98%, and NPV was 81.61% (Figure 3 and Table 3).
3.3 Verification and clinical application of the nomogram
A nomogram was created to estimate the probability of PTC. Using the nomogram derived total score, we stratified patients into low (≤54, malignancy probability ≤15%), intermediate (54-68, malignancy probability15-50%), and high-risk (>68, malignancy probability >50%) categories. For high-risk nodules, immediate FNA or surgery is recommended. Intermediate-risk cases may benefit from selective FNA guided by clinical factors, while low-risk nodules warrant surveillance, avoiding unnecessary biopsies. (Figure 5). The calibration plot in Figure 6 illustrates the comparison between the predicted positive rate derived from the nomogram and the actual observations. According to the Hosmer-Lemeshow goodness-of-fit test, both the training set (P = 0.913) and the test set (P = 0.854) show a strong fit. The Brier scores for the training and test sets are 0.08 and 0.15, respectively, indicating that our prediction model demonstrates overall robustness. The DCA shows that the model offers considerable advantages within the range of 0.15 to 0.90 (Figure 7). To further assess the effectiveness of the models, we performed statistical comparisons of the ROC curves using DeLong’s test (Table 4). The findings indicate that in the training set, the Com-model was the most effective. In the test set, the Com-model surpassed the Multi-model, with no significant difference noted when compared to the ultrasound Rad-model.
Figure 5. Nomogram of the Com-Model.
Figure 6. Calibration Curves of the Com-Model in the Training Set (A) and Test Set (B).
Figure 7. Clinical Decision Curves of the Com-Model in the Training Set (A) and Test Set (B).
Table 4. Delong test of three models.
3.4 Effectiveness of the Rad-model assistance
Before utilizing the Rad-model for diagnostic assistance, the AUC for junior physicians was 0.748, while that for senior physicians was 0.837 (Table 5 and Figure 8). Following the implementation of the Rad-model, the average AUC for junior physicians significantly increased to 0.851, and for senior physicians, it improved to 0.862. DeLong’s test indicated a significant enhancement in the diagnostic performance of junior physicians after model assistance (<0.001), suggesting that the Rad-model effectively shortened their learning curve. Furthermore, there was no significant difference in AUC between senior and junior physicians after receiving model assistance, indicating that the model successfully narrowed the diagnostic gap between physicians with varying levels of experience (Table 6).
Table 5. The performance of Pre- and Post-diagnostic Radiomic Model Assistance of junior and senior physicians.
Figure 8. The performance of Pre- (A) and Post-diagnostic (B) Radiomic Model Assistance of junior and senior physicians.
Table 6. The DeLong test for comparing Pre- and Post-diagnostic Radiomic Model Assistance of junior and senior physicians.
4 Discussion
This study integrated multimodal ultrasound imaging features with ultrasound radiomics analysis to develop and validate a Com-model for the preoperative differentiation of benign and malignant thyroid nodules classified as C-TIRADS 4A. Experimental data demonstrated that the model achieved an AUC of 0.956 in the training set and 0.863 in the test set, showcasing reliable diagnostic performance. This model enables rapid and accurate differentiation of benign and malignant C-TIRADS 4A thyroid nodules, thereby effectively reducing the risk of unnecessary biopsies for benign nodules.
This study confirmed that age is an independent risk factor for PTC. The results showed that the risk of malignancy in nodules was significantly higher in patients aged ≤51 years, a finding consistent with the research by Wang et al. (14) This phenomenon may be attributed to increased public health awareness and the widespread adoption of regular health checkups, enabling more cases to be identified at an early stage of the disease. Additionally, advancements in modern diagnostic techniques have significantly improved the detection rate of PTC. The study included only patients aged 18 and above, as the number of patients under 18 undergoing thyroid surgery in multicenter institutions was minimal (most of these patients had already received treatment in pediatric specialty hospitals). The exclusion of adolescent patients aimed to avoid analytical bias caused by potential biological differences between adults and adolescents, thereby ensuring the accuracy of the research data and the validity of the conclusions.
A higher ultrasound elasticity score indicates a greater likelihood of malignancy in nodules (15, 16), reflecting that malignant nodules typically exhibit higher stiffness compared to benign ones. Research by Luo et al. (17) demonstrated that the combination of SWE with American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) significantly improves the efficiency and accuracy of diagnosing thyroid nodule characteristics. Similarly, Zhang et al. (18) found that adding elastography to multimodal ultrasound for predicting the benign or malignant nature of C-TIRADS 4A nodules significantly enhances diagnostic consistency, sensitivity, and specificity, outperforming any single method. In this study, STE Mean is an important risk factor. This result further validates the effectiveness and importance of ultrasound elastography in assisting the differentiation of benign and malignant thyroid nodules. Specifically, by integrating ultrasound elastography with various other ultrasound features, the model developed in this study not only improves the accuracy of thyroid nodule characterization but also enhances the reliability of clinical decision-making, helping to reduce unnecessary invasive procedures.
Vertical orientation (where the longitudinal growth of the nodule exceeds its transverse growth) is considered an independent risk factor for malignant thyroid nodules (19, 20). From a histological perspective, this phenomenon may be attributed to the active division of tumor cells in the anterior-posterior direction within malignant nodules, while remaining relatively quiescent in other directions. This aggressive growth pattern aims to increase the tumor’s surface area, thereby facilitating more efficient nutrient acquisition and accelerating its growth process.
Additionally, blurred or irregular margins are one of the critical imaging features for assessing the nature of thyroid nodules (21). Studies have shown that the BRAF V600E mutation is the most prevalent genetic alteration in PTC, present in approximately 60–80% of cases. This mutation constitutively activates the MAPK/ERK signaling pathway, promoting uncontrolled cell proliferation and tumorigenesis and promotes the invasiveness and metastatic potential of tumor cells, thereby influencing the morphological appearance of the nodules. Specifically, it leads to blurred, jagged, or spiculated margins, which are often indicative of malignancy. The BRAF V600E mutation has been confirmed to be associated with the high aggressiveness and poor prognosis of PTC (21, 22).
The Com-model achieved an AUC of 0.863 in the test set, outperforming standalone C-TIRADS 4A classification [reported AUC: 0.70-0.82 in prior studies (5, 6)] and ACR TI-RADS [AUC: 0.76-0.85 (5, 18)]. For instance, Zhang et al. (18) reported an AUC of 0.834 for ACR TI-RADS in differentiating C-TIRADS 4A nodules, while our Com-model achieved higher specificity (89.87% vs. 76.5%) and comparable sensitivity (70.91% vs. 72.1%). Notably, the integration of radiomics and multimodal ultrasound features enabled our model to capture subtle malignant characteristics (e.g., microstructural heterogeneity) that conventional systems may overlook. Compared to AI-driven approaches, such as TNet [AUC: 0.865 (23)] and the CNN-based framework by Tao et al. [AUC: 0.872 (16)], our Com-model demonstrated superior generalizability across multicenter data. However, unlike deep learning models requiring large annotated datasets, our radiomics framework relies on interpretable handcrafted features, aligning better with clinical workflows.
This study compared the diagnostic performance of senior and junior physicians in differentiating thyroid nodules before and after the application of the Rad-model. Although junior physicians lack extensive experience in ultrasound diagnosis, their diagnostic accuracy significantly improved with the support of the Rad-model, reaching a level comparable to that of senior physicians. This not only greatly shortened the learning curve for junior physicians, accelerating their path to professional proficiency, but also enhanced the overall efficiency and accuracy of thyroid nodule diagnosis. These findings suggest that the Rad-model can effectively address the challenges posed by limited clinical experience, providing robust technical support for younger physicians and thereby improving the overall quality of healthcare services.
The Com-model exhibited a false-negative rate of 10.00% in the training set and 29.09% in the test set, indicating a non-negligible risk of missing malignant cases, particularly in test set. While this rate aligns with prior studies [e.g., Zhang et al. (24)]. To mitigate the risks of false-negative results, we advocate for a tiered follow-up protocol combining short-interval ultrasound surveillance and clinical risk stratification: 6 months follow-up for intermediate-risk cases and 12 months reassessment for low-risk cases. Future iterations of the model will integrate dynamic imaging biomarkers and molecular testing to further reduce FNR, ensuring early intervention for initially missed malignancies.
Future research will focus on integrating molecular and biochemical profiling (25) with our imaging-based model to achieve a holistic assessment of thyroid nodules. For instance, combining radiomic heterogeneity with BRAF V600E or TERT promoter mutations could improve the identification of aggressive PTC subtypes, while metabolic markers of oxidative stress may refine angioinvasion risk prediction (26). Such multi-omics fusion aligns with the goals of precision oncology, enabling tailored surveillance and treatment strategies for borderline or ambiguous nodules. Additionally, liquid biopsy-derived biomarkers (e.g., ctDNA) (27) could complement ultrasound surveillance by providing real-time molecular insights into nodule dynamics. This approach may bridge the gap between static imaging assessments and the evolving biological behavior of thyroid malignancies.
To ensure consistency across multiple centers, all participating centers use the same ultrasound system and follow standardized imaging protocols. In the future, we will collaborate with medical institutions using different ultrasound systems (such as GE Logiq, Philips EPIQ, Siemens Acuson) to collect external datasets to ensure their applicability in the real world.
This study has several notable limitations. First, while it was impossible to completely eliminate all subjective factors during the analysis of multimodal ultrasound features, we addressed this by involving multiple evaluators and applying consistency tests to correct for potential subjective biases. Second, some cases relied on cytopathological reports, which inherently carry a certain rate of false negatives. To mitigate this issue, we incorporated BRAF gene testing results and conducted a 6 months follow-up observation of patients to more accurately assess the nature of the nodules.
5 Conclusion
In summary, we successfully developed a Com-model that integrates the Multi-model with the Rad-model to differentiate between benign and malignant thyroid nodules. The results demonstrate that this Com-model exhibits significant advantages in distinguishing C-TIRADS 4A thyroid nodules, providing physicians with a rapid and accurate risk assessment tool. It effectively identifies potential malignant lesions while reducing unnecessary invasive examinations or treatments for patients with benign nodules.
Data availability statement
The original contributions presented in the study are included in the article/Supplementary Material. Further inquiries can be directed to the corresponding author.
Ethics statement
The studies involving humans were approved by The Ethics Committee of the First Affiliated Hospital of Henan University of Science and Technology 2024-03-K0160. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
Author contributions
SC: Writing – original draft, Writing – review & editing. QL: Writing – original draft. HW: Data curation, Writing – original draft. HL: Investigation, Writing – original draft. WL: Data curation, Writing – original draft. CL: Data curation, Writing – original draft. LB: Data curation, Writing – original draft. YM: Data curation, Writing – original draft. WG: Software, Writing – original draft. JY: Conceptualization, Project administration, Supervision, Writing – review & editing. ZZ: Formal Analysis, Data curation, Writing – original draft, Writing – review & editing.
Funding
The author(s) declare that financial support was received for the research and/or publication of this article. This study was supported by the Key Project of Luoyang Science and Technology Development Plan (Grant No. 2401209B).
Acknowledgments
We would like to acknowledge all participants in this study for their efforts.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declare that no Generative AI was used in the creation of this manuscript.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fonc.2025.1543020/full#supplementary-material
Abbreviations
AUC, Areas under the curve; C-TIRADS, Chinese-Thyroid Imaging Report and Data System; PTC, Papillary thyroid cancer; DCA, Decision curve analysis; RF, Random Forest; ROC, Receiver Operating Characteristic; ROI, Region of interest; SD, Standard deviation; STE, Sound Touch Elastography; STQ, Sound Touch Quantify; SE, Strain Elastography; CV, Cross-validation; MSE, Mean square error; FNR, False Negative Rate; PPV, Positive Predictive Value; NPV, Negative Predictive Value.
References
1. Bray F, Laversanne M, Sung H, Ferlay J, Siegel RL, Soerjomataram I, et al. Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. (2024) 74:229–63. doi: 10.3322/caac.21834
2. Seib CD, Sosa JA. Evolving understanding of the epidemiology of thyroid cancer. Endocrinol Metab Clin North Am. (2019) 48:23–35. doi: 10.1016/j.ecl.2018.10.002
3. Russ G, Bonnema SJ, Erdogan MF, Durante C, Ngu R, Leenhardt L. European thyroid association guidelines for ultrasound Malignancy risk stratification of thyroid nodules in adults: the EU-TIRADS. Eur Thyroid J. (2017) 6:225–37. doi: 10.1159/000478927
4. Zhou J, Yin L, Wei X, Zhang S, Song Y, Luo B, et al. 2020 Chinese guidelines for ultrasound Malignancy risk stratification of thyroid nodules: the C-TIRADS. Endocrine. (2020) 70:256–79. doi: 10.1007/s12020-020-02441-y
5. Tessler FN, Middleton WD, Grant EG, Hoang JK, Berland LL, Teefey SA, et al. ACR thyroid imaging, reporting and data system (TI-RADS): white paper of the ACR TI-RADS committee. J Am Coll Radiol. (2017) 14:587–95. doi: 10.1016/j.jacr.2017.01.046
6. Haugen BR, Alexander EK, Bible KC, Doherty GM, Mandel SJ, Nikiforov YE, et al. 2015 American thyroid association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American thyroid association guidelines task force on thyroid nodules and differentiated thyroid cancer. Thyroid. (2016) 26:1–133. doi: 10.1089/thy.2015.0020
7. Gong HY, Fu XF, He MX, Zhu J. Combination of two-dimensional ultrasound and contrast-enhanced ultrasound in the diagnosis of benign and Malignant thyroid micro-nodules. Asian J Surg. (2022) 45:1172–3. doi: 10.1016/j.asjsur.2022.01.091
8. Nattabi HA, Sharif NM, Yahya N, Ahmad R, Mohamad M, Zaki FM, et al. Is diagnostic performance of quantitative 2D-shear wave elastography optimal for clinical classification of benign and Malignant thyroid nodules?: A systematic review and meta-analysis. Acad Radiol. (2022) Suppl 3:S114–21. doi: 10.1016/j.acra.2017.09.002
9. Zuo D, Yang L, Jin Y, Qi H, Liu Y, Ren L. Machine learning-based models for the prediction of breast cancer recurrence risk. BMC Med Inform Decis Mak. (2023) 23:276. doi: 10.1186/s12911-023-02377-z
10. Din NMU, Dar RA, Rasool M, Assad A. Breast cancer detection using deep learning: Datasets, methods, and challenges ahead. Comput Biol Med. (2022) 149:106073. doi: 10.1016/j.compbiomed.2022.106073
11. Gao Q, Yang L, Lu M, Jin R, Ye H, Ma T. The artificial intelligence and machine learning in lung cancer immunotherapy. J Hematol Oncol. (2023) 16:55. doi: 10.1186/s13045-023-01456-y
12. Huang S, Yang J, Shen N, Xu Q, Zhao Q. Artificial intelligence in lung cancer diagnosis and prognosis: Current application and future perspective. Semin Cancer Biol. (2023) 89:30–7. doi: 10.1016/j.semcancer.2023.01.006
13. Wen SS, Wu YJ, Wang JY, Ni ZX, Dong S, Xie XJ, et al. BRAFV600E/p-ERK/p-DRP1(Ser616) promotes tumor progression and reprogramming of glucose metabolism in papillary thyroid cancer. Thyroid. (2024) 34:1246–59. doi: 10.1089/thy.2023.0700
14. Wang L, Wang C, Deng X, Li Y, Zhou W, Huang Y, et al. Multimodal ultrasound radiomic technology for diagnosing benign and Malignant thyroid nodules of Ti-Rads 4-5: A multicenter study. Sensors (Basel). (2024) 24:6203. doi: 10.3390/s24196203
15. Liu F, Liu D, Wang K, Xie X, Su L, Kuang M, et al. Deep learning radiomics based on contrast-enhanced ultrasound might optimize curative treatments for very-early or early-stage hepatocellular carcinoma patients. Liver Cancer. (2020) 9:397–413. doi: 10.1159/000505694
16. Tao Y, Yu Y, Wu T, Xu X, Dai Q, Kong H, et al. Deep learning for the diagnosis of suspicious thyroid nodules based on multimodal ultrasound images. Front Oncol. (2022) 12:1012724. doi: 10.3389/fonc.2022.1012724
17. Luo J, Chen J, Sun Y, Xu F, Wu L, Huang P. A retrospective study of reducing unnecessary thyroid biopsy for American College of Radiology Thyroid Imaging Reporting and Data Systems 4 assessment through applying shear wave elastography. Arch Endocrinol Metab. (2020) 64:349–55. doi: 10.20945/2359-3997000000267
18. Zhang WB, Xu W, Fu WJ, He BL, Liu H, Deng WF. Comparison of ACR TI-RADS, Kwak TI-RADS, ATA guidelines and KTA/KSThR guidelines in combination with SWE in the diagnosis of thyroid nodules. Clin Hemorheol Microcirc. (2021) 78:163–74. doi: 10.3233/CH-201021
19. Peng S, Liu Y, Lv W, Liu L, Zhou Q, Yang H, et al. Deep learning-based artificial intelligence model to assist thyroid nodule diagnosis and management: a multicentre diagnostic study. Lancet Digit Health. (2021) 3:e250–9. doi: 10.1016/S2589-7500(21)00041-8
20. Xu C, Fang J, Li W, Sun C, Li Y, Lowe S, et al. Construction and validation of BRAF mutation diagnostic model based on ultrasound examination and clinical features of patients with thyroid nodules. Front Genet. (2022) 13:973272. doi: 10.3389/fgene.2022.973272
21. Chuanji Z, Zheng W, Shaolv L, Linghou M, Yixin L, Xinhui L, et al. Comparative study of radiomics, tumor morphology, and clinicopathological factors in predicting overall survival of patients with rectal cancer before surgery. Transl Oncol. (2022) 18:101352. doi: 10.1016/j.tranon.2022.101352
22. Trimboli P, Scappaticcio L, Treglia G, Guidobaldi L, Bongiovanni M, Giovanella L, et al. (V600E) mutation in thyroid nodules with fine-needle aspiration (FNA) read as suspicious for Malignancy (Bethesda V, Thy4, TIR4): a systematic review and meta-analysis. Endocr Pathol. (2020) 31:57–66. doi: 10.1007/s12022-019-09596-z
23. Zhu YC, AlZoubi A, Jassim S, Jiang Q, Zhang Y, Wang YB, et al. A generic deep learning framework to classify thyroid and breast lesions in ultrasound images. Ultrasonics. (2021) 110:106300. doi: 10.1016/j.ultras.2020.106300
24. Zhang B, Tian J, Pei S, Chen Y, He X, Dong Y, et al. Machine learning-assisted system for thyroid nodule diagnosis. Thyroid. (2019) 29:858–67. doi: 10.1089/thy.2018.0380
25. Antony V, Sun T, Dolezal D, Cai G. Comprehensive molecular profiling of metastatic pancreatic adenocarcinomas. Cancers (Basel). (2025) 17:335. doi: 10.3390/cancers17030335
26. Buczyńska A, Kościuszko M, Krętowski AJ, Popławska-Kita A. Exploring the clinical utility of angioinvasion markers in papillary thyroid cancer: a literature review. Front Endocrinol (Lausanne). (2023) 14:1261860. doi: 10.3389/fendo.2023.1261860
Keywords: C-TIRADS 4A, multimodal ultrasound, ultrasound radiomics, papillary thyroid cancer, benign or malignant
Citation: Cui S, Liu Q, Wang H, Li H, Li W, Li C, Bi L, Mu Y, Guo W, Yao J and Zhang Z (2025) The value of a combined model based on ultra-radiomics and multi-modal ultrasound in the benign-malignant differentiation of C-TIRADS 4A thyroid nodules: a prospective multicenter study. Front. Oncol. 15:1543020. doi: 10.3389/fonc.2025.1543020
Received: 22 January 2025; Accepted: 14 April 2025;
Published: 08 May 2025.
Edited by:
Xin-Wu Cui, Huazhong University of Science and Technology, ChinaReviewed by:
Angelika Buczyńska, Medical University of Bialystok, PolandZhen Wang, Shandong First Medical University, China
Yifeng Zhang, Tongji University, China
Lianzhong Zhang, Henan Provincial People’s Hospital, China
Copyright © 2025 Cui, Liu, Wang, Li, Li, Li, Bi, Mu, Guo, Yao and Zhang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Jundong Yao, eWpkMjMxM0BvdXRsb29rLmNvbQ==
†These authors have contributed equally to this work