Development and Validation of a Nomogram for Predicting Survival in Male Patients With Breast Cancer

Abstract

Male breast cancer (MBC) is rare, and most patients are diagnosed at an advanced stage. We aimed to develop a reliable nomogram to predict breast cancer-specific survival (BCSS) for MBC patients, thus helping clinical diagnosis and treatment. Based on data from the Surveillance, Epidemiology, and End Results (SEER) database, 2,451 patients diagnosed with MBC from 2010 to 2015 were selected for this study. They were randomly assigned to either a training cohort (n = 1715) or a validation cohort (n = 736). The Multivariate Cox proportional hazards regression analysis was used to determine the independent prognostic factors, which were then utilized to build a nomogram for predicting 3- and 5-year BCSS. The discrimination and calibration of the new model was evaluated using the Concordance index (C-index) and calibration curves, while its accuracy and benefits were assessed by comparing it to the traditional AJCC staging system using the net reclassification improvement (NRI), the integrated discrimination improvement (IDI), and the decision curve analysis (DCA). Multivariate models revealed that age, AJCC stage, ER status, PR status, and surgery all showed a significant association with BCSS. A nomogram based on these variables was constructed to predict survival in MBC patients. Compared to the AJCC stage, the C-index (training group: 0.840 vs. 0.775, validation group: 0.818 vs. 0.768), the areas under the receiver operating characteristic curve of the training set (3-year AUC: 0.852 vs. 0.778, 5-year AUC: 0.841 vs. 0.774) and the validation set (3-year AUC: 0.778 vs. 0.752, 5-year AUC: 0.852 vs. 0.794), and the calibration plots of this model all exhibited better performance. Additionally, the NRI and IDI confirmed that the nomogram was a great prognosis tool. Finally, the 3- and 5-year DCA curves yielded larger net benefits than the traditional AJCC stage. In conclusion, we have successfully established an effective nomogram to predict BCSS in MBC patients, which can assist clinicians in determining the appropriate therapy strategies for individual male patients.

Introduction

Breast cancer in men is an uncommon disease, accounting for approximately 1% of all breast cancers and <1% of all cancers in men (1, 2). The lifetime risk of male breast cancer (MBC) is nearly 1:1000, compared to 1:8 for a woman (2). Although male patients account for only a small proportion of breast cancer patients, the incidence of MBC continues to increase by 1.1% annually, and the reported mortality rates are comparable to those in women (3, 4). Increasing evidence indicates that MBC may be different from female breast cancer (FBC), with unique molecular subtypes (5, 6). Therefore, MBC should be considered as a separate disease and should not be treated according to the guidelines for FBC.

The risk factors for MBC are different from those for FBC (7). Furthermore, the gene polymorphism seen in basal-like cancer with BRCA1 mutation in FBC is not frequently presented in MBC (8). However, BRCA2 mutation is more frequent in MBC and is seen in 4–16% of the cases and may be associated with poor survival (4). MBC is often diagnosed at a more advanced stage than FBC, which could be attributed to the lack of symptoms at initial presentation as well as its unknown biology (9, 10). Previous studies have shown that (a) mean age at diagnosis of MBC is 5–10 years more than that for FBC, and (b) the lymph node status is a significant prognostic factor of overall survival in patients with MBC (11, 12). Additionally, studies have identified higher rates of estrogen receptor (ER) positivity in MBC compared to FBC (12).

Clinically, several differences exist between MBC and FBC, in terms of the biological characteristics, hormone levels, and risk factors (7, 13). Therefore, distinct therapeutic approaches and management strategies are required for MBC. Nomograms, which are reliable and convenient prognostic tools, have been widely used to predict specific outcomes in clinical oncology. They can quantitatively predict the prognosis in certain patients using known and vital prognostic factors, and illustrate the numerical probability of clinical outcomes (14, 15). The American Joint Committee for Cancer (AJCC) staging system is a tool commonly used by oncologists to predict disease progression and design therapeutic strategies (16, 17). However, given the various factors that influence the course of cancer, prognosis based on the AJCC staging alone is unreliable. This study aims to establish a comprehensive prognostic evaluation of MBC by building a nomogram to understand the risk factors and prognosis better. We also propose to compare the prognostic value of the nomogram with that of the AJCC staging system based on patient data available in the Surveillance, Epidemiology, and End Results (SEER) database.

Materials and Methods

Patient Selection and Data Processing

Patient data were screened from the SEER database (covering 18 registries) using the latest SEER^* Stat version 8.3.5 (https://seer.cancer.gov/). We initially excluded other cancer diagnosis and selected 2,983 male patients over 18 years of age who were diagnosed with breast cancer between 2010 and 2015. The following variables were evaluated: age, race, marital status, histology, grade, AJCC stage, metastatic sites, ER status, progesterone receptor (PR) status, human epidermal growth factor 2-neu (HER2) status, surgery, radiation, chemotherapy, follow-up time, cancer-specific death, and all-cause death. We excluded patients who were diagnosed at autopsy or by death certificate (n = 15), as well as those who did not have complete information on all the above variables (race unknown: n = 21, grade unknown: n = 259, stage unknown: n = 69, metastatic sites unknown: n = 29, ER unknown: n = 54, PR unknown: n = 15, HER2 unknown: n = 46, surgery unknown: n = 16, and radiation unknown: n = 8). Ultimately, we identified 2,451 eligible patients for our study. All data from the SEER database was free, and this study was approved by the Institutional Research Committee of the First Affiliated Hospital of Xi'an Jiaotong University.

Nomogram Development and Statistical Analyses

For nomogram construction and validation, we randomly divided all the patients into training (n = 1715) and validation (n = 736) cohorts in a ratio of 7:3 (18, 19). Multivariate Cox proportional hazards regression analysis was performed to identify variables (P < 0.05) that significantly affected breast cancer-specific survival (BCSS) and overall survival (OS) in the training group. Using these identified prognostic factors, we constructed a nomogram for predicting 3- and 5-year survival rates in MBC patients.

The nomogram was validated internally in the training cohort and externally in the validation cohort. To evaluate the discriminative ability of the nomogram, we used the concordance index (C-index) and the receiver operating characteristic curve (ROC) and assessed the area under the curve (AUC) (20, 21). The calibration curves were used to compare the association between the actual outcomes and the predicted probabilities (22). Both discrimination and calibration were evaluated using bootstrapping with 1,000 resamples. To compare the accuracy of the new model with that of the traditional AJCC staging model, the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) were determined (23). The clinical usefulness and benefits of the predictive model were estimated by decision curve analyses (DCA) (24).

All statistical analyses were performed using the SPSS 24.0 (SPSS Inc., Chicago, IL, USA) and the R software (version 3.4.3; http://www.r-project.org/). A P < 0.05 was considered to be statistically significant.

Results

Patient Characteristics

We evaluated a total of 2451 MBC patients from 2010 to 2015. The training and validation cohorts consisted of 1,715 and 736 cases, respectively, selected by the random split-sample method (split ratio: 7:3). In the training cohort, the majority of patients were over 65 years old (61.0%), white (80.7%), and married (65.7%). Moreover, ductal carcinoma was the most common histopathologic type of MBC, and grades II and III of tumor differentiation degree accounted for 52.7 and 35%, respectively of all the cases. In patients with metastatic MBC, although the incidence of bone metastases was the highest, they were seen in only 4.3% of the cases. A large proportion of the patients were positive for ER (97.1%), positive for PR (90.7%), and negative for HER2 (84.8%). Besides, most of the MBC patients had undergone surgery, and the median follow-up time for both the sets were both 27 months. Table 1 presents the detailed information for the validation and training cohorts, which were comparable.

Screening for Prognostic Factors for BCSS

Based on the univariate and multivariate Cox proportional hazards regression analysis, we identified five independent prognostic factors in the training cohort. Age (≥65) at diagnosis (hazard ratio, HR = 2.722, P < 0.001), AJCC stage II (HR = 2.280, P = 0.023), AJCC stage III (HR = 6.090, P < 0.001), AJCC stage IV (HR = 21.310, P < 0.001), negative ER (HR = 2.956, P = 0.006), negative PR (HR = 1.825, P = 0.049), and no surgery (HR = 3.563, P < 0.001) were all significantly associated with BCSS in patients with MBC (Table 2). The related data for OS was in the Table S1.

Nomogram Construction

A nomogram based on the selected prognostic factors from the training cohort was developed for the prediction of BCSS at 3 and 5 years (Figure 1). The nomogram demonstrated that AJCC stage contributed the most to prognosis, followed by ER status, surgery, age, and PR status. Each level of every variable was assigned a score on the points scale. By adding the scores for each of the selected variables, a total score was obtained. The prediction corresponding to this total score then helped in estimating the 3- and 5-year BCSS for each individual patient. The related results for OS were shown in the Figure S1.

Figure 1

Validation and Calibration of the Nomogram

The C-indices based on the nomogram (training group = 0.840, validation group = 0.818) were higher than those based on the AJCC stage (training group = 0.775, verification group = 0.768). Furthermore, our model demonstrated better discriminative ability compared to the traditional AJCC stage in both the training (3-year AUC: 0.852 vs. 0.778, 5-year AUC: 0.841 vs. 0.774, Figure 2A) and validation (3-year AUC: 0.778 vs. 0.752, 5-year AUC: 0.852 vs. 0.794, Figure 3A) cohorts for 3- and 5-year BCSS. The calibration plots of the nomogram showed good agreement between the actual observations and the predicted outcomes both in the training (Figure 2B) and validation (Figure 3B) cohorts for 3- and 5-year BCSS. The related results for OS were shown in the Figures S2, S3.

Figure 2

Figure 3

Analysis of accuracy showed that the NRI for the 3- and 5-year follow-ups were 0.193 (95% CI: 0.008–0.477) and 0.044 (95% CI: −0.036–0.437) in the validation cohorts, respectively. Similarly, in the validation set, the IDI for 3-and 5-years were 0.078 (P < 0.001) and 0.065 (P < 0.001), respectively. These results indicate that our nomogram has a greater potential for accurately predicting prognosis compared to the AJCC stage model.

DCA was performed to compare the clinical usability and benefits of the nomogram with that of the traditional AJCC stage. As shown in Figure 4, compared to the AJCC stage model, the new nomogram's 3- and 5-year DCA curves showed larger net benefits across a range of death risk in the validation cohort.

Figure 4

Discussion

MBC belongs to a small fraction of breast carcinomas and is usually diagnosed at an advanced stage. It has been found to have a worse prognosis compared to FBC (5, 13, 25). Therefore, it is necessary to establish a model to predict the risk for MBC, to aid the development of therapeutic strategies for these patients. Although the AJCC staging system is good for determining the prognosis in MBC patients (17), it neglects some significant risk factors such as age, race, and marital status. In the present study, we constructed a more comprehensive model based on a combination of various risk factors to better predict prognosis in MBC patients. This nomogram based on five variables including age, AJCC stage, ER status, PR status, and surgery was capable of making more accurate assessments and predictions in MBC patients compared to the traditional AJCC staging system both in the training and validation cohorts.

In our present investigation, we found that the proportion of male patients with negative ER, negative PR and radiotherapy were lower than the reported proportion of breast cancer in female cases (26, 27), and MC patients had a shorter median follow-up time than FBC patients (28). Additionally, we identified five risk factors that have an impact on the male breast cancer-specific survival (MBCCS). A previous study by Giordano et al. has shown that increasing age, black race, family history of breast cancer and radiation exposure can be risk factors for breast cancer in men (2). Nahleh et al. have reported that age, clinical stage, and lymph node status were independent prognostic factors for survival in MBC (29). Most of the studies have demonstrated that the median age at diagnosis of MBC is 68 years, which is 5–10 years older than that of women when diagnosed with breast cancer (12). Moreover, black men were at a higher risk of MBC than other races (30, 31). Also, MBC patients who had undergone surgery and radiotherapy have a better prognosis (31). The findings of our analysis are consistent with these previous reports.

Considering the influence of the above-mentioned risk factors, the traditional AJCC stage system might not predict survival well in MBC. We, therefore, developed a nomogram for predicting MBCSS by combining all of the effective factors. Consistent with findings in women with breast cancer (32), our model found that AJCC stage and ER status have a significant impact on the total score used for predicting the outcomes. The results showed that the C-index and calibration curve were good in the validation cohort, indicating that the model was reproducible and reliable. In our nomogram, negativity for ER and PR had higher risk scores implying that MBC patients who are negative for hormone receptors (HRs) had a poor prognosis, which is consistent with findings in FBC (33–35). However, it is noteworthy that the hormone levels are different for male and female patients, obviously due to the difference in sex. To the best of our knowledge, this is the comprehensive and intensive large-population study to construct a nomogram for patients with MBC.

We evaluated the value of this novel nomogram for predicting BCSS by comparing it to the traditional AJCC stage. Compared to the AJCC stage, our nomogram had better discriminability and accuracy for predicting 3- and 5-year BCSS. Additionally, using DCA, it was adequately proven that the established nomogram predicted survival with better accuracy than the AJCC staging system. Analogously, several studies have utilized DCA to verify the benefits and clinical utility of the predictive power of models (36, 37). This is the first study to compare a newly established model with the traditional AJCC staging model and demonstrate its better predictive ability for MBC patients. We believe our model will directly help clinicians quantify the risk of cancer specific death and thereby design appropriate therapeutic strategies for individual male patients.

Our study has some limitations that should be acknowledged. First, this is a large-sample retrospective study based on the SEER database, which may have some inherent biases. Second, several potential important parameters and specific information related to prognoses, such as the family history of breast cancer, the surgical margin status, vascular invasion, radiotherapy, and chemotherapy were not available in the SEER database. Third, we excluded patients who had missing data on the collected variables, which could have resulted in a selection bias. Forth, a small number of categorical variables classified according to common criteria, such as lobular carcinoma, grade IV, and metastatic sites, may lower the reliability of the findings. Finally, our nomogram was internally validated. It is important to evaluate it by external validation using other populations with MBC.

In conclusion, to better determine the prognosis in MBC patients, we constructed and validated a nomogram to predict 3-and 5-year BCSS based on a large, population-based cohort. The proposed nomogram considered five independent risk factors namely age, AJCC stage, ER status, PR status and surgery. We have confirmed the excellent discrimination and clinical usability of this nomogram by comparing it to the AJCC staging system.

References

• 1.

  Abdelwahab YousefAJ. Male breast cancer: epidemiology and risk factors. Semin Oncol. (2017) 44:267–72. 10.1053/j.seminoncol.2017.11.002

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2.

  GiordanoSH. Breast cancer in men. N Engl J Med. (2018) 378:2311–20. 10.1056/NEJMra1707939

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3.

  JemalASiegelRWardEHaoYXuJThunMJ. Cancer statistics, 2009. CA Cancer J Clin. (2009) 59:225–49. 10.3322/caac.20006

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4.

  CallariMCappellettiVDe CeccoLMusellaVMiodiniPVeneroniSet al. Gene expression analysis reveals a different transcriptomic landscape in female and male breast cancer. Breast Cancer Res Treat. (2011) 127:601–10. 10.1007/s10549-010-1015-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5.

  AndersonWFJatoiITseJRosenbergPS. Male breast cancer: a population-based comparison with female breast cancer. J Clin Oncol. (2010) 28:232–9. 10.1200/JCO.2009.23.8162

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  WeirJZhaoYDHermanTAlganO. Clinicopathologic features and radiation therapy utilization in patients with male breast cancer: a national cancer database study. Breast Cancer. (2018) 12:1178223418770687. 10.1177/1178223418770687

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 7.

  Leon-FerreRAGiridharKVHiekenTJMutterRWCouchFJJimenezREet al. A contemporary review of male breast cancer: current evidence and unanswered questions. Cancer Metastasis Rev. (2018) 37:599–614. 10.1007/s10555-018-9761-x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  DebSLakhaniSROttiniLFoxSB. The cancer genetics and pathology of male breast cancer. Histopathology. (2016) 68:110–8. 10.1111/his.12862

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9.

  GiordanoSH. A review of the diagnosis and management of male breast cancer. Oncologist. (2005) 10:471–9. 10.1634/theoncologist.10-7-471

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10.

  NilssonCHolmqvistMBergkvistLHedenfalkILambeMFjallskogML. Similarities and differences in the characteristics and primary treatment of breast cancer in men and women - a population based study (Sweden). Acta Oncol. (2011) 50:1083–8. 10.3109/0284186X.2011.602114

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11.

  VaysseCSroussiJMallonPFeronJGRivainALNgoCet al. Prediction of axillary lymph node status in male breast carcinoma. Ann Oncol. (2013) 24:370–6. 10.1093/annonc/mds283

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12.

  FerzocoRMRuddyKJ. The epidemiology of male breast cancer. Curr Oncol Rep. (2016) 18:1. 10.1007/s11912-015-0487-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13.

  LautrupMDThorupSSJensenVBokmandSHaugaardKHoejrisIet al. Male breast cancer: a nation-wide population-based comparison with female breast cancer. Acta Oncol. (2018) 57:613–21. 10.1080/0284186X.2017.1418088

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14.

  IasonosASchragDRajGVPanageasKS. How to build and interpret a nomogram for cancer prognosis. J Clin Oncol. (2008) 26:1364–70. 10.1200/JCO.2007.12.9791

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15.

  XiongZDengGHuangXLiXXieXWangJet al. Score for the survival probability in metastasis breast cancer: a nomogram-based risk assessment model. Cancer Res Treat. (2018) 50:1260–9. 10.4143/crt.2017.443

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  LeeSBSohnGKimJChungIYLeeJWKimHJet al. A retrospective prognostic evaluation analysis using the 8th edition of the American Joint Committee on Cancer staging system for breast cancer. Breast Cancer Res Treat. (2018) 169:257–66. 10.1007/s10549-018-4682-5

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 17.

  PlichtaJKRenYThomasSMGreenupRAFayanjuOMRosenbergerLHet al. Implications for breast cancer restaging based on the 8th edition AJCC staging manual. Ann Surg. (2018). 10.1097/SLA.0000000000003071. [Epub ahead of print].

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18.

  CollinsGSReitsmaJBAltmanDGMoonsKG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. (2015) 350:g7594. 10.1136/bmj.g7594

  • CrossRef
  • Google Scholar

• 19.

  LeiZLiJWuDXiaYWangQSiAet al. Nomogram for preoperative estimation of microvascular invasion risk in hepatitis B virus-related hepatocellular carcinoma within the milan criteria. JAMA Surg. (2016) 151:356–63. 10.1001/jamasurg.2015.4257

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20.

  HanleyJAMcNeilBJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. (1982) 143:29–36. 10.1148/radiology.143.1.7063747

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  WolbersMKollerMTWittemanJCSteyerbergEW. Prognostic models with competing risks: methods and application to coronary risk prediction. Epidemiology. (2009) 20:555–61. 10.1097/EDE.0b013e3181a39056

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22.

  CoutantCOlivierCLambaudieEFondrinierEMarchalFGuilleminFet al. Comparison of models to predict nonsentinel lymph node status in breast cancer patients with metastatic sentinel lymph nodes: a prospective multicenter study. J Clin Oncol. (2009) 27:2800–8. 10.1200/JCO.2008.19.7418

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23.

  PencinaMJD'AgostinoRBSrD'AgostinoRBJrVasanRS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. (2008) 27:157–72; discussion 207–12. 10.1002/sim.2929

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24.

  VickersAJElkinEB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. (2006) 26:565–74. 10.1177/0272989X06295361

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 25.

  FentimanISFourquetAHortobagyiGN. Male breast cancer. Lancet. (2006) 367:595–604. 10.1016/S0140-6736(06)68226-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 26.

  ReddyKGStrasslePDMcGuireKP. Role of age, tumor grade, and radiation therapy on immediate postmastectomy breast reconstruction. Clin Breast Cancer. (2017) 18:313–9. 10.1016/j.clbc.2017.11.021

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27.

  YangLPSunHFZhaoYChenMTZhangNJinW. Clinicopathological characteristics and survival outcomes in pleomorphic lobular breast carcinoma of the breast: a SEER population-based study. Cancer Med. (2017) 6:2867–75. 10.1002/cam4.1244

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 28.

  ChenZYangJLiSLvMShenYWangBet al. Invasive lobular carcinoma of the breast: a special histological type compared with invasive ductal carcinoma. PLoS ONE. (2017) 12:e0182397. 10.1371/journal.pone.0182397

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 29.

  NahlehZASrikantiahRSafaMJaziehARMuhlemanAKomrokjiR. Male breast cancer in the veterans affairs population: a comparative analysis. Cancer. (2007) 109:1471–7. 10.1002/cncr.22589

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 30.

  AndersonWFAlthuisMDBrintonLADevesaSS. Is male breast cancer similar or different than female breast cancer?Breast Cancer Res Treat. (2004) 83:77–86. 10.1023/B:BREA.0000010701.08825.2d

  • CrossRef
  • Google Scholar

• 31.

  WangKWangQJXiongYFShiYYangWJZhangXet al. Survival comparisons between early male and female breast cancer patients. Sci Rep. (2018) 8:8900. 10.1038/s41598-018-26199-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 32.

  FuJWuLJiangMLiDJiangTHongZet al. Clinical nomogram for predicting survival outcomes in early mucinous breast cancer. PLoS ONE. (2016) 11:e0164921. 10.1371/journal.pone.0164921

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 33.

  GongYLiuYRJiPHuXShaoZM. Impact of molecular subtypes on metastatic breast cancer patients: a SEER population-based study. Sci Rep. (2017) 7:45411. 10.1038/srep45411

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 34.

  LeoneJPLeoneJZwengerAOIturbeJLeoneBAVallejoCT. Prognostic factors and survival according to tumour subtype in women presenting with breast cancer brain metastases at initial diagnosis. Eur J Cancer. (2017) 74:17–25. 10.1016/j.ejca.2016.12.015

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 35.

  MartinAMCagneyDNCatalanoPJWarrenLEBellonJRPungliaRSet al. Brain metastases in newly diagnosed breast cancer: a population-based study. JAMA Oncol. (2017) 3:1069–77. 10.1001/jamaoncol.2017.0001

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 36.

  LiangWYangPHuangRXuLWangJLiuWet al. A combined nomogram model to preoperatively predict histologic grade in pancreatic neuroendocrine tumors. Clin Cancer Res. (2018) 25:584–94. 10.1158/1078–0432.CCR-18-1305

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 37.

  Van CalsterBWynantsLVerbeekJFMVerbakelJYChristodoulouEVickersAJet al. Reporting and interpreting decision curve analysis: a guide for investigators. Eur Urol. (2018) 74:796–804. 10.1016/j.eururo.2018.08.038

  • Pubmed Abstract
  • CrossRef
  • Google Scholar