Development and Validation of a Simple-to-Use Nomogram to Predict Early Death in Metastatic Pancreatic Adenocarcinoma

Abstract

Background:

Pancreatic adenocarcinoma (PCa) is a highly aggressive malignancy with high risk of early death (survival time ≤3 months). The present study aimed to identify associated risk factors and develop a simple-to-use nomogram to predict early death in metastatic PCa patients.

Methods:

Patients diagnosed with metastatic PCa between 2010 and 2015 from the Surveillance, Epidemiology, and End Results (SEER) database were collected for model construction and internal validation. An independent data set was obtained from China for external validation. Independent risk variables contributed to early death were identified by logistic regression models, which were then used to construct a nomogram. Internal and external validation was performed to evaluate the nomogram using calibration curves and the receiver operating characteristic curves.

Results:

A total of 19,464 patients in the SEER cohort and 67 patients in the Chinese cohort were included. Patients from the SEER database were randomly divided into the training cohort (n = 13,040) and internal validation cohort (n = 6,424). Patients in the Chinese cohort were selected for the external validation cohort. Overall, 10,484 patients experienced early death in the SEER cohort and 35 in the Chinese cohort. A reliable nomogram was constructed on the basis of 11 significant risk factors. Internal validation and external validation of the nomogram showed high accuracy in predicting early death. Decision curve analysis demonstrated that this predictive nomogram had excellent and potential clinical applicability.

Conclusion:

The nomogram provided a simple-to-use tool to distinguish early death in patients with metastatic PCa, assisting clinicians in implementing individualized treatment regimens.

Introduction

All over the world, pancreatic adenocarcinoma (PCa) remains one of the most lethal cancers and was reported to be the fourth leading cause of cancer-related deaths in 2020 (1). On account of its insidious symptoms and high metastatic potential, more than 50% of patients with PCa are diagnosed at an advanced stage, which results in a dismal 5-year relative survival of 6% (2, 3).

Although most patients with metastatic pancreatic adenocarcinoma (mPCa) have no therapeutic options besides systemic chemotherapy for finite improvement of overall survival (OS), the objective response rates of first-line chemotherapy are less than 50% (4–6). Immunotherapy, which yields brilliant results in other cancers (7, 8), achieves unsatisfactory results in PCa, mostly because of its immunosuppressive tumor microenvironment (9). Thus, patients with mPCa have a poor prognosis and only 19.2% survive beyond 1 year of diagnosis (10), and approximately 50% of patients survive past 3 months according to data collected in the SEER database, a condition defined as early death. Thus, patients with mPCa are prone to early death because of the delayed early diagnosis and restricted outcomes of treatment. Exploration of factors contributing to early death is beneficial for the development of individualized strategies, which can help to improve patient survival and reduce disease burden. To date, there have been no in-depth studies investigating mortality within 3 months of diagnosis of mPCa. Thus, a simple-to-use and predictive model able to distinguish the potential risk factors contributing to early death would be valuable for patients with mPCa.

Tumor incidence and survival data for approximately 34.6% of the US cancer registry population have been recorded in the Surveillance, Epidemiology, and End Results (SEER) database (11). Studies based on a very large multicenter database provide more convincing evidence than single-center studies. Herein, after extracting baseline information, a set of patients with mPCa from the SEER database and an independent Chinese cohort were chosen to recognize factors related to early death and a simple-to-use nomogram was developed for predicting its incidence.

Methods

Ethics Approval and Consent to Participate

The authors obtained authorization to exact and analyze the research data stored in the SEER program from the National Cancer Institute, USA (reference number 10528-Nov2020). Informed patient consent was not required to access data through the SEER database. This study was conducted in strict accordance with the 1964 Helsinki Declaration and subsequent amendments or similar ethical standards. This retrospective study of the Chinese cohort was approved by the ethics committee of Zhongda Hospital, Medical School of Southeast University.

Patient Cohorts and Characteristics

Data including clinical information were extracted using SEER*Stat version 8.3.9. For this study, we selected patients with stage IV PCa in the SEER database registered from 2010 to 2015. The inclusion criteria were as follows (1): the inclusion site codes C25.0–C25.3 and C25.7–C25.9 and (2) histological codes 8140/3, 8225/3, 8480/3, 8481/3, 8500/3, and 8521/3 [according to the International Classification of Tumor Diseases Third Edition (ICD-O-3)]. The cases who were (1) with a history of previous malignancy (2), diagnosed by death certificate or autopsy, and (3) with T0 stage or incomplete information including cause of death, survival months, and race were excluded. Since surgery was not included in the standard treatment guidelines for mPCa, patients lacking specific information on T stage, N stage, or grade were retained in the cohort. Figure 1 shows the flow diagram of the patient selection criteria. Considering the malignancy of pancreatic cancer and definition of previous studies, early death was defined as death within 3 months after first diagnosis (12, 13).

Figure 1

Patients from the SEER database were randomized at a ratio of 2:1 and assigned to the training cohort and internal validation cohort, respectively, for the construction and verification of the nomogram. The following demographic and clinical characteristics were collected: age at diagnosis, race, sex, primary site, grade, T stage (American Joint Commission on Cancer [AJCC] 7th version), N stage (AJCC 7th version), liver metastasis, bone metastasis, brain metastasis, lung metastasis, surgery, radiotherapy, chemotherapy, cause of death, vital status, and survival months. For external validation, the same characteristics from another set of patients were collected from Zhongda Hospital, Medical School of Southeast University, between 2014 and 2019, which was selected by the same inclusion and exclusion criteria as SEER cohort.

Nomogram Development and Statistical Analysis

The basic characteristics of the included patients were described by number and percentage (n, %). Each variable’s contribution in predicting early death of mPCa in the training cohort was tested by univariate logistic analysis. Variables that were statistically significant were further analyzed by multivariate logistic regression. The odds ratios (OR) with corresponding 95% confidence interval (CI) was calculated. Risk factors which were statistically significant in the multivariate analysis were used to construct a predictive nomogram to predict the risk of early death. Nomogram performance was evaluated with respect to discrimination and calibration. For discrimination ability, the nomogram was evaluated using the area under the receiver operating characteristic (ROC) curve (AUC) (14). Calibration curves were plotted to verify the accuracy and reliability of the nomogram (15). Internal and external validations were performed to validate the nomogram. Moreover, decision curve analysis (DCA) was plotted to measure the applicability of the nomogram to clinical practice (16, 17). All statistical analyses were performed with SPSS v 20.0 (IBM Corp., Armonk, NY, USA) and R software v3.6.4 (https://www.r-project.org/). Statistically significance was considered for two-sided p-values <0.01. The R packages rms, pROC, and rmda were applied for data processing.

Results

Demographic and Clinical Characteristics

A total of 19,464 patients with mPCa were enrolled from the SEER database in this study, among which 10,484 patients experienced early death. A total of 10,133 patients experienced early death because of PCa. Thus, 13,040 patients were divided into the training cohort and 6,424 were divided into the internal validation cohort. The bulk of early death appeared in participants who were of male sex (54.8%), white (78.3%), and aged between 65 and 79 years (45.0%). The pancreatic head (31.2%) was the most common location associated with early death among patients. Except for the unknown grades (80.8%), the rate of early death in poorly/undifferentiated mPCa versus well/moderately mPCa was 11.5% versus 7.7%. The most common site for metastasis in patients with early death was the liver (78.4%). With regard to treatment, most patients with early death were not treated surgically (96.5%), and only a few patients received radiotherapy (3.5%) or chemotherapy (31.9%). As for the external validation cohort, 67 patients from our center were included in this study, with 35 patients experiencing early death. Similarly, most patients (51.4%) with early death were between 45 and 79 years, and 51.4% of them were male. Liver metastasis (94.3%) was the most common metastatic site. Table 1 shows the incidence of early death in patients with mPCa, and Table 2 summarizes the demographic and clinical characteristics of the mPCa patients in the training cohort, internal validation cohort, and external validation cohort.

Risk Factors for Early Death in the Training Cohort

To further analyze candidate risk factors for early death of mPCa, a binary logistic regression analysis for variables associated with early death is shown in Table 3. Univariate logistic analysis revealed that age, sex, primary tumor site, grade, T stage, N stage, liver metastasis, bone metastasis, brain metastasis, lung metastasis, surgery, radiotherapy, and chemotherapy were significantly related to early death. In addition, analysis using a multivariate logistic model identified 11 independent risk factors associated with the early death of mPCa, which included age, sex, primary tumor site, grade, liver metastasis, bone metastasis, brain metastasis, lung metastasis, surgery, radiotherapy, and chemotherapy.

Nomogram Construction

Based on the significant and independent risk factors identified by the multivariate analysis, a nomogram for predicting early death of mPCa was constructed (Figure 2). Chemotherapy contributed to early death mostly and was followed by surgery, brain metastasis, age, and radiotherapy according to the nomogram. In the tool, the total point score was obtained by summing the calculated points of each variable; thus, the probability of early death in mPCa was estimated simply. For example, a male diagnosed at age 70 years with pancreatic tail and liver metastasis, with poorly differentiated grade and who had received only chemotherapy, had a predicted probability of early death of approximately 48% using this nomogram.

Figure 2

Performance and Validation of the Nomogram

The calibration of the model was assessed using calibration curves. Figure 3A shows the high uniformity between the predicted and actual probability of early death in the training cohort, which was then validated using the internal and external validation cohorts (Figures 3B, C). The prediction curve was always accompanied by a curve indicating the actual probability. In addition, ROC analysis was used to evaluate predictive efficiencies for early death of the nomogram model. The ROC curves of early death in the training cohort (Figure 3D), the internal validation cohort (Figure 3E), and the external validation cohort (Figure 3F) presented excellent discrimination. Meanwhile, the AUC values of the nomogram were 0.802 (95% CI 0.795–0.810), 0.798 (95% CI 0.787–0.808), and 0.774 (95% CI 0.662–0.886), respectively. As a result of the discrimination test, this nomogram showed reliable predictive ability for early death events.

Figure 3

Clinical Utility

To evaluate the clinical applicability of the nomogram, DCA (Figure 4), an advanced method for analyzing the net clinical benefits of predictive models showed that the most favorable threshold probability for predicting early death in the training cohort with the nomogram was 0.2–0.8. As demonstrated by the favorable threshold probability, it indicated that the nomogram could assist clinicians to assess early death of mPCa patients accurately.

Figure 4

Discussion

PCa is a significant public health problem and is characterized by a high mortality rate with increasing incidence worldwide (18). Improvements in median survival based on standard treatment regimens have not been satisfactory compared to other cancers (4, 19–21). According to the SEER database, more than 50% of patients with mPCa experience early death, defined as survival ≤ 3 months. Most studies on the prognosis of PCa have focused on the long-term survival of patients (22, 23) and risk factors associated with early mortality in resectable pancreatic adenocarcinoma (24, 25). However, advanced tumors have a higher risk of early death, and the assessment of risk factors for early death in mPCa to establish individualized treatment regimens is necessary. Furthermore, a systematic review of phase III trial studies for advanced pancreatic cancer mentioned that early mortality in patients with advanced pancreatic cancer was 23.3%, possibly due to essentially underreported vascular thromboembolic events (26). If it were possible to predict patients at high risk of early death, this could be useful to provide information on the utility of targeted thromboprophylaxis and possibly improve the poor prognosis. For example, the best supportive therapy could be provided depending on the patient’s performance status. Moreover, distinguishing patients with high risk of early death is beneficial for developing clinical trials for advanced pancreatic cancer. Therefore, in this study, we established a predictive nomogram on the basis of independent risk factors to recognize whether patients with mPCa experienced early death.

Demographic information, including age, sex, and race, were identified to be intimately related to the prognosis of mPCa (27–29). In this study, the nomogram also evaluated the impact of these demographic characteristics on early death of mPCa. Among these, and consistent with previous studies, age and sex were significantly associated with early death of mPCa, but the contribution of race was not observed. Apart from demographic factors, early death of mPCa was mainly related to clinical factors such as histological staging, distant metastasis, and therapies including surgery, radiotherapy, and chemotherapy. Surgery has an important impact on the improvement of early death in mPCa, whereas some studies have emphasized the beneficial value of surgery in advanced, and especially, in oligometastatic pancreatic cancer (30, 31). However, considering the small number of patients that had received surgery in our study, it might be more prudent to set strict indications for surgery in mPCa based on the clinical conditions of the patient. Therefore, large and prospective trials are required to reveal the value of surgery for mPCa. There were some differences in case characteristics between the SEER and Chinese cohorts, which might be related to prevalence, economic status, religious beliefs, and eligibility for health insurance.

Many models for prognostic stratification based on epidemiological and clinicopathological features performed good clinical utility (23, 32). In our study, the nomogram was constructed based on the very large sample numbers from the SEER database, which suggested that results were reliable and stable. Through curve analysis, irrespective of the internal or external validation, the nomogram performed well in terms of discrimination and accuracy. In our study, the AUC values of the nomogram were all more than 0.7 in the training and validation cohorts. However, a large AUC and good agreement between predicted and observed results does not directly represent the clinical utility of the nomogram (33, 34). Thus, DCA, an advanced tool used to examine efficiencies of diagnostic tests and predictive models (35), was also performed in this study. The results demonstrated that the nomogram could perform well in terms of predictive efficiency and clinical application. Besides, histological sampling of the pancreatic primary tumor or liver metastases could provide a specific genetic signature to predict survival. Recent studies revealed that endoscopic ultrasound-guided fine-needle biopsy (EUS-FNB) had excellent diagnosis value in both solid pancreatic lesions and cystic pancreatic lesions (36–38). Even for mPCa, endoscopic ultrasound-guided fine-needle tissue acquisition (EUS-TA) could provide diagnosis and evaluation of liver metastases, which is the most common metastatic location of pancreatic cancer (39). EUS-TA not only played an important role in the diagnosis but also provided a new method for risk assessment and prognostic stratification of PCa (40). To be exact, EUS-TA was a cost-effective and efficient way to extract cells or tissues, especially for mPCa, which were used to perform predictive molecular marker and gene expression analyses. By combining the results of EUS-TA with epidemiological and clinicopathological features to create a more efficient model for risk assessment, it would benefit individualized treatment of mPCa.

Admittedly, there were several limitations to this study that cannot be ignored. First, some potential risk factors related to early death such as peritoneal metastases, unhealthy lifestyle habits (such as alcohol consumption and smoking), and history of past illness were lacking in the SEER database. Second, results might be influenced by selection bias from excluded and incomplete data. Third, although validation of the nomogram with an external cohort may help avoid overfitting of the model, the number of cases in the external validation cohort may have been insufficient. However, data from our center are well represented and fit the nomogram as external validation. Moreover, both internal and external validation cohorts confirmed the excellent applicability of this nomogram, indicating that the nomogram model is suitable for PCa patients of different races. For better external validation, large samples from international multicenter cohorts are desired.

In conclusion, a comprehensive and accurate nomogram based on risk factors associated with early mortality in mPCa was developed. The simple-to-use nomogram might provide oncologists with a suitable tool to devise individualized and precise treatment strategies, and could be further applied to improve survival outcomes for patients with mPCa.

Funding

This research was supported by a grant from Jiangsu Provincial Medical Youth Talent (QNRC2016809).

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

• 1

  SiegelRLMillerKDJemalA. Cancer Statistics, 2020. CA Cancer J Clin (2020) 70(1):7–30. doi: 10.3322/caac.21590

  • CrossRef
  • Google Scholar

• 2

  WangYHuGFZhangQQTangNGuoJLiuLYet al. Efficacy and Safety of Gemcitabine Plus Erlotinib for Locally Advanced or Metastatic Pancreatic Cancer: A Systematic Review and Meta-Analysis. Drug Des Devel Ther (2016) 10:1961–72. doi: 10.2147/DDDT.S105442

  • CrossRef
  • Google Scholar

• 3

  FramptonJE. Liposomal Irinotecan: A Review in Metastatic Pancreatic Adenocarcinoma. Drugs (2020) 80(10):1007–18. doi: 10.1007/s40265-020-01336-6

  • CrossRef
  • Google Scholar

• 4

  AnsariDTingstedtBAnderssonBHolmquistFSturessonCWilliamssonCet al. Pancreatic Cancer: Yesterday, Today and Tomorrow. Future Oncol (2016) 12(16):1929–46. doi: 10.2217/fon-2016-0010

  • CrossRef
  • Google Scholar

• 5

  Von HoffDDRamanathanRKBoradMJLaheruDASmithLSWoodTEet al. Gemcitabine Plus Nab-Paclitaxel Is an Active Regimen in Patients With Advanced Pancreatic Cancer: A Phase I/II Trial. J Clin Oncol (2011) 29(34):4548–54. doi: 10.1200/JCO.2011.36.5742

  • CrossRef
  • Google Scholar

• 6

  RamanathanRKMcDonoughSLPhilipPAHingoraniSRLacyJKortmanskyJSet al. Phase IB/II Randomized Study of FOLFIRINOX Plus Pegylated Recombinant Human Hyaluronidase Versus FOLFIRINOX Alone in Patients With Metastatic Pancreatic Adenocarcinoma: SWOG S1313. J Clin Oncol (2019) 37(13):1062–9. doi: 10.1200/JCO.18.01295

  • CrossRef
  • Google Scholar

• 7

  AdamsSSchmidPRugoHSWinerEPLoiratDAwadaAet al. Pembrolizumab Monotherapy for Previously Treated Metastatic Triple-Negative Breast Cancer: Cohort A of the Phase II KEYNOTE-086 Study. Ann Oncol (2019) 30(3):397–404. doi: 10.1093/annonc/mdy517

  • CrossRef
  • Google Scholar

• 8

  ReckMRodriguez-AbreuDRobinsonAGHuiRCsosziTFulopAet al. Pembrolizumab Versus Chemotherapy for PD-L1-Positive Non-Small-Cell Lung Cancer. N Engl J Med (2016) 375(19):1823–33. doi: 10.1056/NEJMoa1606774

  • CrossRef
  • Google Scholar

• 9

  O’ReillyEMOhDYDhaniNRenoufDJLeeMASunWet al. Durvalumab With or Without Tremelimumab for Patients With Metastatic Pancreatic Ductal Adenocarcinoma: A Phase 2 Randomized Clinical Trial. JAMA Oncol (2019) 5(10):1431–8. doi: 10.1001/jamaoncol.2019.1588

  • CrossRef
  • Google Scholar

• 10

  BilimoriaKYBentremDJKoCYRitcheyJStewartAKWinchesterDPet al. Validation of the 6th Edition AJCC Pancreatic Cancer Staging System: Report From the National Cancer Database. Cancer (2007) 110(4):738–44. doi: 10.1002/cncr.22852

  • CrossRef
  • Google Scholar

• 11

  SEER. Incidence Data, 1975-2018. Available at: https://seer.cancer.gov/data/ (Accessed 14 May 2021).

  • Google Scholar

• 12

  ZhuYFangXWangLZhangTYuD. A Predictive Nomogram for Early Death of Metastatic Gastric Cancer: A Retrospective Study in the SEER Database and China. J Cancer (2020) 11(18):5527–35. doi: 10.7150/jca.46563

  • CrossRef
  • Google Scholar

• 13

  SongZWangYZhouY. Zhang D. A Novel Predictive Tool for Determining the Risk of Early Death From Stage IV Endometrial Carcinoma: A Large Cohort Study. Front Oncol (2020) 10:620240. doi: 10.3389/fonc.2020.620240

  • CrossRef
  • Google Scholar

• 14

  JanssensAMartensFK. Reflection on Modern Methods: Revisiting the Area Under the ROC Curve. Int J Epidemiol (2020) 49(4):1397–403. doi: 10.1093/ije/dyz274

  • CrossRef
  • Google Scholar

• 15

  KramerAAZimmermanJE. Assessing the Calibration of Mortality Benchmarks in Critical Care: The Hosmer-Lemeshow Test Revisited. Crit Care Med (2007) 35(9):2052–6. doi: 10.1097/01.CCM.0000275267.64078.B0

  • CrossRef
  • Google Scholar

• 16

  Van CalsterBWynantsLVerbeekJFMVerbakelJYChristodoulouEVickersAJet al. Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators. Eur Urol (2018) 74(6):796–804. doi: 10.1016/j.eururo.2018.08.038

  • CrossRef
  • Google Scholar

• 17

  IasonosASchragDRajGVPanageasKS. How to Build and Interpret a Nomogram for Cancer Prognosis. J Clin Oncol (2008) 26(8):1364–70. doi: 10.1200/JCO.2007.12.9791

  • CrossRef
  • Google Scholar

• 18

  SaadAMTurkTAl-HusseiniMJAbdel-RahmanO. Trends in Pancreatic Adenocarcinoma Incidence and Mortality in the United States in the Last Four Decades; a SEER-Based Study. BMC Cancer (2018) 18(1):688. doi: 10.1186/s12885-018-4610-4

  • CrossRef
  • Google Scholar

• 19

  DucreuxMSeufferleinTVan LaethemJLLaurent-PuigPSmolenschiCMalkaDet al. Systemic Treatment of Pancreatic Cancer Revisited. Semin Oncol (2019) 46(1):28–38. doi: 10.1053/j.seminoncol.2018.12.003

  • CrossRef
  • Google Scholar

• 20

  BillerLHSchragD. Diagnosis and Treatment of Metastatic Colorectal Cancer: A Review. JAMA (2021) 325(7):669–85. doi: 10.1001/jama.2021.0106

  • CrossRef
  • Google Scholar

• 21

  SmythECNilssonMGrabschHIvan GriekenNCLordickF. Gastric Cancer. Lancet (2020) 396(10251):635–48. doi: 10.1016/S0140-6736(20)31288-5

  • CrossRef
  • Google Scholar

• 22

  SongWMiaoDLChenL. Nomogram for Predicting Survival in Patients With Pancreatic Cancer. Onco Targets Ther (2018) 11:539–45. doi: 10.2147/OTT.S154599

  • CrossRef
  • Google Scholar

• 23

  LiuJHWangSFLiBRZhuX. Development and Validation of an Individualized Nomogram for Predicting Pancreatic Adenocarcinoma-Specific Survival: A SEER Population Analysis of 5,805 Patients. Eur Rev Med Pharmacol Sci (2020) 24(20):10483–95. doi: 10.26355/eurrev_202010_23401

  • CrossRef
  • Google Scholar

• 24

  BalzanoGDugnaniECrippaSScaviniMPasqualeVAleottiFet al. A Preoperative Score to Predict Early Death After Pancreatic Cancer Resection. Dig Liver Dis (2017) 49(9):1050–6. doi: 10.1016/j.dld.2017.06.012

  • CrossRef
  • Google Scholar

• 25

  SohalDPShrotriyaSGlassKTPelleyRJMcNamaraMJEstfanBet al. Predicting Early Mortality in Resectable Pancreatic Adenocarcinoma: A Cohort Study. Cancer (2015) 121(11):1779–84. doi: 10.1002/cncr.29298

  • CrossRef
  • Google Scholar

• 26

  SgourosJMaraveyasA. Excess Premature (3-Month) Mortality in Advanced Pancreatic Cancer Could be Related to Fatal Vascular Thromboembolic Events. A Hypothesis Based on a Systematic Review of Phase III Chemotherapy Studies in Advanced Pancreatic Cancer. Acta Oncol (2008) 47(3):337–46. doi: 10.1080/02841860701687267

  • CrossRef
  • Google Scholar

• 27

  NippRTramontanoACKongCYPandharipandePDowlingECSchragDet al. Disparities in Cancer Outcomes Across Age, Sex, and Race/Ethnicity Among Patients With Pancreatic Cancer. Cancer Med (2018) 7(2):525–35. doi: 10.1002/cam4.1277

  • CrossRef
  • Google Scholar

• 28

  WangHLiuJXiaGLeiSHuangXHuangX. Survival of Pancreatic Cancer Patients Is Negatively Correlated With Age at Diagnosis: A Population-Based Retrospective Study. Sci Rep (2020) 10(1):7048. doi: 10.1038/s41598-020-64068-3

  • CrossRef
  • Google Scholar

• 29

  ShiMZhouBYangSP. Nomograms for Predicting Overall Survival and Cancer-Specific Survival in Young Patients With Pancreatic Cancer in the US Based on the SEER Database. PeerJ (2020) 8:e8958. doi: 10.7717/peerj.8958

  • CrossRef
  • Google Scholar

• 30

  WangLYangLChenLChenZ. Do Patients Diagnosed With Metastatic Pancreatic Cancer Benefit From Primary Tumor Surgery? A Propensity-Adjusted, Population-Based Surveillance, Epidemiology and End Results (SEER) Analysis. Med Sci Monit (2019) 25:8230–41. doi: 10.12659/MSM.917106

  • CrossRef
  • Google Scholar

• 31

  De SimoniOScarpaMTonelloMPilatiPTolinFSpolveratoYet al. Oligometastatic Pancreatic Cancer to the Liver in the Era of Neoadjuvant Chemotherapy: Which Role for Conversion Surgery? A Systematic Review and Meta-Analysis. Cancers (Basel) (2020) 12(11):3402. doi: 10.3390/cancers12113402

  • CrossRef
  • Google Scholar

• 32

  HuangLBalavarcaYvan der GeestLLemmensVVan EyckenLDe SchutterHet al. Development and Validation of a Prognostic Model to Predict the Prognosis of Patients Who Underwent Chemotherapy and Resection of Pancreatic Adenocarcinoma: A Large International Population-Based Cohort Study. BMC Med (2019) 17(1):66. doi: 10.1186/s12916-019-1304-y

  • CrossRef
  • Google Scholar

• 33

  VickersAJVan CalsterBSteyerbergEW. Net Benefit Approaches to the Evaluation of Prediction Models, Molecular Markers, and Diagnostic Tests. BMJ (2016) 352:i6. doi: 10.1136/bmj.i6

  • CrossRef
  • Google Scholar

• 34

  BalachandranVPGonenMSmithJJDeMatteoRP. Nomograms in Oncology: More Than Meets the Eye. Lancet Oncol (2015) 16(4):e173–80. doi: 10.1016/S1470-2045(14)71116-7

  • CrossRef
  • Google Scholar

• 35

  VickersAJCroninAMElkinEBGonenM. Extensions to Decision Curve Analysis, a Novel Method for Evaluating Diagnostic Tests, Prediction Models and Molecular Markers. BMC Med Inform Decis Mak (2008) 8:53. doi: 10.1186/1472-6947-8-53

  • CrossRef
  • Google Scholar

• 36

  CrinoSFDi MitriRNguyenNQTarantinoIde NucciGDeprezPHet al. Endoscopic Ultrasound-Guided Fine-Needle Biopsy With or Without Rapid On-Site Evaluation for Diagnosis of Solid Pancreatic Lesions: A Randomized Controlled Non-Inferiority Trial. Gastroenterology (2021) 161(3):899–909.e5. doi: 10.1053/j.gastro.2021.06.005

  • CrossRef
  • Google Scholar

• 37

  LarghiAManfrinEFabbriCCrinoSFCorrealeLChiarelloGet al. Interobserver Agreement Among Expert Pathologists on Through-the-Needle Microforceps Biopsy Samples for Evaluation of Pancreatic Cystic Lesions. Gastrointest Endosc (2019) 90(5):784–92.e4. doi: 10.1016/j.gie.2019.07.011

  • CrossRef
  • Google Scholar

• 38

  CrinoSFAmmendolaSMeneghettiABernardoniLConti BellocchiMCGabbrielliAet al. Comparison Between EUS-Guided Fine-Needle Aspiration Cytology and EUS-Guided Fine-Needle Biopsy Histology for the Evaluation of Pancreatic Neuroendocrine Tumors. Pancreatology (2021) 21(2):443–50. doi: 10.1016/j.pan.2020.12.015

  • CrossRef
  • Google Scholar

• 39

  FacciorussoARamaiDConti BellocchiMCBernardoniLManfrinEMuscatielloNet al. Diagnostic Yield of Endoscopic Ultrasound-Guided Liver Biopsy in Comparison to Percutaneous Liver Biopsy: A Two-Center Experience. Cancers (Basel) (2021) 13(12):3062. doi: 10.3390/cancers13123062

  • CrossRef
  • Google Scholar

• 40

  RimbasMCrinoSFGasbarriniACostamagnaGScarpaALarghiA. EUS-Guided Fine-Needle Tissue Acquisition for Solid Pancreatic Lesions: Finally Moving From Fine-Needle Aspiration to Fine-Needle Biopsy? Endosc Ultrasound (2018) 7(3):137–40. doi: 10.4103/eus.eus_23_18

  • CrossRef
  • Google Scholar