ORIGINAL RESEARCH article

Front. Surg., 03 November 2025

Sec. Visceral Surgery

Volume 12 - 2025 | https://doi.org/10.3389/fsurg.2025.1628956

Development and validation of a practical prediction model for post-ERCP pancreatitis using machine learning

  • 1. Department of Hepatobiliary Surgery, General Hospital of Ningxia Medical University, Yinchuan, China

  • 2. School of Clinical Medicine, General Hospital of Ningxia Medical University, Yinchuan, China

  • 3. Neonatal Intensive Care, General Hospital of Ningxia Medical University, Yinchuan, China

  • 4. Department of Thoracic and Cardiovascular Surgery, General Hospital of Ningxia Medical University, Yinchuan, China

  • 5. School of Basic Medical Sciences, Ningxia Medical University, Yinchuan, China

  • 6. Department of Pathogenic Biology and Medical Immunology, School of Basic Medical Sciences, Ningxia Medical University, Yinchuan, China

Abstract

Background:

Post-endoscopic retrograde cholangiopancreatography (ERCP) pancreatitis (PEP) is one most frequent and severe complication of ERCP. In consideration of recent advancements in both endoscopic and artificial intelligence research, it is possible to construct a practical risk prediction model to facilitate the identification of PEP patients at elevated risk.

Aim:

We developed and validated a concise predictive model for post-ERCP pancreatitis risk with logistic regression (LR), LightGBM, Support Vector Machine (SVM), XGBoost, and Multilayer Perceptron (MLP) neural network models.

Methods:

We selected 688 patients undergone ERCP to form the basic dataset, with 70% for training and 30% for validation. Subsequently, Stepwise Backward Selection Based on Logistic Regression was utilized to select pertinent clinical features, incorporating the machine learning (ML) models to construct the final predictive model. The efficacy of the model was evaluated by various metrics. These newly identified clinical features were then incorporated into a simplified, points-based risk scoring system for potential bedside application and further evaluation.

Results:

Based on the collected data and the results of stepwise backward regression, we identified the following features as potentially significant clinical variables that influence the risk of post-ERCP pancreatitis: periampullary diverticulum, pancreatic stent placement, pancreatic guidewire passages, dilation of the extrahepatic bile duct, age, and coronary artery disease, and constructed a prediction model. Following this, several ML models were constructed to assess the performance of this model. All ML models demonstrated superior performance to conventional logistic regression (LR) models in terms of AUC curves, with XGBoost, SVM, LightGBM, and MLP models all achieving at least acceptable performance levels. Finally, we developed a simplified scoring system based on LightGBM model with an AUC of 0.75.

Conclusions:

We developed and validated a concise predictive model for post-ERCP pancreatitis risk, and a simplified scoring system based on the LightGBM model. This model facilitates individual risk prediction and preventive strategy selection.

Introduction

Post-endoscopic retrograde cholangiopancreatography (ERCP) pancreatitis (PEP) is one most frequent and serious complication following ERCP, with an incidence rate from 2.1% to 15.1% (). Although most PEP cases are mild or moderate, specific conditions can significantly extend hospitalization for patients and be fatal in severe instances (). Based on these concerns, multitude of risk prediction models for PEP have been suggested, incorporating various patient and procedure-related factors such as gender, difficult cannulation, history of pancreatitis, pancreatic duct cannulation, etc. (). However, due to the intricate interplay of risk factors that may even synergize, these models often suffer from limited discriminative power, complexity, or lack of external validation, making their limited application in clinical practice.

In recent years, machine learning (ML) has gained considerable attention in clinical medical settings. ML algorithms analyze a wealth of variables with complex relationships using methods such as supervised, unsupervised, and semi-supervised learning, offering advantages such as intuitiveness and high predictive efficiency (). Research indicates that computer-aided diagnostic models substantially assist clinicians in diagnosing and predicting diseases (). At present, many ML models and algorithms based on diverse architectures have been developed, showing impressive performance in predicting significant diseases in the medical field (). Hence, our goal is to develop and validate practical PEP prediction models using the latest ERCP database from General Hospital of Ningxia Medical University. This study pursues two objectives: (1) to compare the performance of different ML models in predicting PEP with stringent inclusion and exclusion criteria; (2) to identify a clinically relevant model with as few predictive factors as possible through innovative approaches, aiding endoscopists in decision-making and planning postoperative management.

Materials and methods

Patient characteristics

We conducted a retrospective analysis of patients' clinical data from diagnostic or therapeutic ERCPs at General Hospital of Ningxia Medical University from May 2022 to June 2023. After applying rigorous inclusion and exclusion criteria, we included a total of 688 cases in the training and validation sets. Eligible patients were those with relevant biliary-pancreatic diseases indicated for ERCP, who had provided written informed consent for the procedure and had granted verbal or written permission for postoperative examinations. Patients were excluded if they presented with acute pancreatitis, had a history of previous ERCP or gastrointestinal reconstructive surgery. Patients that were under 18 years-old (), had incomplete medical records or surgical videos, did not complete necessary follow-up examinations in a timely manner, did not undergo a full cannulation attempt and abandoned the surgery, or were treated by an endoscopist with less than 50 cannulate procedures (), were also excluded (Figure 1).

Figure 1

Study endpoint and definitions of outcomes

The primary endpoint was established according to current international guidelines and consensus on the incidence of PEP. The criteria for the occurrence of PEP were defined as follows: (1) New onset of abdominal pain was consistent with pancreatitis (acute persistent upper left abdominal pain); (2) Serum amylase levels were three times greater than the upper limit of normal within 24 h after the procedure; (3) Imaging evidence of pancreatitis such as peripancreatic fluid extravasation, pancreatic gland enlargement due to edema, pancreatic duct dilation, pancreatic tissue necrosis, or formation of a pancreatic pseudocyst (two out of these three criteria must be met by the Atlanta Consensus 2012) (, ). Failed cannulation was defined as the inability to correctly enter the bile or pancreatic duct despite all techniques and efforts (). Difficult cannulation was characterized by a cannulation time exceeding 5 min, more than five papillary contacts, or incorrect entry into the pancreatic duct more than once by ESGE Guidelines 2019 (, ). We further monitored preoperative serum calcium ion levels, classifying them using our hospital's upper limit of normal value at 2.12 mmol. In this study, the threshold of patient's age was set at <60. The biliary brush cytology sampling methods included cytological brushing or forceps biopsy (). “Failure to clear bile duct stones” was determined as stones that were not retrieved or not completely retrieved following the full cannulation process. The patient retained a nasobiliary drain at the conclusion of the procedure ().

Data processing

To ensure the stability of the results, patients with missing video recording data were excluded (n = 142), which did not affect the results of this study. As the incidence of patients with PEP at our institution was approximately 11%, there was an imbalance in the proportion of patients with and without PEP. To address the imbalance in sample sizes at different levels of the outcome variable, we used the Adaptive Synthetic Sampling (ADASYN) algorithm. This approach was used to survey patients with PEP and normal samples accordingly, achieving a 1:1 match with patients who did not develop PEP postoperatively ().

Data separation and feature filtering

Prior to oversampling with ADASYN, we performed data separation by randomly dividing the dataset into two subsets (training and test) with a 7:3 ratio. These subsets were used for modelling and validation, respectively. To minimize the overfitting risk introduced by ADASYN oversampling, we implemented a robust validation strategy. Specifically, we employed 10-fold cross-validation during the model training and tuning phase. Furthermore, the final model was evaluated on a completely independent test set that was not involved in either the oversampling or cross-validation processes. Feature selection within the training set was conducted by a stepwise regression approach grounded in logistic regression with backward elimination (). Covariance test showed no significant abnormality. This procedure was implemented using the Mass package in R software (version 4.2.2). Variables were selected according to the principle of minimising the Akaike Information Criterion (AIC) and a threshold of p-value < 0.01. This process aimed to identify a subset of features that provided an optimal balance between power and dimensionality ().

ML model building, evaluation and interpretation

The dataset was subjected to both linear and non-linear ML models, including logistic regression (LR), support vector machines (SVM), gradient boosting trees (GBT) and neural network models. The SVM was implemented using both Linear and Radial Basis Function (RBF) kernels, the GBT was implemented using XGBoost (eXtreme Gradient Boosting) and LightGBM (Light Gradient Boosting Machine), while the neural network model was implemented using a Multi-Layer Perceptron (MLP). The predictive performance of all models was evaluated using five metrics: Area Under the Receiver Operating Characteristic Curve (AUROC), precision, recall, F1 score and accuracy. Finally, ML models with acceptable performance were selected based on the AUC curve.

Statistical analyses

Categorical variables were assigned numerical codes, and categorical variables were assigned numerical codes, and ization was conducted using EXCEL 2018. Python was used for data processing and model development, relying on the scikit-learn library for the construction of machine learning algorithms. Data statistics, variable selection and construction of conventional LR models were performed using R software (R4.2.2), with a p-value < 0.01 indicating statistical significance. All authors of this article had access to the research data, participated in data collection, reviewed and approved the final manuscript.

Results

Study population and baseline characteristics

A total of 1,748 patients who underwent ERCP procedures met the criteria for this study. Of these, 514 patients with pre-operative acute pancreatitis (hyperamylasemia) were excluded. Further 396 patients with previous ERCP or gastrointestinal reconstruction history were also excluded. A further 150 patients were excluded because the procedure was abandoned or performed by inexperienced endoscopist, the medical records or video data were incomplete. Each exclusion criterion was counted only once. Therefore, 688 patients were included in the final analysis. The study flow chart was shown in Figure 1. Table 1 showed the baseline characteristics of the study population, focusing on the variable PEP. The distribution of key characteristics, including sex, age, presence of diabetes, coronary heart disease, hypertension, history of biliary surgery, cholecystectomy, acute pancreatitis, Ca + and total bilirubin levels, and various clinical procedures and interventions, were reported. We then performed the PCA dimensionality reduction plot for ADASYN and found on difference between PEP (happen) and non-PEP (Not) patients (Figure 1B).

Table 1

CharacteristicPEP
Y, N = 88aN, N = 600a
Sex
Male45 (51.1%)327 (54.5%)
Female43 (48.9%)273 (45.5%)
Age
>6041 (46.6%)392 (65.3%)
<6047 (53.4%)208 (34.7%)
Diabetes
N76 (86.4%)517 (86.2%)
Y12 (13.6%)83 (13.8%)
Coronary heart disease
N81 (92.0%)535 (89.2%)
Y7 (8.0%)65 (10.8%)
Hypertension
N68 (77.3%)452 (75.3%)
Y20 (22.7%)148 (24.7%)
History of biliary surgery
N79 (89.8%)556 (92.7%)
Y9 (10.2%)44 (7.3%)
History of cholecystectomy
N80 (90.9%)542 (90.3%)
Y8 (9.1%)58 (9.7%)
History of acute pancreatitis
Y10 (11.4%)33 (5.5%)
N78 (88.6%)567 (94.5%)
Ca + (mmol/L)
>22.140 (45.5%)307 (51.2%)
<22.148 (54.5%)293 (48.8%)
Total bilirubin level (umol/L)
<22 umol/L69 (78.4%)453 (75.5%)
≥22 umol/L19 (21.6%)147 (24.5%)
Dilatation of extrahepatic bile ducts
Y65 (73.9%)503 (83.8%)
N23 (26.1%)97 (16.2%)
Direct bilirubin level (umol/L)
<5 umol/L34 (38.6%)249 (41.5%)
≥5 umol/L54 (61.4%)351 (58.5%)
Gallbladder stone
N64 (72.7%)433 (72.2%)
Y24 (27.3%)167 (27.8%)
Common bile duct stones
Y61 (69.3%)422 (70.3%)
N27 (30.7%)178 (29.7%)
Acute cholangitis
Y67 (76.1%)432 (72.0%)
N21 (23.9%)168 (28.0%)
Benign biliary stricture
N82 (93.2%)569 (94.8%)
Y6 (6.8%)31 (5.2%)
Malignancy biliary stricture
N73 (83.0%)523 (87.2%)
Y15 (17.0%)77 (12.8%)
Malignant diseases of the pancreas
N80 (90.9%)527 (87.8%)
Y8 (9.1%)73 (12.2%)
pancreatic duct calculus
N87 (98.9%)597 (99.5%)
Y1 (1.1%)3 (0.5%)
Mechanical stone crushing
N85 (96.6%)589 (98.2%)
Y3 (3.4%)11 (1.8%)
Stone residue
N9 (10.2%)27 (4.5%)
Y79 (89.8%)573 (95.5%)
Total duration of cannulation
≥5min23 (26.1%)94 (15.7%)
<5min65 (73.9%)506 (84.3%)
Total number of cannulation
≥5 times24 (27.3%)73 (12.2%)
<5 times64 (72.7%)527 (87.8%)
Number of guidewire entries into the pancreatic duct
Pancreatic guidewire passages ≤166 (75.0%)545 (90.8%)
Pancreatic guidewire passages >122 (25.0%)55 (9.2%)
Pancreatic injection
N72 (81.8%)556 (92.7%)
Y16 (18.2%)44 (7.3%)
Pancreatic stent placement
N82 (93.2%)524 (87.3%)
Y6 (6.8%)76 (12.7%)
Endoscopic biliary stenting
N74 (84.1%)521 (86.8%)
Y14 (15.9%)79 (13.2%)
EST
Y77 (87.5%)540 (90.0%)
N11 (12.5%)60 (10.0%)
Endoscopic nasobiliary drainage
Y78 (88.6%)537 (89.5%)
N10 (11.4%)63 (10.5%)
Sampling from the biliary tract
N82 (93.2%)582 (97.0%)
Y6 (6.8%)18 (3.0%)
Periampullary diverticulum
N40 (45.5%)466 (77.7%)
Y48 (54.5%)134 (22.3%)
Balloon dilatation
N62 (70.5%)450 (75.0%)
Y26 (29.5%)150 (25.0%)

The characteristics and their values considered in this study.

a

n (%).

Feature filtering and prediction model construction

Based on the characteristic sets listed in Table 1, we used a stepwise regression method with logistic regression and backward elimination to select variables for feature filtering. Seven characteristic sets were included in the features with smaller AIC curves in the training set and a p-value < 0.01: Periampullary diverticulum (PAD), pancreatic stent placement (PSP), Pancreatic injection (PI), number of guidewire passages into the pancreatic duct (NGP), dilatation of extrahepatic bile ducts (DEBD), age, and coronary heart disease (CHD). Based on the logistic regression analysis results table (Tables 2–4), the conventional LR model predicts the probability of pancreatitis using the following equation:

Table 2

Dependent: PEPValueOR (univariable)OR (multivariable)
History of cholecystectomyN––
Y0.55 (0.35–0.85, p = 0.008)0.63 (0.36–1.08, p = 0.097)
Balloon dilatationN––
Y1.04 (0.81–1.35, p = 0.744)1.35 (0.97–1.87, p = 0.072)
ESTN––
Y0.81 (0.56–1.16, p = 0.246)0.63 (0.39–1.00, p = 0.051)
Periampullary diverticulumN––
Endoscopic biliary stentingY4.26 (3.33–5.48, p < 0.001)7.06 (5.16–9.76, p < 0.001)
–N-–
Y1.05 (0.75–1.46, p = 0.776)–
Pancreatic duct calculusN––
Y0.98 (0.18–5.31, p = 0.981)–
Sampling from the biliary tractN––
Pancreatic stent placementY1.78 (1.00–3.28, p = 0.054)4.08 (2.01–8.53, p < 0.001)
N––
Y0.38 (0.24–0.58, p < 0.001)0.40 (0.23–0.67, p = 0.001)
Pancreatic injectionN––
Y1.82 (1.24–2.70, p = 0.003)1.70 (1.04–2.80, p = 0.036)
Number of guidewire entries into the pancreatic ductPancreatic guidewire passages ≤1––
Pancreatic guidewire passages >12.99 (2.15–4.21, p < 0.001)3.92 (2.58–6.01, p < 0.001)
Total number of cannulations≥5 times––
<5 times1.95 (1.43–2.67, p < 0.001)–
Total duration of cannulation≥5min-–
<5min1.66 (1.24–2.22, p = 0.001)1.34 (0.94–1.93, p = 0.109)
Stone residueN––
Y1.97 (1.23–3.22, p = 0.006)2.20 (1.21–4.07, p = 0.010)
Dilatation of extrahepatic bile ductsNondilated extrahepatic bile duct1.80 (1.36–2.40, p < 0.001)1.85 (1.30–2.65, p = 0.001)
Dilatation of extrahepatic bile ducts––
Malignancy biliary strictureN––
Y1.37 (1.00–1.90, p = 0.051)2.34 (1.44–3.84, p = 0.001)
Malignant diseases of the pancreasN––
Y0.61 (0.42–0.90, p = 0.013)–
Benign biliary strictureN––
Y0.81 (0.47–1.39, p = 0.451)0.52 (0.26–1.02, p = 0.057)
Acute cholangitisN––
Y1.28 (0.99–1.65, p = 0.065)3.74 (2.01–7.08, p < 0.001)
Gallbladder stoneN––
Y0.91 (0.71–1.17, p = 0.467)0.78 (0.57–1.07, p = 0.129)
Mechanical stone crushingN––
Y0.98 (0.42–2.31, p = 0.963)–
Endoscopic nasobiliary drainageN––
Y1.16 (0.80–1.70, p = 0.430)–
Total bilirubin level (umol/L)<22 umol/L––
≥22 umol/L1.45 (1.10–1.91, p = 0.009)–
History of acute pancreatitisN––
Y1.53 (0.97–2.43, p = 0.067)1.87 (1.07–3.30, p = 0.029)
Direct bilirubin level (umol/L)<5 umol/L––
≥5 umol/L1.21 (0.96–1.53, p = 0.103)1.31 (0.98–1.75, p = 0.068)
Common bile duct stonesN––
Y1.04 (0.82–1.34, p = 0.728)0.33 (0.18–0.59, p < 0.001)
History of biliary surgeryN––
Y0.98 (0.63–1.51, p = 0.923)–
SexMale––
Female0.83 (0.66–1.04, p = 0.104)-
DiabetesN––
Y0.82 (0.58–1.15, p = 0.244)–
Age>60––
<600.43 (0.34–0.54, p < 0.001)0.29 (0.22–0.39, p < 0.001)
HypertensionN––
Y0.65 (0.49–0.86, p = 0.003)–
Coronary heart diseaseN––
Y0.54 (0.36–0.82, p = 0.004)0.44 (0.26–0.75, p = 0.003)
Ca + (mmol/L)>22.1––
<22.10.78 (0.62–0.98, p = 0.030)0.73 (0.55–0.96, p = 0.024)

Logistic regression with stepwise variable reduction.

Table 3

Step DfDevianceResid. DfResid. DevAIC
Sex0.0021273101,1801,282.5511,346.551
Mechanical stone crushing0.0077540361,1811,282.5591,344.559
Endoscopic biliary stenting0.0777817371,2821,282.6371,342.637
Total bilirubin level0.1109312181,1831,282.7471,340.747
pancreatic duct calculus0.5937709741,1841,283.3411,339.341
Malignant diseases of the pancreas0.4769351891,1851,283.8181,337.818
Total number of cannulations0.4769351891,1861,284.4501,336.450
History of biliary surgery0.5688428981,1871,285.0191,335.019
Hypertension0.8436454191,1881,286.4211,334.421
Diabetes0.8436454191,1891,287.2651,333.265
Endoscopic nasobiliary drainage1.5114385201,1901,288.7761,332.77

Logistic regression with stepwise variable reduction.

Table 4

TermVIFVIFCI_lowVIF_CI_highToleranceTolerance_CI_lowTolerance_CI_highSE_factor
Sex1.2211.1551.3161.1050.8190.7600.866
Age1.2311.1631.3261.1090.8120.7540.860
Diabetes1.8561.7202.0181.3620.5390.4960.581
coronary heart disease1.7021.5821.8471.3050.5870.5410.632
Hypertension1.5041.4051.6271.2260.6650.6150.712
History of biliary surgery1.1681.1091.2591.0810.8560.7940.902
History of cholecystectomy1.1101.0601.2020.9000.8320.9431.054
History of acute pancreatitis1.0921.0451.1860.9150.8430.9561.045
Ca + (mmol/L)1.0921.0461.1861.0450.9150.8430.956
Total bilirubin level (umol/L)
Dilatation of
1.7611.6351.9131.3270.5680.5230.612
Extrahepatic bile ducts1.1381.0841.2291.0670.8780.8130.923
Direct bilirubin level(umol/L)1.8741.7362.0381.3690.5340.4910.576
Gallbladder stone1.1791.1181.2711.0860.8480.7870.894
Common bile duct stones2.6822.4602.9391.6380.3730.3400.407
Acute cholangitis3.5253.2153.8791.8780.2840.2580.311
Benign biliary stricture1.3301.2511.4351.1530.7520.6970.799
Malignancy biliary stricture2.7842.5513.0521.6680.3590.3280.392
Malignant diseases of the pancreas2.0141.8612.1931.4190.4970.4560.537
pancreatic duct calculus1.1301.0771.2211.0630.8850.8190.929
Mechanical stone crushing1.0001.000Inf1.0001.0000.0001.000
Stone residue1.2831.2091.3831.1330.7790.7230.827
Total duration of cannulation1.1241.0711.2151.0600.8900.8230.933
Total number of cannulations1.3001.2241.4011.1400.7690.7140.817
Number of guidewire entries into the pancreatic duc1.2751.2021.3741.1290.7840.7280.832
Pancreatic injection1.3361.2561.4411.1560.7490.6940.796
Pancreatic stent placement1.2791.2051.3781.1310.7820.7260.830
Endoscopic biliary stenting2.9752.7223.2651.7250.3360.3060.367
EST1.4521.3591.5691.2050.6890.6370.736
Endoscopic nasobiliary drainage2.6242.4072.8741.6200.3810.3480.415
Sampling from the biliary tract1.5731.4661.7031.2540.6360.5870.682
Periampullary diverticulum1.2431.1741.3391.1150.8050.7470.852
Balloon dilatation1.2551.1851.3531.1200.7970.7390.844

Covariance diagnostics.

To enhance the interpretability and clinical reliability of our model, we conducted a comprehensive Shapley additive explanations (SHAP) analysis and calibration analysis. For the five models, the SHAP summary figures intuitively showed the contribution of each feature to the model prediction, and highlights the most influential clinical variables (Figure 2). After that, we chose the intersection features selected by LASSO regression and stepwise methods for modeling, and finally removed the pancreatic injection (PI) feature from the equation. The final prediction equation was:

Figure 2

At the same time, we considered the CHD patients taking aspirin. Among 72 patients diagnosed with CHD, 50 patients (69.4%) regularly took aspirin before ERCP surgery, and only one patient (2.0%) had PEP incidence. For the 22 non-aspirin group, the PEP incidence rate was 18.2% (4/22), showing significant difference compared with aspirin-taken group (p-value = 0.024), indicating that aspirin effect on PEP incidence for CHD patients.

Model performance evaluation

The predictive performance that was measured by the receiver operating characteristic (ROC) model by comparing to the conventional logistic regression (LR) model, was depicted (Figures 3A–F). Models with an area under the ROC curve (AUC) exceeding 0.75 were considered as acceptable. This criterion was met by the following models: LR (Figure 3A) with a test AUC of 0.775 [95% confidence interval (CI) 0.727–0.823]; LR (ML) (Figure 3B) with a test AUC of 0.788 (95% CI 0.7347–0.8408); support vector machine (SVM) (Figure 3C) with a test AUC of 0.812 (95% CI 0.7612–0.8624); XGBoost (Figure 3D) with a test AUC of 0.840 (95% CI 0.7955–0.8851); LightGBM (Figure 3E) with a test AUC of 0.807 (95% CI 0.7575–0.8574); and the multilayer perceptron (MLP) model (Figure 3F) with a test AUC of 0.835 (95% CI 0.7887–0.8822). A detailed performance description for each ML model is supplemented in Table 5.

Figure 3

Table 5

model nameAccuracyAUC95% CISensitivitySpecificityPPVNPVPrecisionRecallF1ThresholdTask
LR0.6870.7620.7336–0.79050.4890.8500.7290.6690.7290.4890.5850.518PEP-train
LR0.7070.7880.7347–0.84080.5160.8640.7560.6860.7560.5160.6130.518PEP-test
SVM0.7540.7960.7696–0.82300.6970.8010.7430.7620.7430.6970.7190.430PEP-train
SVM0.7430.8120.7612–0.86240.6980.7790.7210.7590.7210.6980.7100.246PEP-test
XGBoost0.7500.8340.8110–0.85670.7010.7900.7330.7620.7330.7010.7170.451PEP-train
XGBoost0.7500.8400.7955–0.88510.6900.7990.7370.7590.7370.6900.7130.451PEP-test
LightGBM0.7340.8100.7852–0.83460.6570.7980.7280.7390.7280.6570.6910.468PEP-train
LightGBM0.7180.8070.7575–0.85740.8650.5970.6370.8440.6370.8650.7340.406PEP-test
MLP0.7290.8200.7957–0.84440.8750.6080.6480.8560.6480.8750.7450.430PEP-train
MLP0.7500.8350.7887–0.88220.9050.6230.6630.8890.6630.9050.7650.430PEP-test

The performance of all ML models.

Then we performed calibration analysis for each model, the calibration curves indicated that Logistic Regression (LR) demonstrated nearly ideal calibration, while LightGBM and Xgboost also showed reasonable calibration performance. In contrast, the calibration curves for SVM and MLP exhibit minor deviations from the ideal line, which remain within acceptable limits and do not substantially affect the clinical interpretability of the predictions (Figure 4A). Finally, we included a decision-curve analysis (DCA) to evaluate clinical benefit at different thresholds. This analysis allows for a direct comparison of the net clinical benefit across a range of threshold probabilities, providing a crucial perspective on the model's value in clinical decision-making beyond traditional discrimination metrics (Figure 4B).

Figure 4

Finally, we developed a simplified, points-based risk scoring system for potential bedside application based on the relative SHAP importance using the LightGBM model, which showed higher AUC score than other models. Using a pre-specified threshold of 3.46 points (derived from the risk distribution in our cohort), patients can be stratified into a Low-Risk group (score ≤3.46) and a High-Risk group (score >3.46) (Figure 5A). This simplified model, despite its ease of use, retained a clinically acceptable discriminative ability with an AUC of 0.75 on our validation set (Figure 5B).

Figure 5

Discussion

Major findings

In this study, we developed a predictive model for post-ERCP pancreatitis using ML techniques, employing stepwise regression for feature selection. Based on stringent inclusion and exclusion criteria, we independently identified six factors showing highest correlation with outcomes. We consider periampullary diverticulum, pancreatic duct stent placement, more than one guidewire passage into the pancreatic duct, non-dilated extrahepatic bile duct, age, and coronary heart disease as the most significant clinical variables affecting the risk of PEP. In subsequent ML model development, all ML models outperformed the conventional LR model, with Support Vector Machines (SVM), Gradient Boosting Trees (GBT), and Multi-Layer Perceptron (MLP) models achieving acceptable or superior performance. Figure 6 depicts the overall research process.

Figure 6

Comparison with the current models

Post-ERCP pancreatitis (PEP) represents one of the most frequent and severe complications associated with ERCP (). This complication can substantially diminish patients' quality of life, augment healthcare expenditures, and in its gravest manifestations, result in patient mortality (). Although a multitude of predictive models have been developed that demonstrate adept performance, their implementation is often impeded by their complexity, which hinders their integration into routine clinical practice. Furthermore, there may be an insufficient level of recognition among younger endoscopists regarding the potential risk of PEP occurrence. Therefore, it is necessary to develop a prediction model with acceptable accuracy and relatively straightforward predictors for clinical use. In 2021, Koichi Fujita et al. created a practical scoring system for PEP based on a multicenter study in Japan. Considering the weight of seven predictive factors, they developed a model with an acceptable fit and accuracy (AUROC = 0.791) (). In 2022, Chan Hyuk Park et al. proposed a model based on high-risk factors before and after ERCP but failed to validate the effectiveness of their preoperative risk prediction model (). Recently, our group used a logistic regression-based backward algorithm to select predictive factors for the establishment of traditional and machine learning models for PEP. Our machine learning model showed better performance and prospects for clinical application than previous models. Traditional multivariate regression is susceptible to small sample bias, particularly in the context of low probability events such as PEP. This approach may also exhibit limited generalization capabilities when addressing complex nonlinear relationships and instances where positive event samples are scarce. To circumvent these constraints, employing distinct ML algorithm-based models using preoperative and postoperative data can potentially enhance predictive performance and surpass the limitations inherent in existing models ().

ML model performance

ML models have been widely applied in the medical field due to their advantages in handling complex nonlinear relationships and high-dimensional data (). Substantial literatures have discussed the establishment of ML models related to acute pancreatitis. Our group also paid attention to the work published by Livia Archibugi and colleagues in May 2023, which may be the first discussion on using ML to predict the risk of PEP. They innovatively used the SHapley Additive exPlanations (SHAP) method to open the “black box” of algorithms and study how each feature contributes to the model. Unfortunately, neither traditional nor ML models showed AUROC curve values that could be referenced and applied clinically (Gradient Boosting = 0.67, LR = 0.56). They suggested that this might be related to insufficient model training due to data imbalance caused by the low incidence rate (6%) of PEP (). After augmenting the training set data using the ADASYN method, all models satisfactorily predicted the occurrence of postoperative pancreatitis in the validation set, with XGBoost showing the best AUC: 0.840 (95% Cl 0.7955–0.8851). Meanwhile, the above results also need to be further validated by statistical comparison such as DeLong test using bigger sample size in future. We recognize that ADASYN may induce the risk of overfitting, leading to forecast deviation. Meanwhile, it can indeed effectively improve the performance of classifiers and neural networks when the dataset is highly imbalanced; therefore, it is feasible to use these data to predict PEP events. That is, endoscopists can achieve dynamic prediction based on these features and decide on perioperative strategies, allowing nursing decisions to be more precise and personalized.

Features for PEP prediction

After ERCP, the interpretation and communication of ML models pose a challenge due to the existence of algorithmic “black boxes”. In this investigation, a model-agnostic interpretive approach was employed to discern the underlying clinical elements that may contribute to the onset of post-endoscopic pancreatitis in patients. Using a sequence backward stepwise regression algorithm, we are instrumental in predicting post we identified key clinical variables that are instrumental in predicting PEP. These variables include the presence of periampullary diverticulum, placement of a pancreatic duct stent, the frequency of guidewire cannulation into the pancreatic duct, non-dilated extrahepatic bile ducts, patient age, and the presence of coronary heart disease. Notably, pancreatic duct stent placement, guidewire passages, non-dilated extrahepatic bile ducts, and patient age are all recognized as either risk or protective factors according to the guidelines set forth by the European Society of Gastrointestinal Endoscopy (ESGE) or the American Society for Gastrointestinal Endoscopy (ASGE) (, , ). NGP has been used as an independent risk factor for PEP in one recent study (). In another study, the non-dilated extrahepatic bile ducts factor was found to be a significant predictor for PEP using the comprehensive systematic review and meta-analysis method (). Periampullary diverticulum may be somewhat controversial. From the data collected, most of the patients with periampullary diverticulum (PAD) included in our study had intra-diverticular papilla or a papilla less than 2 cm from the diverticulum. These diverticula could potentially affect the sphincter of Oddi, which normally serves as the main valve for the pancreaticobiliary tract. In patients with PAD, the presence of a diverticulum often causes the papilla to lose its normal morphology partially or completely, leading to variations in the direction of the pancreaticobiliary ducts (–). Additionally, the lack of duodenal smooth muscle support at the site of the diverticulum can lead to reduced tension in the distal walls of the pancreaticobiliary ducts, diminished emptying capacity, and post-sphincterotomy tissue edema exacerbates this condition (). A compelling piece of evidence is that sphincter of Oddi dysfunction (SOD) has been identified as an important and independent risk factor for PEP in most studies (). However, regarding coronary heart disease (CHD), current guidelines do not indicate it as an independent protective or risk factor for PEP. Patients with CHD are often on long-term medications, including aspirin. Yet, in our data, the group with CHD exhibited a lower incidence of PEP, aligning with the findings of a study published by Harsh K. Patel et al. They compared 1,374,773 ERCP procedures and found a lower incidence of PEP in patients with a history of myocardial infarction (MI) or coronary revascularization surgery (PCI or CABG) (14.1% vs. 15.4%, p < 0.001), though they did not discuss the reasons for this finding in their conclusion (). In many cases, patients undergoing urgent ERCP for conditions such as common bile duct stones or suppurative cholangitis may not switch to heparin bridging therapy in a timely manner. It is speculated that the lower incidence rate of PEP observed in this patient group could be attributed to the protective effect of nonsteroidal anti-inflammatory drugs (NSAIDs) administered preoperatively to those with CHD against PEP. This reflects the interaction between model variables, and since the Backward selection algorithm is generally more flexible in variable selection, considering more interactions and nonlinear relationships, it is better equipped to capture the complex influences and obtain solid association between CHD and other variables in future (). It also reminds us that when assessing a patient's PEP risk, it may be necessary to consider the impact of medications routinely taken by the patient on PEP, which could affect the choice of preventive measures and treatment strategies ().

Clinical implication and future application

Amidst the extensive discourse on the application and dosage of NSAIDs and other potential prophylactic medications for PEP, as proposed by various clinical guidelines—including those from ESGE, ASGE, and the Chinese Guideline for ERCP (2018 Edition)—it is imperative to consider the evidence underpinning these recommendations. it is questionable how many patients undergo a comprehensive PEP risk assessment preoperatively (, , , ). Therefore, the development of a streamlined predictive model for this population is worthy considering for endoscopists. Initially, a thorough evaluation based on the patient's physical condition and radiological findings should be conducted by endoscopists preoperatively. Patients with pre-existing CHD often took aspirin, which, although found to reduce the incidence of PEP in our data, increases the risk of bleeding due to the continued use of anticoagulants when the sphincterotomy is performed during the ERCP procedure (, ). Furthermore, if a periampullary diverticulum is discovered during the procedure, particularly Li-Tanaka type I or II diverticula that severely affect the papillary morphology (), the relationship between the opening of the papilla and the axis of the bile duct should be carefully considered to choose an appropriate cannulation method to reduce the incidence of postoperative complications (). In this study, we have identified some unique factors for early identification of PEP, which can provide guidance for future multicenter clinical trials. In future multicenter experiments, with independent validation and more case data, a scoring system for PEP can be established. Based on the confirmed model, we could develop an online risk assessment tool in the future to estimate the risk of PEP in ERCP patients (). In summary, if patients present with the high-risk factors mentioned above, clinicians should take corresponding measures to improve patient outcomes and expedite discharge.

Strengths and limitations

This investigation exhibited several notable strengths that warrant emphasis. Firstly, the study implemented a rigorous modeling procedure that encompassed stringent criteria for inclusion and exclusion, exhaustive data processing, meticulous feature selection, model construction, and assessment. This methodological approach yielded a predictive model characterized by streamlined factors yet relatively practical outcomes. Meanwhile, a checklist of reporting guidelines such as transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) should be performed to improve reproducibility (38). Secondly, this study applied ML methods for the first time to validate the filtered model features, offering higher predictive accuracy than traditional linear models even when dealing with complex data patterns and associations. However, there are several limitations to acknowledge. Primarily, this study utilized a retrospective single-center cohort study due to the limitation of time and resource, limiting the generalizability of this model to other regions due to the differences in patient demographics or procedural practices. Despite the strictest inclusion and exclusion criteria, the small sample size necessitates prospective observational studies for more rigorous validation. The smaller sample size also poses the risk of overfitting in ML models. Secondly, due to the low incidence of PEP postoperatively at our institution, there is a significant imbalance between the samples of PEP patients and non-PEP patients. For model evaluation, precision, recall, and other metrics of the predicted model at different thresholds should be reported besides the AUC assessment. This study relies on stepwise logistic regression for feature screening, which may miss prediction patterns that can only be recognized by nonlinear models (such as high-order interaction terms or threshold effects). Although the final XGBoost model can still partially compensate for such omissions through tree structure, future research should explore ML methods based on regularization to better adapt to the needs of complex models. The interpretability tools for ML models such as SHAP or LIME also should be considered to confirm the model's clinical decision-making basis. Meanwhile, the information about drugs usage was not fully considered in this study because of the limitation of clinical information of patients. Although an adaptive synthetic sampling algorithm is used to balance the sample sizes, external validation is still required in future research. In summary, these limitations call for a large-scale and multicenter cohort study in future to validate this model.

Conclusion

In conclusion, we developed and validated a streamlined predictive model for PEP, enhancing our understanding of PEP risk factors in our population. The XGBoost and MLP models outperformed other algorithms, highlighting key preoperative and intraoperative variables. These findings can be used to construct specific clinical application scenarios or tool to guide endoscopists in optimizing clinical outcomes for patients with biliary and pancreatic conditions.

Statements

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding authors.

Ethics statement

The studies involving humans were approved by Ningxia Medical University General Hospital Medical Research Ethics Review Committee with Clinical trial number: 2020-633. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

TD: Conceptualization, Data curation, Investigation, Validation, Writing – original draft. GD: Writing – review & editing. HY: Writing – review & editing. HW: Writing – review & editing. WW: Writing – review & editing. TM: Writing – review & editing. JM: Writing – review & editing. HW: Writing – review & editing. QW: Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. This research was supported by the Ningxia Autonomous Region Key R&D Program Project (2021BEG02038).

Acknowledgments

We thank the collaborators for their contribution and involvement in the study.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declare that no Generative AI was used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Summary

Keywords

post-ERCP pancreatitis, artificial intelligence, predictive model, machine learning, risk prediction

Citation

De T, Du G, Yin H, Wang H, Wang W, Ma T, Ma J, Wang H and Wang Q (2025) Development and validation of a practical prediction model for post-ERCP pancreatitis using machine learning. Front. Surg. 12:1628956. doi: 10.3389/fsurg.2025.1628956

Received

15 May 2025

Accepted

03 October 2025

Published

03 November 2025

Volume

12 - 2025

Edited by

Wandong Hong, First Affiliated Hospital of Wenzhou Medical University, China

Reviewed by

Livia Archibugi, San Raffaele Hospital (IRCCS), Italy

Md.Enamul Hoq, University of Arkansas for Medical Sciences, United States

Muhammad Daniyal Waheed, Maroof International Hospital, Pakistan

Updates

Copyright

*Correspondence: Qi Wang Hao Wang

Disclaimer

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Outline

Figures

Cite article

Copy to clipboard


Export citation file


Share article

Article metrics