Development and External Validation of a Nomogram to Predict Recurrence-Free Survival After R0 Resection for Stage II/III Gastric Cancer: An International Multicenter Study

Background: The benefit of adjuvant chemotherapy varies widely among patients with stage II/III gastric cancer (GC), and tools predicting outcomes for this patient subset are lacking. We aimed to develop and validate a nomogram to predict recurrence-free survival (RFS) and the benefits of adjuvant chemotherapy after radical resection in patients with stage II/III GC. Methods: Data on patients with stage II/III GC who underwent R0 resection from January 2010 to August 2014 at Fujian Medical University Union Hospital (FMUUH) (n = 1,240; training cohort) were analyzed by Cox regression to identify independent prognostic factors for RFS. A nomogram including these factors was internally and externally validated in FMUUH (n = 306) and a US cohort (n = 111), respectively. Results: The multivariable analysis identified age, differentiation, tumor size, number of examined lymph nodes, pT stage, pN stage, and adjuvant chemotherapy as associated with RFS. A nomogram including the above 7 factors was significantly more accurate in predicting RFS compared with the 8th AJCC-TNM staging system for patients in the training cohort. The risk of peritoneal metastasis was higher and survival after recurrence was significantly worse among patients calculated by the nomogram to be at high risk than those at low risk. The nomogram's predictive performance was confirmed in both the internal and external validation cohorts. Conclusion: A novel nomogram is available as a web-based tool and accurately predicts long-term RFS for GC after radical resection. The tool can also be used to determine the benefit of adjuvant chemotherapy by comparing scores with and without this intervention.


INTRODUCTION
Gastric cancer (GC) is the fifth most common malignancy worldwide and ranks third in cancer-related mortality (1). Radical gastrectomy is still the main treatment. However, even with radical resection, postoperative recurrence is common, affecting ∼18 to 45.5% of patients (2)(3)(4)(5). Patients with stage I GC have good prognosis and low recurrence rates, while outcomes for those with stage II/III GC patients vary widely and can be challenging to predict (3, [6][7][8]. Postoperative recurrence is the leading cause of death among patients with stage II/III GC (2,8). Currently, the most widely used system for estimating survival and risk of recurrence is the 8th edition of the American Joint Committee on Cancer (AJCC) tumor-node-metastasis (TNM) staging system. However, for patients with non-metastatic GC, the AJCC-TNM system considers only two variables (pT, pN), and its ability to predict recurrence is still limited (2,9). A predictive model including 6 variables that significantly better predicts overall survival (OS) than the AJCC-TNM staging system was developed by Han et al. but its predictive power for recurrence in patients with stage II/III GC was not evaluated (10). To better plan follow-up and treatment strategies for these patients, an individualized predictive tool to predict recurrence would be of value.
Another challenge in treating patients with stage II/III GC is determining who will benefit from adjuvant (postoperative) chemotherapy. While this treatment strategy has become standard for these patients (11)(12)(13) and has improved their long-term prognosis, whether all patients with stage II/III GC need adjuvant chemotherapy has been questioned (14,15). A model to predict its benefit in patients with stage II/III GC was introduced by Jiang et al. but its predictive performance was not ideal (concordance index was 0.686), and it was not externally validated using Western data (16).
Therefore, in the present study, we used data from a large-volume center in China to construct a nomogram that can effectively predict postoperative recurrence and chemotherapy benefit in patients with stage II/III GC after radical surgery. The model was internally and externally validated using data from our center and a Western cohort, respectively. This is the first predictive model, which is available as a web-based tool, for postoperative recurrence and chemotherapy benefits based on international, multicenter data.

Datasets
The database at Fujian Medical University Union Hospital (FMUUH) was reviewed following approval from the Institutional Review Board (IRB). The inclusion criteria were as follows: histologically confirmed primary gastric cancer, no distant metastasis, and R0 gastrectomy performed between January 2010 and August 2014. The exclusion criteria included receiving neoadjuvant chemotherapy, pathologic stage I disease according to the 8 th -AJCC-TNM staging system, remnant gastric cancer, and postoperative death within 3 months.
To examine the generalizability of the model, data on patients that satisfied the aforementioned inclusion and exclusion criteria were obtained from FMUUH between September 2014 and August 2015 (internal validation) and from the Mayo Clinic between January 2005 and December 2012 (external validation) following IRB approval.
Tumor stage, including pT, pN, and final stage, was determined according to the 8 th AJCC classification system (17).

Follow-Up and Treatment
Follow-up visits for both cohorts generally consist of clinic visits every 3 months for the first 2 years and every 6 months for years 3 to 5. Most routine patient follow-up appointments include a physical examination, laboratory tests, chest radiography, abdominal ultrasonography, or CT, and an annual or biannual endoscopic examination for patients with a remnant stomach (18). Disease recurrence was diagnosed with radiologic findings on cross-sectional imaging or biopsies of suspicious lesions (3).
For those who could tolerate adjuvant chemotherapy, adjuvant chemotherapy was routinely recommended for patients with pathological stage II and III disease (19). The adjuvant chemotherapy consisted of either single-agent 5-fluorouracil (5-FU) or a combination of 5-FU and cisplatin/oxaliplatin or paclitaxel (19). To simplify of the nomogram, patients were classified as having received chemotherapy or not, regardless of the number of cycles (16).

Definition and Categorization of Recurrence
Recurrences were categorized by site involved as previously described: (2,3,20) locoregional, peritoneal, distant, or multiple. Multiple recurrences were defined as the presence of recurrent disease in 2 or more sites. Early recurrence was defined as recurrence occurring within 12 months (3). Patients for which the exact site or sites of recurrence were unknown because of diagnosis in other hospitals were excluded from the analysis of recurrence patterns.

Nomogram Construction, Validation, and Calibration
A Cox proportional hazards regression model was used to identify independent prognostic factors associated with RFS. Variables with p < 0.05 in the univariable analysis were subsequently included in the multivariable analysis, from which a nomogram was formulated in R for predicting the probability of 5-year recurrence-free survival (RFS).
The nomogram was subjected to 1,000 bootstrap resamples for internal and external validation. Its performance in predicting outcomes was evaluated by calculating the concordance index (C-index), area under the curve (AUC), and Akaike information criterion (AIC) (3). The nomogram was calibrated by comparing predicted with observed RFS after bias correction.

Statistical Analysis
Continuous variables are reported as means ± SD or medians (interquartile ranges). Categorical variables were compared using the χ (2) or Fisher's exact test and continuous variables by t-test. RFS was assessed using the Kaplan-Meier method. Postrecurrence survival was defined as the period from the date of recurrence to the date of death or final follow-up. The nonlinear relationship between nomogram-derived scores and RFS was modeled using restricted cubic splines (21). The nomogram's clinical usefulness was evaluated by decision curve analysis, which calculates the rate of true and false positives for various risk thresholds (3). The cohort was dichotomized into lowrisk and high-risk subgroups by the median nomogram score among patients in the training cohort. Statistical analyses were performed using SPSS v.18.0 for Windows (SPSS Inc., Chicago, IL, USA) and R (https://www.r-project.org/). The R package "DynNom" was used to develop the web-based nomogram. P < 0.05 were considered statistically significant.

Characteristics of the Training, Internal Validation, and External Validation Cohorts
A total of 1,240 patients with stage II/III GC who underwent radical gastrectomy were included in the training cohort. There were 306 and 111 patients were included in the internal and external validation cohort, respectively (Supplementary Figure 1).

Nomogram Development, Calibration and Internal Validation
Multivariable analysis identified age, differentiation, tumor size, number of examined lymph nodes, pT stage, pN stage, and adjuvant chemotherapy as associated with RFS ( Table 2). We included the above variables in the predictive model to establish a nomogram and made it available online (https:// qq406918430.shinyapps.io/DynamicPrediction/) (Figure 1,  Supplementary Figure 2). The benefit of chemotherapy can be calculated by using the web-based calculating tool to determine the 5-year RFS probabilities both for the situation in which the patient receives or does not receive it; the difference between the two is the net survival benefit.
In the training cohort, the calibration curve showed excellent agreement between nomogram-predicted and actual observed 5-year RFS (Figure 2A). Supplementary Figure 3 shows the  Table 3). The ROC curve of 5year RFS showed an excellent predictive value (AUC 0.841) ( Figure 2B). The restricted cubic splines also confirmed the correlation between nomogram score and risk of recurrence (Supplementary Figure 4).
In the internal validation cohort, the same findings were observed in the calibration curve (Supplementary Figure 5A). In addition, the model performed better than the 8th AJCC-TNM in predicting 5-year RFS and 5-year OS (higher C-index and smaller AIC value) ( Table 3) with a high AUC for 5-year RFS (0.829) (Supplementary Figure 5B).

Nomogram External Validation and Clinical Applicability
In the external validation cohort, the calibration curve also showed good agreement between the 5-year RFS rates predicted by the nomogram and the actual 5-year RFS rates ( Figure 2C). In addition, the model performed better than the 8th AJCC-TNM in predicting 5-year RFS and 5-year OS (higher C-index and smaller AIC value) ( Table 3). The ROC curve for 5-year RFS showed an excellent predictive value (AUC: 0.752) (Figure 2D).
Decision curves showed that using the nomograms to predict the 5-year RFS rates provides more benefit than the 8th AJCC-TNM in the three cohorts. (Figures 2E,F,  Supplementary Figure 5C).

Risk Group Stratification Based on Nomogram Score
The median nomogram score of the training cohort, 212, effectively distinguished populations of different recurrence risk in the training, internal and external validation cohorts ( Figures 3A,B, Supplementary Figure 5D). We next analyzed the relationship between risk group and recurrence pattern in patients for whom the exact site(s) of recurrence as known [n = 465 for the training cohort and n = 168 for the combined validation cohorts (due to the small sample size of internal and external cohorts)] and found that the proportion of peritoneal  (Figures 4A,B). Within this subset of patients (465 in the training cohort and 168 in the combined validation cohorts), the post-recurrence survival of patients at high risk was inferior to that of low-risk patients in both cohorts (Figures 3C,D).

DISCUSSION
The present study used data from 2010 to 2014 in a highvolume Eastern cancer center to construct a nomogram, which showed good predictive value for 5-year RFS and 5-year OS among patients with stage II/III GC. Its predictive value was also validated internally and externally using data from the U.S. In addition, the model has good predictive efficacy than the 8th AJCC-TNM. More importantly, because adjuvant chemotherapy was included in the model, it allows simple calculation of the benefit of this treatment for an individual patient. To simplify the nomogram's use, we have made it available as a free webbased calculator. The nomogram we have developed represents an advance over other recent tools for predicting the outcomes of patients with GC after gastrectomy. Indeed, there have been several nomograms established for GC so far. The model constructed by Han et al. had significantly better predictive value for OS than the AJCC-TNM staging system (10). However, its predictive value for recurrence was not evaluated. The nomogram introduced by Jiang et al. for predicting the disease-free survival (DFS) of patients with stage II/III gastric cancer had limited predictive value, and did not include adjuvant chemotherapy as a variable despite the study's finding that it was an independent prognostic factor for DFS (16). Several scholars recommended that treatment strategy be included in a nomogram (22,23). These two models also lacked external validation using Western data. Further, a detailed comparison with the other published predictive models (24)(25)(26)(27)(28) for RFS is supplemented in Supplementary Table 1. Strengths of the present study include long-term follow-up information, large sample size and patients from Eastern and Western countries.
Although the small sample size, the validation of the nomogram using Western data is a particular advantage because of the clinical and pathological differences in GC between Asia vs. the U.S. and Europe (29). For example, the proportion of diffuse-type GC is higher in Asian patients, while proximal tumors are more frequent in the West. In addition, contributing factors such as environmental exposure and diet differ between geographic regions, as do standards of treatment. Neoadjuvant therapy is preferred in the U.S. and Europe, which is not common in China. In the present study, we focused on the efficacy of postoperative adjuvant chemotherapy and only included patients without neoadjuvant therapy. The postoperative adjuvant chemotherapy consisted of either single-agent 5-fluorouracil (5-FU) or a combination of 5-FU and cisplatin/oxaliplatin or paclitaxel in both China and the U.S. It is interesting that despite the selection bias of including only Western patients who did not receive neoadjuvant therapy, who are presumably the more frail and older patients, the nomogram still demonstrated remarkable predictive value for these patients. In addition, despite the differences in background characteristics, our model still showed strong predictive power in the external validation cohort (AUC 0.780), making it widely applicable.
The ability of the nomogram developed herein to calculate the benefit of adjuvant chemotherapy represents another important step forward. While randomized clinical trials have shown improvements in DFS and OS with adjuvant capecitabine + oxaliplatin (12) and S-1 (13), their benefit for all stage II/III GC patients remains unclear. While Jiang et al. two predictive models allowed calculation of the difference in survival for the same patient if they did or did not receive chemotherapy, their predictive value was not ideal, and required two separate manual calculations (16). In addition, the fact that the model was built on data from two separate groups likely created bias, and it was not validated in a Western population. In contrast, our model was created using data from all stage II/III GC patients together, allows more straightforward calculation of chemotherapy benefit, and showed good predictive performance (C-index: 0.774; AUC: 0.841) that was validated in a Western population. In addition, the model is available as a web-based tool, which makes the calculation more easily (Supplementary Figure 2). We just type in the patient's information on the web page, the 5-year RFS probabilities both for the situation in which the patient receives or does not receive ACT are calculated automatically. The difference between the two is the net survival benefit from the addition of ACT. However, the threshold difference in RFS at which ACT provides a net benefit remains undetermined, calling for further prospective studies to identify a specific cut-off. The current study also sheds light on postoperative recurrence patterns, which is important for developing appropriate followup and treatment strategies. Among the 633 patients with recurrence, those with a nomogram score >212 were more prone to peritoneal metastasis (∼30%). Differences in recurrence patterns have been associated with varying pathological stage (8) and Lauren type (2), and increased risk of peritoneal metastasis has previously been linked to gastric signet ring cell histology and neural invasion (9). As the outcomes of patients with peritoneal metastasis are poor, early monitoring is essential, as is reducing its incidence. To that end, early postoperative hyper thermic intraperitoneal chemotherapy (HIPEC) has been shown to reduce peritoneal metastasis (3 vs. 23%, p < 0.05) (30), with similar findings in another randomized controlled trial (RCT) from Russia (31). Therefore, further RCTs determining whether early postoperative HIPEC may be appropriate treatment for patients considered at high risk for peritoneal metastasis, such as those with high nomogram scores, are warranted.
It's known that peri-operative chemotherapy for GC patients is commonly used in the West. While postoperative chemotherapy may seem redundant for patients with neoadjuvant chemotherapy, it may be beneficial, as a study by Schumacher et al. showed that patients with advanced gastric cancer who underwent surgery with neoadjuvant therapy had improved R0 resection rates, but no survival benefit (32). However, the present study and previous studies on the efficacy of adjuvant chemotherapy have excluded patients who received neoadjuvant chemotherapy, so determining its benefit in such patients requires further study.
The present study has several limitations. Many levels of selection bias could result from its retrospective nature and non-random assignment of adjuvant chemotherapy, which was based on clinician experience and patient condition. Second, due to the data limitation and to simplify the model and improve its clinical applicability, we did not explore the effect of the number of chemotherapy cycles on prognosis. Third, due to the data limitation, we only considered the survival benefit of adjuvant chemotherapy and not its side effects, which vary from person to person (33). Previous studies showed that grade 3 or 4 adverse events were reported in 10%∼55% patients with different chemotherapy regimens (34)(35)(36), which were in the acceptable range. However, PAC should be withheld from patients who were identified in the no-benefit group to avoid unnecessary adverse effects related to PAC. Finally, we did not consider the impact of tumor immunerelated indicators and microsatellite instability. Even though an increasing number of studies have shown them to be associated with prognosis lack of benefit of chemotherapy in gastric cancer (14,15), they are not likely to come into common use recently.

CONCLUSION
Using data from Asia and the U.S., we established and validated a predictive model that can effectively predict 5-year RFS and the benefit of adjuvant chemotherapy after radical resection of stage II/III GC. We showed that the model has significantly better predictive power and clinical applicability than the AJCC-TNM staging system. The tool is available as a simple calculator via the web, making it easier for clinicians to apply. Further large-scale, prospective, external validation is warranted.

DATA AVAILABILITY STATEMENT
The dataset analyzed for this study is available from the corresponding author on reasonable request.

ETHICS STATEMENT
The studies involving human participants were reviewed and approved by Fujian Medical University Union Hospital Ethics Committee and Mayo Clinic Ethics Committee. The patients/participants provided their written informed consent to participate in this study.