Development and Validation of a Score for Screening Suicide of Patients With Neuroendocrine Neoplasms

Background: If the diagnosis of neuroendocrine neoplasm (NEN) increases the risk of patients to commit suicide has not been investigated so far. Identifying NEN patients at risk to commit suicide is important to increase their life quality and life expectancy. Methods and findings: Cancer cases were extracted from the Surveillance, Epidemiology, and End Results program and were divided into the NEN and the non-NEN cohorts. Subsequently, the NEN patients were randomly split into a training data set and a validation data set. Analyzing the training data set, we developed a score for assessing the risk to commit suicide for patients with NEN. In addition, we validated the score using the validation data set and evaluated, if this score could also be applied to other cancer entities by using the test data set, a non-NEN cohort. The odds ratio (OR) of suicide between NEN and non-NEN patients was determined. Moreover, the performance of a score was evaluated by the receiver operating characteristic curve and the area under the curve (AUC). Compared to non-NEN, NEN significantly increased the risk of suicide to 1.8-fold (NEN vs. non-NEN; OR, 1.832; P < 0.001). In addition, we observed that age, gender, race, marital status, tumor stage, histologic grade, surgery, and chemotherapy were associated with suicide among NEN patients; and a synthesized score based on these factors could significantly distinguish suicide individuals from non-suicide individuals in the training data set (AUC, 0.829; P < 0.001) and in the validation data set (AUC, 0.735; P < 0.001). This score also had a good performance when it was assessed by the test data set (AUC, 0.690; P < 0.001). This demonstrates that the score might also be applicable to other cancer entities. Conclusions: This population-based study suggests that NEN patients have a higher risk of suicide than non-NEN patients. In addition, this study provided a score, which can identify NEN patients at high-risk of committing suicide. Thus, this score in combination with current screening and prevention strategies for suicide may improve life quality and life expectancy of NEN patients.


INTRODUCTION
Neuroendocrine neoplasm (NEN) arises from neuroendocrine cells and is commonly observed in lung, stomach, intestines, and pancreas (1,2). Recent evidence proved that the incidence of lung and gastroenteropancreatic NEN increased 6.4-fold from 1973 to 2012 in the USA (3). Interestingly, distinct from other tumors, NEN is indolent. In addition, the survival time of these patients is longer than that of patients suffering from adenocarcinoma or squamous cell carcinoma (3). Hence, improving the surveillance and quality of life after diagnosis of NEN has become an important clinical issue.
Suicide has developed into an enormous public health problem all around the globe, and it is one of the leading causes of death in the USA (4)(5)(6). Notably, several studies suggest that the suicide incidence of cancer survivors is significantly higher than that of patients suffering from chronic diseases or in the general population (7,8). However, suicide can potentially be prevented, when individuals at risk for suicide are identified and proper psychological support is given. A mandatory prerequisite is the effective recognition of patients at elevated risk of suicide.
Previous studies suggest that NEN is positively associated with psychiatric disorder symptoms, such as depression, anxiety, or psychosis (9)(10)(11). In addition, it is well-known that psychiatric disease is a master contributor to suicides (12). Thus, we hypothesized that NEN may increase the risk of suicide when directly compared to non-NEN. However, to our knowledge, no study addressing this issue has been published. Even though several studies have emphasized the screening of anxiety and depression (13), and some studies have investigated the risk factors of suicide (14)(15)(16), none of them provided and evaluated a strategy, which can effectively identify NEN patients at an elevated risk of suicide.
Thus, the primary purpose of this study was to evaluate if NEN increases the risk of suicide. Secondly, we developed and evaluated a score, which may identify NEN patients at high risk for committing suicide. In this study, we observed that NEN significantly increase the risk of suicide, and we developed and validated a score to identify patients at high-risk for committing of suicide. The score in combination with current screening and psychological management of potential suicides might help to prevent or at least reduce suicidal death in patients suffering from neuroendocrine neoplasms.

Data Source and Study Population
Surveillance, Epidemiology, and End Results (SEER) program is an authoritative source of cancer patients' information in the Abbreviations: NEN, neuroendocrine neoplasm; SEER, Surveillance, Epidemiology, and End Results; PMR, proportionate mortality ratio; OR, odds ratios; CIs, confidence intervals; β i , β coefficient; W iRef , weight of the reference category; W ij , weight of the other categories; ROC, receiver operating characteristic; AUC, area under the curve; NR3C1, neuron-specific glucocorticoid receptor; PHQ-9, nine-item patient health questionnaire; HADS, hospital anxiety and depression scale; pNET, pancreatic neuroendocrine tumors; PDAC, pancreatic ductal adenocarcinoma.
USA. Dependent on the number of registries, the SEER program has four databases: SEER 9, SEER 13, SEER 18, and SEER 21. This study was performed with the data obtained from SEER 18, which includes ∼28% of the USA population (17).

Study Variables
Notably, the SEER program does not record the suicidal ideation or previous suicide attempts. We defined suicidal behavior dependent on the code of 50,220, which suggested that the patients died of suicide and self-inflicted injury (16). In order to determine the proportionate mortality ratio (PMR) of suicide, we identified the dead cases from the SEER program.
In order to determine the risk factors of suicide, we defined the following variables as covariates: population characteristics (age at diagnosis, gender, race, and marital status); tumor characteristics (tumor stage and histologic grade), and treatment (surgery and chemotherapy). Notably, the histologic grade of tumors was not defined according to the recommendations of WHO and the European Neuroendocrine Tumor Society classification (19,20). It was determined based on the morphological description in the pathology reports. For example, SEER grade I suggested that tumors were classified as welldifferentiated; grade II suggested that tumors were classified as moderately differentiated; grade III suggested that tumors were classified as poorly differentiated, and grade IV suggested that tumors were classified as undifferentiated or anaplastic (17).

Statistical Analysis
The mortality was calculated using SEER * Stat software and presented as per 100,000 population (Supplementary Table 1). The PMR of suicide was calculated by the following formula: (the number of patients committing suicide/the number of patients died) × 100% (Figure 2). The difference of the ratio between the NEN cohort and the non-NEN cohort was evaluated by Chi-squared test. In order to evaluate if NEN was a risk factor for committing suicide, a univariate logistic regression was performed, and the variables which have P < 0.05 were included in the multiple logistic regression (enter method, Table 1).
To develop the score, we followed the suggestion of the TRIPOD statement (21), 70 and 30% of the NEN patients were randomly split into a training data set and a validation data set, respectively (for basic characteristics of patients in both data sets see Table 2). To determine the risk factors for committing suicide, univariate logistic regression analyses were performed using the training data set; the odds ratios (ORs) and 95% confidence intervals (CIs) of variables were calculated (Figure 3). The β coefficient (β i ) of each category was determined by multiple logistic regression analysis ( Table 3). In order to determine the weight of each category, the age was converted into a categorical variable, and the weight was assigned as the value of average age (22,23). In addition, for the other categorical variables such as gender, race, marital status, tumor stage, histologic grade, surgery, and chemotherapy, the weight of the reference category (W iRef ) was defined as 0 and the weight of the other categories (W ij ) were assigned as 1. Subsequently, the weight of each category was adjusted by the following formula: β i × (W ij -W iRef ). In order to determine the constant score B ( Table 3) of FIGURE 2 | The proportionate mortality ratio (PMR) of suicide. The PMR of suicide of NEN patients was significantly higher than that of non-NEN patients in cohorts with all listed types of cancer (A) as well as only lung and bronchus (B), pancreatic (C), appendix (D), and rectum cancer (E). However, no significant difference for PMR of suicide between NEN and non-NEN patients was found, when analyzing stomach cancer (F), small intestine cancer (G), and colon cancer cohorts (H). The P-values were determined by Chi-squared test, and P < 0.05 indicates a significant difference.
Frontiers in Psychiatry | www.frontiersin.org the system, we defined B to reflect the decrease of risk when age has increased 10 years (22): B = −0.05 × 10 = −0.5. Finally, the score of each category was computed by β i × (W ij -W iRef )/B (22)(23)(24). In order to evaluate and validate this score, the performance of this strategy was determined by receiver operating characteristic (ROC) curve analysis using the training data set, the validation data set, and non-NEN cohort, respectively. The area under the curve (AUC) was presented. When the AUC is higher than 0.5, it suggests that the score can discriminate between suicidal patients and non-suicidal patients. All statistical analyses were performed using SPSS 19.0 (IBM, Armonk, New York), and P < 0.05 was considered to indicate statistical significance.

NEN Increases the Risk of Suicide
During the period of 2000-2016, 95 NEN patients (mortality, 0.00629/100,000; 95% CI, 0.00503/100,000-0.00776/100,000) and 1,773 non-NEN patients (mortality, 0.13089/100,000; 95% CI, 0.12483/100,000-0.13718/100,000) died of suicide (Figure 2A and Supplementary Table 1). Interestingly, we observed that the PMR of the NEN cohort was over 2-fold higher than that of the non-NEN cohort (Figure 2A). In addition, we evaluated the PMR in different tumor sites. Again, we observed that the PMR of NEN patients was higher than that of non-NEN patients in lung and bronchus cancer (Figure 2B), pancreatic cancer (Figure 2C), cancer of the appendix (Figure 2D), rectum cancer (Figure 2E), stomach cancer (Figure 2F), small intestine cancer (Figure 2G), and colon cancer ( Figure 2H). In order to evaluate if NEN is a risk factor for suicide, the multiple logistic regression analysis was subsequently performed. We observed that NEN significantly increased the risk of suicide by 1.8-fold in comparison to non-NEN ( Table 1).

Development of a Score for Predicting Suicides Among NEN Patients
In order to develop a score for predicting suicide, we excluded 3,374 participants, because the data of race, marital status, tumor stage, or surgical treatment were not recorded in the SEER program. Thus, 25,565 NEN patients were included in the subsequent analyses, and were randomly split into a training data set and a validation data set. The clinicopathological characteristics of each data set were summarized in Table 2. We did not observe significant differences of any clinicopathological characteristics between the two data sets. The ORs and 95% CIs of each FIGURE 3 | The univariate logistic regression of suicide using the training data set. The univariate logistic regression demonstrated that age, gender, race, marital status, cancer stage, cancer grade, surgery, and chemotherapy were associated with suicide of neuroendocrine neoplasm (NEN). The odds ratio (OR) was presented by a square, and the 95% confidence interval (CI) was presented by the horizontal lines with bars.
variable were calculated using the training data set. We observed that being young, male, American Indian/Alaska Native, single, localized tumor, well-differentiated, surgery, and non-chemotherapy were significantly (P < 0.05) associated with suicide (Figure 3). In order to develop the score system, the β i and the weight of each category were determined as indicated in Table 3, and the score of each category was determined by β i ×(W ij -W iRef )/B (Tables 3, 4).

Validation of the Score for Predicting Suicide Among NEN
The performance of the score in predicting suicide was evaluated by ROC curve using the training data set. There, we observed that the score could indeed significantly (P < 0.001) discriminate suicide patients from non-suicide patients (AUC, 0.829; 95% CI, 0.775-0.883; Figure 4A). To evaluate if the score is significantly superior to each of the single predictors such as age, gender, chemotherapy, surgery, tumor stage, race, material status, and tumor grade, we additionally performed the ROC curves of these predictors (Supplementary Figure 1). These analyses demonstrated that the AUC of the score was significantly higher than that of each single predictor ( Figure 4B). In addition, in order to validate the performance of the score, the ROC curve analysis was performed using the validation data set of NEN cohort. We observed that the AUC of the validation data set was 0.735 (95% CI, 0.647-0.823; P < 0.001, Figure 4C). Moreover, we tested if this score was also able to predict the suicide of patients in the test data set (non-NEN cohort). The AUC of this test data set was 0.690 (95% CI, 0.678-0.702; P < 0.001, Figure 4D). This suggests that this score has acceptable performance and is a stable predictive system for suicide among patients with NEN as well as among patients with other tumor entities. In order to determine the optimal cut-off of this score system, the Youden's index was determined by using the training data set. We observed that the optimal cut-off of the score is 10. This optimal cut-off had a sensitivity of 63.50% and a specificity of and 87.5%.

DISCUSSION
Suicide is regarded to be a serious public health problem, especially among cancer patients (8,14). Although, previous studies (7,8) have determined the incidence of suicide in several cancers within the SEER program, they did not evaluate the incidence of suicide in NEN. In this study, we observed that the PMR of NEN patients is significantly higher than non-NEN patients, especially when the cancer was located in the appendix or pancreas. This suggests that suicide may be more frequent in NEN patients than in non-NEN patients. We proved, for the first time, that NEN is an independent risk factor for suicide. This is supported by the observation that NEN increases the risk of suicide to 1.8-fold in direct comparison to non-NEN (Table 1). However, the underlying reasons why NEN patients more frequently commit suicide are still unclear. Park et al. (25) suggested that the Bcl-1 polymorphisms of neuron-specific glucocorticoid receptor  (NR3C1) may contribute to the susceptibility to suicide in cancer patients. This hypothesis is supported by the concept that dysregulation of NR3C1 leads to depression and suicide by disturbing the function of the hypo-thalamic-pituitary-adrenal axis (26,27). Consistent with previous studies (16,28,29), we observed that some features of cancer patients such as being young, male, a member of the white race, unmarried, or surgery as well as having cancer in the localized or regional stage increased the risk of committing suicide. However, in contrast to Osazuwa-Peters's study (15), we observed that patients whose tumor was located in lung and bronchus, stomach, or pancreas had a significantly reduced risk of committing suicide, when compared to patients whose tumor was located in colon. In Osazuwa-Peters's study, the authors defined the colon and rectal tumor as an entire cohort. Notably, Tamas et al. (30) suggests that the biological and clinical characters of colon cancer and rectal cancer are different. Hence, we split colorectal cancer into two distinct cohorts, and we observed that tumor located in rectum significantly increased the risk of suicide, when compared to tumor located in colon.
Currently, most cancer patients die of non-cancer causes, such as accidents, heart diseases, and suicide (29). However, the suicide among cancer patients can be directly prevented and several strategies have been established (13,31,32). For example, treatments of psychiatric disorders, screening programs for highrisk persons, education for the general public, and restricting access to lethal means (33). Indeed, some studies proved that physician education and restricting access to lethal means, such as guns, domestic gas and barbiturates, could reduce suicide rates (33,34).
Some screening questionnaires, such as nine-item patient health questionnaire (PHQ-9), hospital anxiety and depression scale (HADS), and Diagnostic and Statistical Manual of Mental Disorders (DSM), have been used for depression screening in cancer patients (13). In addition, the pan-Canadian guideline (35) and the American society of clinical oncology guideline (13) suggest that all patients with cancer should be screened for depressive symptoms at their initial visit and during the treatment until the end-of-life. Notably, most of these tools were originally used to detect depression and anxiety in the general population. They consequently did not include tumor-specific factors, such as tumor stage and grade as well as treatment variables. This may reduce the sensitivity and specificity of these screening tools. Indeed, Hartung et al. (36) demonstrated that the screening performance of both the PHQ-9 and the HADS was limited, when they were applied for depression screening in cancer patients. Thus, our score in combination with current screening tools might successfully identify cancer patients at high-risk of committing suicide.
Our study has two clinical implications: First, prevention of suicide in NEN patients is especially necessary. This conclusion bases on the observation that compared to non-NEN, NEN increases the risk of suicide by 1.8-fold. Thus, a routine screening among NEN patients, especially when the cancer is localized in the appendix or pancreas, may effectively reduce the suicide rate of these patients. In addition, raising awareness of physicians to this topic may be important for patients suffering from pancreatic neuroendocrine tumors (pNET). It is well-known that pancreatic ductal adenocarcinoma (PDAC) is a lethal disease, and the 5year survival of patients is only around 10% (37). However, the 5-year survival of pNET patients is 50% (3), and there is a high risk that pNET patients actually think they suffer from a PDAC. This may increase the depression and anxiety of these pNET patients. Indeed, the present study suggests that the pancreatic NEN patients have a more than 3-fold higher risk of suicide in comparison with non-NEN patients ( Figure 2C). These results imply that patients with longer survival might have an elevated risk to experience depression symptoms and succeed in their suicide attempts when compared to ones with FIGURE 4 | The predictive ability of the score. The receiver operating characteristic (ROC) curve, the area under the curve (AUC) and the confidence interval (CI) suggest a significant discriminatory power of the score when distinguishing between suicide patients from non-suicide patients (A). In addition, the discriminatory power of the score is significantly superior to single factors such as age, gender, race, marital status, chemotherapy, surgical procedure, cancer stage, and cancer grade (B). Moreover, the SEER score could distinguish suicide from non-suicide patients in the validation cohort (C), and the non-NEN cohort (D).
shorter overall survival. Thus, physicians' education may help these patients eliminating depression as well as anxiety, and thus reducing the suicide rate. Second, this study proved that the score (Table 4), which includes general population characteristics (age at diagnosis, gender, race, marital status), tumor characteristics (tumor stage and histologic grade), and treatment parameters (surgery, chemotherapy) can significantly distinguish individuals having committed suicide from those that did not. This supports the hypothesis that incorporation of this score into current screening strategies may prevent suicide among patients suffering from NEN.

LIMITATIONS
Notably, there are some limitations of our studies. First, even though SEER program is an authoritative source of information on cancer incidence and mortality, this database did not identify patients who committed unsuccessful suicide attempts. We, therefore, failed to calculate the incidence of suicide, and only determined the PMR of suicide among NEN cohort and non-NEN cohort. Second, we identified the participants of this study by the cause of death, however, we were unable to evaluate whether there was misclassification bias in our study. Previous studies suggest that suicide may be misclassified as accidents or unintentional death (38,39). Third, suicide is a complex behavior, and several factors, such as psychiatric disorders, occupation, childhood adversity, family history, and antidepressants can increase the risk of suicide (40,41). However, the SEER program did not provide these information, and we could not evaluate these factors in this study. In addition, although, this SEER scale has a probability of over 80% to correctly distinguish suicide patients from non-suicide individuals; the false negative rate is 36.5% and the false positive rate is 12.5% when considering 10 as the optimal cut-off point.

CONCLUSION
In conclusion, this study, for the first time, demonstrated that patients with NEN have a higher risk of committing suicide than patients with non-NEN. In addition, we developed and validated a suicide score, which can distinguish suicide and non-suicide individuals among these patients. This score may improve the current screening strategies and may reduce the suicide rate especially among patients with NEN.

DATA AVAILABILITY STATEMENT
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author/s.

AUTHOR CONTRIBUTIONS
XZ and PG co-supervised the study. LL wrote the manuscript. LL and YS analyzed and interpreted the data. XZ, LL, DZ, ML, and CM performed the critical revision and language check of manuscript. All authors contributed to the article and approved the submitted version.

SUPPLEMENTARY MATERIAL
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.

2021.638152/full#supplementary-material
Supplementary Figure 1 | The performance of predictive factors to distinguish between suicide patients from non-suicide patients. The receiver operating characteristic (ROC) curves, the area under the curve (AUC), and the confidence interval (CI) suggest that age (A), gender (B), chemotherapy (C), surgery (D), and stage (E) could significantly distinguish suicide patients from non-suicide patients, however, race (F), marital status (G), and grade (H) did not have this ability to discriminate.