The participants in the study were recruited from outpatients attending the Endocrinology Section of the Sichuan Provincial People’s Hospital. Participants were selected according to the following criteria: (1) over 18 years of age; (2) diagnosed as a T2D patient and received hypoglycemic treatment (the diagnostic criteria for T2D were in line with China’s 2017 guidelines on preventing and treating type 2 diabetes (Chinese Diabetes Society, 2017)); and (3) HbA1c levels were measured on the day of collection. (4) Researchers explained the purpose and scope of the survey to the subjects, and those who agreed to take part were retained in the study. Ethics approval was obtained through the Ethics Committee of the Sichuan Provincial People’s Hospital (approval # 2018-53).

Characteristics of participants were obtained from face-to-face interviews and electronic medical records (EMRs). The adherence status was defined according to the proportion of days covered (PDC). PDC higher than 80% was regarded as good medication compliance (Wu et al., 2020).

HbA1c values on the day of visiting the clinic were measured at the clinical laboratory of Sichuan Provincial People’s Hospital and collected from EMRs. In this study, a value of HbA1c more than 7.0% was defined as positive and less than 7.0% was defined as negative. Furthermore, parents who had a positive HbA1c were considered to be needed for detection on the day of attending the Endocrinology Section.

After data collection ended, the information was converted to an Excel format. Each column represented a candidate variable, and each row represented a sample. To acquire high-quality data for modeling, a series of interventions were performed, including data pre-screening, data imputing, and variable selection.

First, data pre-screening was carried out using the following criteria: (1) the columns with missing values > 90% were removed, (2) the columns with a single value occupying >90% were removed, and (3) the columns with the coefficients of variation <0.1 were removed.

Missing information was inevitable in clinical data, such as the FBG value and PBG value. Missing data were filled using four imputing methods, including simple imputing (marked as SI), random forest imputing (marked as RF), k-nearest neighbor imputing (marked as KNN), and optimal deletion (marked as OD).

In order to eliminate irrelevant variables, reduce the number of variables, and improve the accuracy of the model, variable selection was performed. In this study, four algorithms—LASSO (Tibshirani, 1997), ridge regression (Marquardt and Snee, 1975), ElasticNet (Simon et al., 2011), and Boruta (Kursa M B, 2010)—were used to screen the key variables. The four aforementioned algorithms were marked as LA, RD, EN, and BOR, respectively.

80 % of the data were assigned as the training set and the rest as the test set. The training set was used to train a classification model, and the test set was used to evaluate the model performance. Meanwhile, to assess whether the FGB value on the day was the important variable, the original data with the FGB value were used to train the models.

Sixteen datasets were generated in the training set by four data imputation methods and four variable selection methods. Then, six machine learning algorithms were employed on each dataset, respectively, to develop a total of 112 models. Machine learning algorithms in this study included random forest (RF), logistic regression (LR), multilayer perceptron (MLP), extreme gradient boosting (XGBoost), light gradient boosting machine (LGBM), and categorical boosting (CB).

RF, an ensemble learning algorithm proposed by Breiman, is very commonly used for classification (Breiman, 2001). Individual decision trees are built using a random subset of the training dataset in the training process. The final classification is then based on the majority voting results of all decision trees (Singha et al., 2019).

LR is widely used to solve binary classification problems (Jaillard et al., 2020). It predicts the probability of whether a dependent variable belongs to a particular class. The principle of LR is to first fit the decision boundary and then establish the probability relationship between the boundary and the classification so as to obtain the probability in the case of two classifications (Wang et al., 2020).

MLP, also known as a feed-forward neural network, is one of the most common deep learning approaches (Wan et al., 2018). It is mainly used to address supervised learning problems by learning the dependencies between the input layer (the variables) and output layer (the classification decision) using a fully connected hidden layer (Wang et al., 2020).

XGBoost (Chen and Guestrin, 2016), LightGBM (Guolin et al., 2017), and CatBoost (Bentéjac et al., 2021) were the three most popular implementations of gradient-boosting tree-based ensemble methods (Friedman, 2001). While built on structurally similar ideas, these libraries differ slightly on how decision trees are grown or how categorical variable data are handled, and only experimentation can validate which performs the best.

The test set was used for external validation. A confusion matrix was used to evaluate the accuracy of classifier classification. In this study, a confusion matrix combined with relevant economic indicators, renamed as model benefit (MB), was used to redefine model performance to evaluate the accuracy of classifier classification. The test fee for HbA1c at our hospital was ¥73 per test. The mean additional economic burden of therapeutic inertia (Lindvig et al., 2021) was regarded as the cost of missed detection. In accordance with the current exchange rate, the exchange rate of the renminbi (RMB, ¥) against the Danish Krone is approximated to 1:1. The cost of missed detection was ¥786.77. The calculation formula was as follows: M B ¥ = T o t a l   c o s t T C , ¥ − M o d e l   c o s t M C , ¥ , T C ¥ = T o t a l   p a r t i c i p a n t s × H b A 1 c   t e s t   f e e , M C ¥ = T P * × H b A 1 c   t e s t   f e e + F P * × H b A 1 c   t e s t   f e e + F N * × c o s t   o f   m i s s e d   d e t e c t i o n − T N * × H b A 1 c   t e s t   f e e , * : T P = t r u e   p o s i t i v e , F P = f a l s e   p o s i t i v e , F N = f a l s e   n e g a t i v e , T N = t r u e   n e g a t i v e . In addition, the area under the receiver operating characteristic curve (AUC), area under the precision recall curve (AUPRC), and decision curve analysis (DCA) were summarized to assess the model performance. The contribution of each variable to the predictive model was estimated with SHapley Additive exPlanation (SHAP).

The best model (assessed by MB) was employed to estimate the impact of sample sizes on the predictive performance (Wu et al., 2020). The total samples were randomly separated into the training set and the test set at a ratio of 8:2. First, 10% of the training set was extracted to train the model, and the AUC was evaluated in the test set. The selected samples from the training set increased from 10% to 100% with a stepwise increase of 10%. These steps were repeated 10 times to generate 10 independent repeated AUC values. The relationship of sample size with the prediction performance of models was assessed according to the inflection point change of the line graph. The steeper broken line indicated that a larger sample size would improve the prediction performance of the model, and the gentler slope indicated that the performance of models was affected a little by the sample size.

Above all, the concise workflow for the development and validation of models is summarized in Figure 1.

The continuous variables were described by mean and standard deviation, whereas categorical variables were expressed in terms of frequencies and percentages. Multivariate analyses were performed to identify the factors associated with the model performance. Multivariate analysis was performed by multi-linear regression analysis. Model development was performed using the sklearn package and SHAP package in Python (Python Software Foundation, Python Language Reference, version 3.6.8) on PyCharm (developed by JetBrains.r.o., version 11.0.4). The grid search technique was applied to calculate hyperparameter values optimally.

Overall, 980 patients completed the survey, among which 571 were male and 409 were female. The mean age was 59.2 ± 11.9 years. A total of 513 patients were defined as positive (52.3%). Participants were grouped according to the HbA1c value, and detailed characteristics of the participants are shown in Table 1.

After data pre-screening, 15 variables were removed (nationality, marital status, with or without complications, vascular complications, neurological complications, complications with lesions of the extremities, ocular complications, nephropathy complications, complications (other diseases), glinides, thiazolidinediones, GLP-1 Ras, SGLT2 inhibitors, SGLT2 inhibitors, and the use of Chinese medicine).

A total of 60 datasets were set up by applying different imputing methods and variable selection methods with 41 variables. The different numbers of variables and samples in each dataset are listed in Table 2.

A total of 192 models (whether they included FBG or not) were validated in the test set, considered as external validation, and the performance metrics were output. As shown in Table 3; Figure 2, the five best models (excluded FBG) were listed in sequence according to the MB value. The five best models were trained in the No. 12 dataset (applied modified random forest as the imputing method and Ridge as the selection method). The MB, AUC, accuracy, and AUPRC values of the best model (model 1) were 3163.800 (¥), 0.852, 0.811, and 0.845, respectively.

As listed in Table 4; Figure 3, the five best models (included FBG) were trained in No. 11 and No. 10 datasets (applied modified random forest as the imputing method and ElasticNet or LASSO as the selection method). The MB, AUC, accuracy, and AUPRC values of the best model (model 1) were 43475.750 (¥), 0.972, 0.944, and 0.974, respectively. The calibration and DCA curves also displayed excellent predictive performances (Figures 2C, D; Figures 3C, D). The model that included FBG produced superior forecast performance compared to the model that excluded the FBG value.

SHAP was used to interpret the results from the best model. The result of SHAP in the best model (excluded FBG) is shown in Figure 4. As shown in Figure 4A, SHAP evaluation quantifies the contribution of a feature in a single sample. As results in Figure 4B, previous HbA1c values, having a rational and reasonable diet, course of diabetes, BMI, interval of measurement, duration of treatment regimen, type of manufacturers of metformin, age, waistline, and marital status were the 10 most important variables.

The results of the contribution of variables in the best model (included FBG) are shown in Figure 5. As illustrated in Figure 5A, waistline, previous HbA1c values, interval of measurement, number of oral drugs, psychological status, EQ-5D scores, type of manufacturers of metformin, and FBG values provided a positive contribution to the SHAP value, while exercise session and course of diabetes provided a negative contribution. As presented in Figure 5B, the 10 most important variables were FBG values, previous HbA1c values, having a rational and reasonable diet, health status scores, type of manufacturers of metformin, interval of measurement, EQ-5D scores, occupational status, and age.

The adequacy of the sample size was verified using the resampling bootstrapping method, and the results are plotted in Figure 6. The AUC gradually increased and the dispersion of the AUC value decreased as the percentage of the sample size increased. When the sample size reached 60%, the curve flattened. The results indicated that the performance of the model might be slightly affected when expanding the sample size.

As shown in Table 5, the number of samples could significantly affect the model prediction performance, including MB, AUC, accuracy, precision, and recall (p < 0.01). We found that the number of variables could affect the MB and recall of the prediction model significantly (p < 0.01). MB and AUC were influenced by screening methods (p < 0.05). Precision was affected by imputing methods (p < 0.01).