A machine learning-based prediction of diabetic retinopathy using the Korea national health and nutrition examination survey (2008–2012, 2017–2021)

Abstract

Background:

Machine learning technology that uses available clinical data to predict diabetic retinopathy (DR) can be highly valuable in medical settings where fundus cameras are not accessible.

Objective:

This study aimed to develop and compare machine learning algorithms for predicting DR without fundus image.

Methods:

We used data from Korea National Health and Nutrition Examination Survey (2008–2012 and 2017–2021) and enrolled individuals aged ≥ 20 years with diabetes who received fundus examination. Predictive models for DR were developed using logistic regression and three machine learning algorithms: extreme gradient boosting, decision tree, and random forest. Model performance was evaluated using area under the receiver operating characteristic curve (AUC) and accuracy for the diagnosis of DR, and feature importance was determined using Shapley Additive Explanations (SHAP).

Results:

Among the 3,026 diabetic participants (male, 50.7%; mean age, 63.7 ± 10.5 years), 671 (22.2%) had DR. The random forest model, using 16 variables, achieved the highest AUC of 0.748 (95% confidence interval, 0.705–0.790) with a sensitivity 0.669, specificity of 0.729 and an accuracy of 0.715. As interpreted by SHAP, HbA1c, fasting glucose levels, duration of diabetes, and body mass index were identified as common key determinants influencing the model’s outcomes.

Conclusion:

The DR prediction models using machine learning techniques demonstrated reliable performance even without fundus imaging, with the random forest model showing particularly strong results. These models could assist in managing DR by identifying high-risk patients, enabling timely ophthalmic referrals.

Introduction

Diabetes mellitus is one of the major chronic diseases, and its prevalence is rapidly increasing worldwide. One of the severe complications of diabetes is diabetic retinopathy (DR), a leading cause of blindness among working-age adults worldwide (1). Globally, it is estimated that approximately one-third of patients with diabetes have some form of DR, with the prevalence ranging from 35 to 40% (2). In 2019, it was reported that around 146 million people were affected by DR, and this number is projected to increase to 191 million by 2030 (3). Early detection and intervention are crucial in preventing severe vision loss and improving the quality of life for diabetic patients, as DR often presents without symptoms even in late stages (4). However, non-adherence to DR examination is common among diabetic patients globally, and thus, a better way to screen high-risk patients for DR is required (5–8). Recently, the evolution of machine learning technology has sparked significant interest and gained popularity, particularly for enhancing clinical decision-making (9). Machine learning techniques enable clinicians to leverage complex datasets to predict outcomes such as disease progression, treatment response, and patient outcomes with high accuracy and efficiency (10, 11). In medical settings without access to a fundus camera, machine learning technology that utilize available clinical data to predict DR can be highly beneficial.

This study aims to develop a machine learning model to predict the risk of DR in diabetic patients using clinical variables from Korea National Health and Nutrition Examination Survey (KNHANES) data to provide an efficient and accurate diagnostic support system for DR in clinical settings.

Materials and methods

Study design and participants

This study was conducted using the KNHANES data (2008–2012 and 2017–2021) (12). Based on self-reported questionnaire-based information, the participants who answered, “yes” to the question “Have you ever been diagnosed with diabetes by a doctor before?” were classified as diabetic patients. Participants were selected based on the following criteria: age over 20, diagnosed with diabetes, having fundus examination results, and no missing data in the variables used for analysis. The final study sample included 3,026 participants. To develop and evaluate a predictive diagnostic model for DR, the sample was split into a training set (2,420 participants) and a test set (606 participants) in an 8:2 ratio (Figure 1). DR was identified based on the presence of any characteristic lesions defined by the Early Treatment Diabetic Retinopathy Study severity scale using a non-mydriatic fundus camera. Grading was performed by experienced retinal specialists, with DR diagnosed in the presence of microaneurysms, hemorrhages, hard exudates, cotton wool spots, intraretinal microvascular abnormalities, venous beading, or retinal neovascularization (12). DR grading was not additionally assessed in all years, whereas the presence or absence of DR was determined.

FIGURE 1

Model development and performance evaluation

We developed prediction models using logistic regression and three machine learning algorithms: extreme gradient boosting (XGB), Decision Tree (DT), and random forest (RF). These algorithms employ various approaches to map predictor values to probabilities between zero and one. The evaluation of predicted performance was carried out using the area under the receiver operating characteristic curve (AUC) along with 95% confidence intervals (CI) for each prediction model (13). A comparison was made between the performance of the logistic regression and the machine learning models. Additionally, performance was measured using sensitivity, specificity, positive predictive value, and negative predictive value, all of which were determined based on a single cut-off value designed to maximize the Youden index (calculated as sensitivity + specificity - 1). Model calibration was assessed by comparing the observed event probabilities with the predicted ones.

To enhance the random forest model’s performance, we applied hyperparameter tuning using RandomizedSearchCV, a method that efficiently samples a wide range of hyperparameter combinations to identify the optimal settings. This approach is particularly advantageous when dealing with large datasets or numerous hyperparameters, as it reduces computational cost compared to exhaustive methods like GridSearchCV.

For the random forest model, we tuned hyperparameters including the number of trees (n_estimators: [100, 150, 173, 200, 250, 600]), maximum tree depth (max_depth: [10, 15, 20, None]), minimum samples to split a node (min_samples_split: [2, 4, 6, 10]), minimum samples at a leaf (min_samples_leaf: [1, 2, 4, 6]), features considered for splitting (max_features: [’auto’, ‘sqrt’, ‘log2’]), sampling method (bootstrap: [True, False]), and splitting criterion (criterion: [’gini’, ‘entropy’]). The optimal parameters identified were n_estimators = 173, max_depth = None, min_samples_split = 6, min_samples_leaf = 4, max_features = ’sqrt’, bootstrap = True, and criterion = ’entropy’.

For the Logistic Regression model, hyperparameter optimization focused on the regularization strength (C: np.logspace(−4, 4, 20)), penalty type (penalty: [’l1’, ‘l2’] for Lasso and Ridge regularization), and solver algorithm (solver: [’liblinear’, ‘saga’]). These parameters helped improve model generalization and prevent overfitting.

For the Decision Tree model, the grid search evaluated the split criterion (criterion: [’gini’, ‘entropy’]), split strategy (splitter: [’best’, ‘random’]), maximum depth (max_depth: [None, 5, 10, 15, 20]), minimum samples for node splitting (min_samples_split: [2, 5, 10]), minimum samples at leaf nodes (min_samples_leaf: [1, 2, 4]), and features for splitting (max_features: [’sqrt’, ‘log2’]). These settings enhanced model complexity control and interpretability.

For the XGBoost model, tuning focused on the learning rate (learning_rate: [0.1, 0.01, 0.001]), tree depth (max_depth: [3, 5, 7]), minimum child weight (min_child_weight: [1, 3, 5]), training data subsampling (subsample: [0.8, 0.9, 1.0]), features used per tree (colsample_bytree: [0.8, 0.9, 1.0]), and boosting rounds (n_estimators: [100, 200, 300]). The grid search approach optimized the model’s ability to learn complex patterns while maintaining generalization.

Feature selection techniques

Recursive Feature Elimination with Cross-Validation (RFECV) was used with 5-fold cross-validation. RFECV is a robust method for feature selection that iteratively removes the least important features based on model performance until the optimal subset is found. By using 5-fold cross-validation, the model’s performance is validated across different subsets of data, reducing the risk of overfitting. This approach ensures that only the most relevant features contribute to the predictive model, enhancing both accuracy and interpretability.

Feature Analysis using SHapley Additive exPlanations (SHAP)

SHAP interprets machine learning models by showing feature importance and their impact on predictions. It uses values from game theory to quantify each feature’s contribution to the model’s output. This helps data scientists and stakeholders understand and trust the model, enhancing transparency and enabling model refinement (14).

We used a broad spectrum of patient characteristics and health indicators as variables in our analysis, including age, sex, duration and treatment of diabetes mellitus, smoking status, comorbidities, waist circumference, systolic blood pressure, diastolic blood pressure, body mass index (BMI), and laboratory results.

Statistical analysis

The continuous variables were analyzed using the independent t-test, and the categorical variables were analyzed and compared by the Chi-squared or Fisher’s exact tests. All statistical analysis was conducted with Python (version 3.12.2). The statistical significance criterion was set to be two-sided, and p-values < 0.05 were considered statistically significant.

Ethical adherence

This study was reviewed and approved by the Institutional Review Board (IRB) of Seoul National University Bundang Hospital (IRB No: X-2212-796-901). Informed consent was waived by the IRB due to the anonymized data and retrospective design of the study. The study followed the principles of the Declaration of Helsinki.

Results

A total of 3,026 diabetic participants aged ≥ 20 years were included in the analysis. All the participants had fundus examination results, with a prevalence of DR of 22.2% (n = 671). The baseline characteristics of the training and test datasets are shown in Table 1.

The predictive performances and test characteristics of the models derived using logistic regression analysis and machine learning models are summarized in Table 2. In the validation using the test dataset, RF demonstrated the highest predictive performance achieving an AUC of 0.748 (95% CI, 0.705–0.790) with a sensitivity 0.669, specificity of 0.729 and an accuracy of 0.715. The AUC was 0.744 (95% CI, 0.701–0.788) in the logistic regression model, 0.731 (95% CI, 0.684–0.776) in the XGB model, and 0.651 (95% CI, 0.597–0.702) in the DT model (Figure 2).

FIGURE 2

As depicted in Figure 3 and Table 3, key features for prediction across all models were HbA1c levels, fasting blood glucose levels, duration of diabetes mellitus, and BMI among the top eight variables in each model. HbA1c (SHAP value, 0.044; effects, 0.183) was the most influential factor in the RF model outcomes, followed by fasting glucose levels (SHAP value, 0.043; effects, 0.180), duration of diabetes mellitus (SHAP value, 0.040; effects, 0.168), insulin usage (SHAP value, 0.018; effects, 0.074), age (SHAP value, 0.016; effects, 0.068), and other factors.

FIGURE 3

Discussion

This study aimed to develop an optimized machine learning model to predict the diagnosis of DR among adults with diabetes using the KNHANES data, a nationwide survey database representative of the South Korean population.

In this study, we developed various models for the prediction of DR diagnosis and compared the performance of XGB, DT, and RF models with that of the conventional logistic regression model. Among the models developed for predicting DR diagnosis using machine learning algorithms, the RF model showed the best performance.

Previous studies have developed deep learning systems that can detect and classify DR using fundus images. Dai et al. reported an average AUC of 0.955 for DR grading, while Bhimavarapu et al. achieved 99.41% accuracy using fundus images (15, 16). Fundus image-based DR prediction offers the advantage of high accuracy by directly capturing retinal pathology, enabling precise assessment of DR diagnosis and severity. However, in many regions, access to regular fundus imaging is limited due to various challenges (6–8). Additionally, AI-based fundus image analysis faces issues such as inconsistent image quality, poor pupil dilation, patient compliance, and suboptimal acquisition techniques, all of which require specialized equipment and trained personnel (17, 18). In such cases, identifying patients in need of DR screening using easily obtainable clinical data becomes clinically important. Our model leverages clinical information to facilitate early DR screening without requiring specialized equipment or personnel, making it more practical for routine clinical use. Furthermore, clinical variables are readily available and cost-effective. However, a limitation of our model is that its accuracy is lower compared to fundus imaging-based DR prediction. Additionally, the model’s performance may vary depending on the availability of the clinical variables included in the analysis.

There have also been several non-image-based machine learning studies for DR prediction. Islam et al. conducted a study using clinical information from a Chinese cohort and achieved an accuracy of 90.01% with an XGBoost-based model (19). Similarly, Zhao et al. validated a DR prediction model using baseline demographic and clinical characteristics in patients with type 2 diabetes mellitus, with the XGBoost model demonstrating the highest predictive performance (accuracy: 88.9%) (20). These two studies likely achieved higher accuracy due to their larger sample sizes compared to our study, because similar studies with relatively smaller sample sizes have reported results comparable to ours (accuracy, 73.5–79.5%) (21–23).

The potential integration of our optimized RF model into clinical decision support systems (CDSS) used by primary care physicians, such as those in internal medicine and family practice, offers several notable advantages. By utilizing relatively stable predictive outcomes, a CDSS can enhance diabetes management through more accurate and targeted referrals for ophthalmological evaluation. This could lead to significant reductions in healthcare costs by preventing unnecessary consultations and focusing resources on those most in need of specialized care (24, 25). Additionally, timely and accurate referrals can significantly improve patients’ quality of life by preventing vision loss and enabling better management of their condition (26, 27). This advantage could be maximized if this system is applied to large-scale populations, such as those in corporations, schools, and the military.

This study has several limitations. First, DR grading was not additionally assessed in all years, and only the presence or absence of DR was determined, making subanalysis based on DR grading not possible. Additionally, this model was confirmed an Asian ethnic group, requiring generalizability. To address these limitations, future research should include larger, more diverse patient groups and utilize detailed datasets to develop models that can more accurately identify individuals in need of hospital referrals for DR.

However, this study also has several strengths. First, it utilizes nationwide community-based data to ascertain the prevalence and risk factors of DR. This data provides a representative sample of the South Korean population. Second, the information was collected, examined, and interpreted using standardized protocols, ensuring objectivity and reliability of the data. Third, we developed an optimized model for DR risk assessment using various algorithms, enhancing the predictive accuracy and applicability of our findings. Our approach is convenient for the use in hospitals during routine health check-ups or diabetes management visits even without fundus examination. Furthermore, this method could effectively complement existing fundus photography-based models, providing a comprehensive and resource-efficient diagnostic tool for DR. As the next step, we have developed software to integrate this model into practical clinical use, and it is currently in the commercialization preparation stage.

Conclusion

The prediction models for DR with machine learning techniques using nationwide survey data have the potential to be utilized as CDSS. Our results suggest that these models can aid in the management of DR by identifying patients at high risk, thereby facilitating timely ophthalmic referrals. Consequently, we believe this approach will be able to significantly contribute to the prevention of vision loss through improved DR risk management in people with diabetes.

References

• 1.

  Na K Lee W Kim Y . Trends in the prevalence of blindness and correlation with health status in Korean adults: A 10-year nationwide population-based study.J Korean Med Sci. (2023) 38:e213. 10.3346/jkms.2023.38.e213

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2.

  Lee R Wong T Sabanayagam C . Epidemiology of diabetic retinopathy, diabetic macular edema and related vision loss.Eye Vis (Lond). (2015) 2:17. 10.1186/s40662-015-0026-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3.

  Ting D Cheung G Wong T . Diabetic retinopathy: Global prevalence, major risk factors, screening practices and public health challenges: A review.Clin Exp Ophthalmol. (2016) 44:260–77. 10.1111/ceo.12696

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4.

  Yau J Rogers S Kawasaki R Lamoureux E Kowalski J Bek T et al Global prevalence and major risk factors of diabetic retinopathy. Diabetes Care. (2012) 35:556–64. 10.2337/dc11-1909

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5.

  Kim M Park S Joo K Woo S . Trends and barriers in diabetic retinopathy screening: Korea National health and nutritional examination survey 2016-2021.J Korean Med Sci. (2024) 39:e203. 10.3346/jkms.2024.39.e203

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  Altman D Jimenez C Hsu S Hudson T Luong J Fong DS . Factors influencing patient adherence to diabetic retinopathy screening and follow-up: An exploratory qualitative study.ARVO Annu Meeting. (2019) 60:1022.

  • Google Scholar

• 7.

  Fathy C Patel S Sternberg P Kohanim S . Disparities in adherence to screening guidelines for diabetic retinopathy in the United States: A comprehensive review and guide for future directions.Semin Ophthalmol. (2016) 31:364–77. 10.3109/08820538.2016.1154170

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  Liu Y Zupan N Shiyanbola O Swearingen R Carlson J Jacobson N et al Factors influencing patient adherence with diabetic eye screening in rural communities: A qualitative study. PLoS One. (2018) 13:e0206742. 10.1371/journal.pone.0206742

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9.

  Chun J Kim H . The present and future of artificial intelligence-based medical image in diabetes mellitus: Focus on analytical methods and limitations of clinical use.J Korean Med Sci. (2023) 38:e253. 10.3346/jkms.2023.38.e253

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10.

  Obermeyer Z Emanuel E . Predicting the future - big data, machine learning, and clinical medicine.N Engl J Med. (2016) 375:1216–9. 10.1056/NEJMp1606181

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11.

  Beam A Kohane I . Big data and machine learning in health care.JAMA. (2018) 319:1317–8. 10.1001/jama.2017.18391

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12.

  Song S Choi K Han J Jee D Jeoung J Jo Y et al Methodology and rationale for ophthalmic examinations in the seventh and eighth Korea national health and nutrition examination surveys (2017-2021). Korean J Ophthalmol. (2021) 35:295–303. 10.3341/kjo.2021.0016

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13.

  Park S Yeo N Kang S Ha T Kim T Lee D et al Early prediction of mortality for septic patients visiting emergency room based on explainable machine learning: A real-world multicenter study. J Korean Med Sci. (2024) 39:e53. 10.3346/jkms.2024.39.e53

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14.

  Liu C Chen Y Sifan H Chen X Liu F . Assessing the determinants of corporate risk-taking using machine learning algorithms.Systems. (2023) 11:263. 10.3390/systems11050263

  • CrossRef
  • Google Scholar

• 15.

  Dai L Wu L Li H Cai C Wu Q Kong H et al A deep learning system for detecting diabetic retinopathy across the disease spectrum. Nat Commun. (2021) 12:3242. 10.1038/s41467-021-23458-5

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  Bhimavarapu U Battineni G . Deep learning for the detection and classification of diabetic retinopathy with an improved activation function.Healthcare (Basel). (2022) 11:97. 10.3390/healthcare11010097

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 17.

  Ahsan M Luna S Siddique Z . Machine-learning-based disease diagnosis: A comprehensive review.Healthcare (Basel). (2022) 10:541. 10.3390/healthcare10030541

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18.

  Bellemo V Lim Z Lim G Nguyen Q Xie Y Yip M et al Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: A clinical validation study. Lancet Digit Health. (2019) 1:e35–44. 10.1016/S2589-7500(19)30004-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 19.

  Islam M Rahman M Rabby M Alam M Pollob S Ahmed N et al Predicting the risk of diabetic retinopathy using explainable machine learning algorithms. Diabetes Metab Syndr. (2023) 17:102919. 10.1016/j.dsx.2023.102919

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20.

  Zhao Y Li X Li S Dong M Yu H Zhang M et al Using machine learning techniques to develop risk prediction models for the risk of incident diabetic retinopathy among patients with type 2 diabetes mellitus: A cohort study. Front Endocrinol (Lausanne). (2022) 13:876559. 10.3389/fendo.2022.876559

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  Tsao H Chan P Su E . Predicting diabetic retinopathy and identifying interpretable biomedical features using machine learning algorithms.BMC Bioinformatics. (2018) 19:283. 10.1186/s12859-018-2277-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22.

  Ogunyemi O Kermah D . Machine learning approaches for detecting diabetic retinopathy from clinical and public health records.AMIA Annu Symp Proc. (2015) 2015:983–90.

  • Google Scholar

• 23.

  Oh E Yoo T Park E . Diabetic retinopathy risk prediction for fundus examination using sparse learning: A cross-sectional study.BMC Med Inform Decis Mak. (2013) 13:106. 10.1186/1472-6947-13-106

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24.

  O’Reilly D Holbrook A Blackhouse G Troyan S Goeree R . Cost-effectiveness of a shared computerized decision support system for diabetes linked to electronic medical records.J Am Med Inform Assoc. (2012) 19:341–5. 10.1136/amiajnl-2011-000371

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 25.

  Huang S Liang Y Li J Li X . Applications of Clinical Decision Support Systems in diabetes care: Scoping review.J Med Internet Res. (2023) 25:e51024. 10.2196/51024

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 26.

  Zayed M Karsan W Peto T Saravanan P Virgili G Preiss D . Diabetic retinopathy and quality of life: A systematic review and meta-analysis.JAMA Ophthalmol. (2024) 142:199–207. 10.1001/jamaophthalmol.2023.6435

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27.

  Clark K Gutierrez J Cody J Padilla B . Implementation of diabetic retinopathy screening in adult patients with type 2 diabetes in a primary care setting.Clin Diabetes. (2024) 42:223–31. 10.2337/cd23-0032

  • Pubmed Abstract
  • CrossRef
  • Google Scholar