Predicting the severity of white matter lesions among patients with cerebrovascular risk factors based on retinal images and clinical laboratory data: a deep learning study

Background and purpose As one common feature of cerebral small vascular disease (cSVD), white matter lesions (WMLs) could lead to reduction in brain function. Using a convenient, cheap, and non-intrusive method to detect WMLs could substantially benefit to patient management in the community screening, especially in the settings of availability or contraindication of magnetic resonance imaging (MRI). Therefore, this study aimed to develop a useful model to incorporate clinical laboratory data and retinal images using deep learning models to predict the severity of WMLs. Methods Two hundred fifty-nine patients with any kind of neurological diseases were enrolled in our study. Demographic data, retinal images, MRI, and laboratory data were collected for the patients. The patients were assigned to the absent/mild and moderate–severe WMLs groups according to Fazekas scoring system. Retinal images were acquired by fundus photography. A ResNet deep learning framework was used to analyze the retinal images. A clinical-laboratory signature was generated from laboratory data. Two prediction models, a combined model including demographic data, the clinical-laboratory signature, and the retinal images and a clinical model including only demographic data and the clinical-laboratory signature, were developed to predict the severity of WMLs. Results Approximately one-quarter of the patients (25.6%) had moderate–severe WMLs. The left and right retinal images predicted moderate–severe WMLs with area under the curves (AUCs) of 0.73 and 0.94. The clinical-laboratory signature predicted moderate–severe WMLs with an AUC of 0.73. The combined model showed good performance in predicting moderate–severe WMLs with an AUC of 0.95, while the clinical model predicted moderate–severe WMLs with an AUC of 0.78. Conclusion Combined with retinal images from conventional fundus photography and clinical laboratory data are reliable and convenient approach to predict the severity of WMLs and are helpful for the management and follow-up of WMLs patients.


Introduction
Stroke is a leading cause of mortality and long-term disability, especially in low-and middle-income countries (1). In China, among 28.76 million prevalent cases of stroke in 2019, and most of the cases were ischemic stroke (2). Cerebral small vascular disease (cSVD), including white matter lesions (WMLs), is a major challenge to brain health, accounting for approximately 30% of the cause of ischemic stroke (3). It was estimated that 36% of cSVD cases were due to WMLs (4).
The severity of WMLs is negatively associated with cognitive function, especially in the patients of Fazekas 3 and 4 WMLs (5,6). In addition, WMLs was also association with new-onset depression in older people in the community and the risk of post-stroke depression and dysfunction of bowel and bladder (7). WMLs are associated with poorer gait and balance (8). And WMLs are a strong predictor of the incident stroke (9). The severity of WMLs progressed over time. The Leukoaraiosis and Disability Study showed that about 73.6% of the participants had WMLs progression during 3 years follow-up and among the patients with severe WMLs, more than 80% of them had WMLs progression (10). Thus, regular screening is crucial for the early identification of WMLs and the prevention of its progression. Magnetic resonance imaging (MRI) has been widely used to detect WMLs in vivo (11). However, MRI is not available for all the candidates who needed WMLs screening or diagnosis due to insufficient economic resource, unavailability of MRI equipment, contraindication of MRI and so on. Convenient substitution of MRI to assess WMLs is helpful for patient management and economic efficiency.
It's worth noting that the brain and retinal vasculature are homology (12). More specifically, the blood-retinal barrier mirrors the blood-brain barrier. In addition, the microcirculation of the brain and retina share a similar autoregulation function to maintain blood flow (13). The retinal microcirculation has a similar embryonic, anatomical, and physiological basis to that of small intracranial vessels (14,15). Thus, the retina is a window through which vasculature and neural tissues can be dynamically and non-invasively observed. Retinal blood vessel lesions have been found to be independently associated with WMLs (16)(17)(18).
Significant progress in deep learning classifiers has been achieved in medical imaging fields, such as radiology, dermatology, pathology and ophthalmology (19). Among various deep learning techniques, convolutional neural networks make automatic, efficient and accurate image-based diagnosis possible (20,21). Specifically, deep learning models based on retinal images rival ophthalmologists when screening diabetic retinopathy (22). Moreover, there is early evidence that identification of central nervous system diseases based on retinal images is possible. Recently, texture characteristics of retinal images discovered by automated retinal image analysis have been found to predict WMLs in community participants (23). A deep learning algorithm has been applied to raw retinal images was put forward to predict WMLs (24). However, it is worth noticing that the capability to predict WMLs of these retinal images-based models needs further improvement. Thus, a comprehensive model that not only includes retinal images but also these clinical data need to be developed. A recent study developed a comprehensive model combined a deep learning model based on retinal images and clinical information, which accurately predicted chronic kidney disease and type 2 diabetes (25). Concerning that hypertension (26), hyperlipidemia (27), impaired glucose metabolism (28), kidney failure (29) and systemic inflammation (30) are risk factors of WMLs, incorporating the clinical information of these risk factors into the retinal images-based prediction model may improve the capability of prediction and enhance the clinical practicability of the model. To the best of our knowledge, no studies have been conducted to develop a deep learning model using retinal images combined with the clinical laboratory data to predict the severity of WMLs.
Therefore, based on the previous findings that retina and its vasculature share similarity with cerebral small vessels, we aimed to develop a comprehensive model including retinal images and clinical laboratory data to predict WMLs severity, which may facilitate screening WMLs patients and follow-up.
We first built up a ResNet deep learning neural network to analyze correlation between the retinal images and the severity of WMLs. Then, we fused a clinical-laboratory signature based on the clinical laboratory data. A model combining demographic-clinical characteristics, the clinical-laboratory signature, and the outputs of the retina deep learning neural network was further developed to predict the severity of WMLs. To test the significance of the retinal images on predicting the severity of WMLs, the clinical model only included demographic-clinical characteristics and the clinicallaboratory signature. The performance of predicting WMLs severity of the two models was compared.

Patient selection
The clinical documentations of patients admitted to the Department of Neurology, the First Affiliated Hospital of Sun Yat-sen University, from January 2018 to January 2020 were retrospectively reviewed. The inclusion criteria were as follows: (1) patients who underwent nonmydriatic fundus photography; (2) patients who underwent brain MRI; and (3) over 18 years old. The exclusion criteria were as follows: (1) poor quality nonmydriatic fundus photography or MRI images; (2)
WMLs were analyzed on T2WI and T2-FLAIR images and semiquantitatively graded based on the Fazekas grading system. Periventricular hyperintensity (PVH) was scored as 0 for the absence of any lesions, 1 for cap or pencil-like lesions, 2 for halo-like lesions with diameters of 6 mm ~ 10 mm, and 3 for irregular lesions with diameters of more than 10 mm. Deep white matter hyperintensity (DWMH) was scored as 0 for the absence of any lesion, 1 for punctatelike lesions, 2 for small confluent lesions, and 3 for large confluent lesions ( Figure 1). For patients with significant brain edema induced by ischemic stroke or intracerebral hemorrhage, the Fazekas score was defined by the presence of a WML in the contralateral hemisphere. The scores of PVH and DWMH were further combined to represent the overall WMLs severity (31). The summation of the PVH score and DWMH score is the final Fazekas score that ranged from 0 to 6. The final Fazekas score was determined by two vascular neurologists (LS and KZ), and any discordant results between the two observers were resolved by consensus. Thus, the numbers of the patients of each final Fazekas score were as follow: final Fazekas score 0 (n = 81), final Fazekas score 1 (n = 62), final Fazekas score 2 (n = 51), final Fazekas score 3 (n = 29), final Fazekas score 4 (n = 16), final Fazekas score 5 (n = 8), and final Fazekas score 6 (n = 12). To minimize the redundancy and maximize the clinical practicability of our model, we assigned the patients with final Fazekas score 0 to 2 in the absent/mild WMLs group and the patients with final Fazekas score 3 to 6 in the moderatesevere WMLs group.

Retinal images acquisition
Fundus photography was performed on both eyes for all patients without pupil dilatation (at a KOWA nonmyd7, Japan). In brief, The transverse (A) and coronal (B) sections of MRI of Fazekas scale to semi-quantitatively quantify the WMLs severity. For deep white matter hyperintensity (DWMH): punctate foci, score 1; small confluences, score 2; large confluences, score 3. For periventricular hyperintensity (PVH): caps or pencil-thin linings, score 1; smooth halos with diameters of 6 mm 10 mm, score 2; irregular lesions with diameters of more than 10 mm, score 3. DWMH, deep white matter hyperintensity; PVH, periventricular hyperintensity.
Frontiers in Neurology 04 frontiersin.org patients sat in front of the fundus photography system and were instructed to focus on the center of the viewing system while the fundus was examined. The field of view was set to 45° to enable visualization of the posterior portion of the examined eye. The position of the camera and focal length were finely adjusted so that the optic disk and macula were clearly viewed. The images of both eyes were stored in a workstation for further analysis ( Figure 2).

Deep learning neural network for retinal images
We used ResNet-18 model as the training model to predict the present moderate-severe WMLs in brain MRI scan by using retinal images. ResNet-18 is a residual network pretrained using the ImageNet dataset, but the associated fully connected softmax layer has been replaced by a supervised classifier. The pre-trained model used in our study is available on the code repository. 1 We further modified this pre-trained model for our study. The convolution kernel of the convolutional layer is 7 × 7. The max pooling layer was directly performed by a convolutional layer with stride 2, followed by batch normalization after each convolution operation and before the ReLU activation function. The image was resized to 224 × 224 to match the input image size required by ResNet-18. Then we used a linear classifier to perform feature scaling on the 2048-dimensional output feature vector of the last convolutional layer for dimensionality reduction, and 1 https://download.pytorch.org/models/resnet18-f37072fd.pth set the final output layer dimension as 2. Each network includes three main modules: the input module, the output module, and the intermediate convolution module. We used the cross-entropy loss function as objective function. The optimizer was Adam, learning rate was 0.001. Finally, the deep-learning model produced a single value ranged 0 to 1 for each retinal image. This value indicates the probability of moderate/severe WMLs that was predicted by this deep-learning model and we took this value for further analysis. The class activation maps for each retina were generated to visualize the weight of predicting WMLs in the sub-regions of retina. The class activation maps for each retinal images were generated by the Grad-CAM algorithm.
Frontiers in Neurology 05 frontiersin.org and albumin/globulin ratio (A/G ratio). The renal function regimen, including creatine and uric acid (UA).
We developed a clinical-laboratory signature representing the metabolism, systemic inflammation, liver and renal function. A least absolute shrinkage and selection operator (LASSO) logistic regression model was implemented to eliminate collinearity, reduce overfitting, filter and select the clinical laboratory data that correlate with moderate-severe WMLs by tuning the hyperparameters that minimize Akaike information criteria (AIC).

Development of the WMLs severity prediction model
We developed a combined model including demographic-clinical characteristics, the clinical-laboratory signature, and the outputs of the retina deep learning neural network to predict WMLs severity.

Statistics
Data for continuous variables are summarized as the mean and standard deviation. Categorical variables were summarized by number and percentage. Univariate logistic regression was carried out in R (v4.0.5) to explore the relationship among demographic-clinical characteristics, individual clinical laboratory data and WMLs severity. The continuous variables were compared by independent-sample t test between patients in the absent/mild WMLs group and patients in the moderate-severe WMLs group. The categorical variables were compared by the chi-square test or Fisher's exact test. Univariate logistic regression was used to explore the association between clinicallaboratory indices and dichotomized WMLs severity. Odds ratios (ORs) and 95% confident intervals (95% CIs) were used to predict the association between clinical-laboratory data and WMLs severity. The predictive model development and compilation of the clinicallaboratory signature were conducted in Python (v 3.7) and the scikitlearn package. 2 The performance of the clinical-laboratory signature, the retinal images, the combined model, and the clinical model was comprehensively predicted by the receiver operating characteristic curve. The area under curve (AUC), accuracy, precision, recall, F1 score, sensitivity, and specificity were calculated. The coefficient of determination (R 2 ) and recall rate were further utilized to predict the performance of the clinical-laboratory signature, and the clinical mode.

Patient clinical characteristics
Two hundred fifty-nine patients were enrolled in our study. The mean age of all patients was 52.3 ± 13.5 years. More than half of the included patients were male. The mean SBP of all patients was 140.4 ± 24.8 mmHg. The mean DBP of all patients was 87.7 ± 15.4 mmHg. The most common comorbidity was hypertension, which was present in 56.8% of all patients. Current smoking (35.1%), statin usage (21.6%), diabetes mellitus (17.4%), alcohol abuse (14.7%) and previous ischemic stroke/TIA (18.5%) were also common among all patients. A small proportion of the patients had atrial fibrillation (2.3%) and coronary heart disease (3.5%). Furthermore, the clinical characteristics of the patients with absent/mild and moderate-severe WMLs were compared. The patients with moderate-severe WMLs were significantly older than those with absent/mild WMLs (p < 0.001). The SBP and DBP on admission were higher in the patients with moderate-severe WMLs than in those with absent/mild WMLs (p < 0.001). The proportions of hypertension, previous ischemic stroke/TIA, and atrial fibrillation were higher in the patients with moderate-severe WMLs than in those with absent/mild WMLs, while the frequency of diabetes mellitus was higher in the patients with absent/mild WMLs. The clinical characteristics of all patients are summarized in Table 1. The diagnoses of our patients of each WMLs group were demonstrated in the Supplementary Table S2. The majority of our patients were diagnosed with cerebral infarction (65.8% for the mild/absent WMLs group, 75.8% for the moderate-severe WMLs group). The proportions of cerebral infarction were not significantly different between the mild/absent and moderate-severe WMLs groups (p = 0.127; Supplementary Table S2). The number of other diagnoses was very small. Hence, the results of the chi-square test for these diagnoses were not reliable. To avoid mistranslating the results, we did not present the p values of these diagnoses.

The laboratory signature and the clinical model
This clinical-laboratory signature included globulin, creatine, Apo-A1, and TG to predict moderate-severe WMLs with a determination R 2 of 0.14. This signature showed an AUC of 0.73, accuracy of 0.77, precision of 0.85, recall rate of 0.18, F1 score of 0.29, sensitivity of 0.18, specificity of 0.95 (Figure 3; Supplementary Table S1).
The clinical model combined with the clinical-laboratory signature and the demographic data including age, sex, ranked SBP, and ranked DBP. This model predicted moderate-severe WMLs with a determination R 2 of 0.17, an AUC of 0.78, accuracy of 0.76, precision of 0.58, recall rate of 0.30, F1 score of 0.40, sensitivity of 0.30, specificity of 0.93 (Figure 3; Supplementary Table S1).

The combined model for predicting the severity of WMLs
The combined model included age, sex, ranked SBP, ranked DBP, clinical-laboratory signature, the outputs of retina deep learning neural network based on the left and right retinal image ( Table 3). The combined model could well recognize moderate-severe WMLs with an AUC of 0.95, accuracy of 0.90, precision of 0.85, recall rate of 0.79, F1 score of 0.82, sensitivity of 0.79, specificity of 0.95 (Figure 3; Supplementary Table S1).

Discussion
To the best of our knowledge, this is the first study to use the combination of retina image, clinical features and laboratory data to develop a comprehensive model for predicting the severity of WMLs. The WMLs is a major threat to public health and early detection and routine follow-up are critical for controlling advance of WMLs and preventing deterioration of cognitive function. WMLs can be induced by various pathological mechanisms. Among all subtypes of WMLs, arteriosclerotic WMLs (namely, age-and vascular risk factor-related WMLs) are the most common in clinical practice (32). We excluded patients with autoimmune diseases, intoxication, metabolic brain disease, and hereditary WMLs. The patients with moderate-severe WMLs were significantly older and had higher SBP and DBP on admission. Previous studies have shown that advanced age and elevated blood pressure are significant causes of arteriosclerosis WMLs (11). Furthermore, elevated homocysteine, a well-known risk factor for arteriosclerosis (11), was found to be higher in the patients with moderate-severe WMLs, which is consistent with previous findings that homocysteine levels are positively correlated with the volume of arteriosclerotic WMLs (33)(34)(35). Hence, the WMLs investigated in the current study was further indicated to be of the arteriosclerotic subtype.
The WMLs significantly impair cognitive function (36) and are risk factors of ischemic stroke (37) and gait dysfunction (38). ResNet has been applied to predict Alzheimer's disease (39), diagnose myocardial infarction (40), analyze blood cells (41), diagnose COVID-19 (42), and diagnose malignant tumor (43) and tumor metastasis (44). In addition to the aforementioned research areas, ResNet deep learning models can also be used to detect retinal exudative lesions (45). The ResNet-18 is a robust deep learning network, where the convolutional layers are pre-trained and can start training the network without a large dataset. Therefore, this residual network overcomes the limitation of using deep learning for medical image studies with small datasets. Among all the models, the right retinal images and the combined model well predicted moderate-severe WMLs and outperformed the clinical model and clinical-laboratory signature. Furthermore, the deep learning network based on the right retinal images alone shared similar performance with the combined model in which the retina image was combined with clinical information. Our results were in line with the previous studies that reflect the pathophysiological basis that the retina shares similar vasculature and vascular risk factors (46-50). Previous studies have shown that chronic kidney disease (51) and dyslipidemia (11) were associated with WMLs. Similarly, in our study, the clinical-laboratory signature including certain laboratory indices of lipid metabolism (Apo-A1, TG) and kidney function (creatine) can predict the severity of WMLs. Thus, in our study, the abovementioned pathophysiological basis may explain the good performance of the comprehensive model combining deep learning neural network in retinal images with clinical laboratory data.
Traditionally, the retinal vascular lesion was manually evaluated according the presentations of exudates, vessel tortuosity, arteriovenous nicking and so on. However, this evaluation method was affected by raters' bias and not quantitative. In our study, we focused on the entire retina image instead of the traditional evaluation method. We developed the deep learning model to analyze the entire image of the retina and show good AUC to predict WMLs. Our result further confirmed the findings of previous studies that the retinal lesions were associated with WMLs. In addition, the regression coefficient of retina deep learning model was higher than the traditional risk factors such as age and blood pressure in the combined model. This result indicated that the retinal images can predict the severity of WMLs.
Several limitations of this study should be mentioned. First, the WMLs severity was dichotomized as absent/mild or moderate-severe according to the Fazekas score in our study. Consequently, the models developed in our study could not quantitatively predict WMLs severity. Recent studies showed that severe WMLs were association with multiple neurological  Frontiers in Neurology 08 frontiersin.org diseases such as cognitive deterioration (6,52,53) and ischemic stroke (37), while few study reported the association of mild WMLs and other neurological diseases. Thus, our model showed clinical significance in predicting moderate-severe WMLs. Second, our study was hospital-based and retrospectively designed. Some of retinal images were excluded due to the poor quality. The number of the eligible left retinal images included in our analysis was less than the right, which may explain that the inconformity of the performance of the deep learning neural network between left and right retinal images. Further studies may be needed to explore the potential difference of the predicting performance between left and right retinal images. On the other hand, the AUC of right retina image was high in our study, which indicates that the retinal image and deep learning network are capable of predicting the severity of WMLs.

Conclusion
In conclusion, we integrated demographic-clinical characteristics, clinical laboratory data and deep learning on retinal images to develop a comprehensive model. This comprehensive model accurately predicted the severity of WMLs and offered non-invasive, highthroughput and low-cost screening tool for early detection of moderate-severe WMLs in the population in community, resourcepoor area or with contraindication of MRI.

Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement
The studies involving human participants were reviewed and approved by Ethics Committee of the First Affiliated Hospital of Sun Yat-sen University. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.