Development and validation of multivariable prediction models of serological response to SARS-CoV-2 vaccination in kidney transplant recipients

Repeated vaccination against SARS-CoV-2 increases serological response in kidney transplant recipients (KTR) with high interindividual variability. No decision support tool exists to predict SARS-CoV-2 vaccination response to third or fourth vaccination in KTR. We developed, internally and externally validated five different multivariable prediction models of serological response after the third and fourth vaccine dose against SARS-CoV-2 in previously seronegative, COVID-19-naïve KTR. Using 20 candidate predictor variables, we applied statistical and machine learning approaches including logistic regression (LR), least absolute shrinkage and selection operator (LASSO)-regularized LR, random forest, and gradient boosted regression trees. For development and internal validation, data from 590 vaccinations were used. External validation was performed in four independent, international validation cohorts comprising 191, 184, 254, and 323 vaccinations, respectively. LASSO-regularized LR performed on the whole development dataset yielded a 20- and 10-variable model, respectively. External validation showed AUC-ROC of 0.840, 0.741, 0.816, and 0.783 for the sparser 10-variable model, yielding an overall performance 0.812. A 10-variable LASSO-regularized LR model predicts vaccination response in KTR with good overall accuracy. Implemented as an online tool, it can guide decisions whether to modulate immunosuppressive therapy before additional active vaccination, or to perform passive immunization to improve protection against COVID-19 in previously seronegative, COVID-19-naïve KTR.

Repeated vaccination against SARS-CoV-2 increases serological response in kidney transplant recipients (KTR) with high interindividual variability. No decision support tool exists to predict SARS-CoV-2 vaccination response to third or fourth vaccination in KTR. We developed, internally and externally validated five different multivariable prediction models of serological response after the third and fourth vaccine dose against SARS-CoV-2 in previously seronegative, COVID-19-naïve KTR. Using 20 candidate predictor variables, we applied statistical and machine learning approaches including logistic regression (LR), least absolute shrinkage and selection operator (LASSO)regularized LR, random forest, and gradient boosted regression trees. For development and internal validation, data from 590 vaccinations were used. External validation was performed in four independent, international validation cohorts comprising 191, 184, 254, and 323 vaccinations, respectively. LASSOregularized LR performed on the whole development dataset yielded a 20-and 10-variable model, respectively. External validation showed AUC-ROC of 0.840, 0.741, 0.816, and 0.783 for the sparser 10-variable model, yielding an overall performance 0.812. A 10-variable LASSO-regularized LR model predicts vaccination response in KTR with good overall accuracy. Implemented as an online tool, it can guide decisions whether to modulate immunosuppressive therapy before additional active vaccination, or to perform passive Introduction SARS-CoV-2 vaccination offers protection from severe coronavirus disease 2019 (COVID-19) regardless of the causative variant for most healthy individuals. (1) In contrast, protection in immunocompromised solid organ transplant (SOT) recipients is limited. The serological response rate after SARS CoV-2 vaccination in kidney transplant recipients (KTR) after three doses of vaccine is strongly impaired in comparison to the general populationresulting in insufficient protection and an unacceptably high COVID-19 mortality within this population (2,3).
Different strategies to induce humoral protection for KTR have been suggested, including repeated vaccination and vaccination under adjusted immunosuppressionbesides SARS-CoV-2-specific monoclonal antibody therapy (4). Existing data are helpful to identify factors associated with insufficient vaccination response, but are not easily interpretable for the single patient or vaccination (5)(6)(7). Specifically, no tool exists to predict individual response to a vaccination. Risk calculators can help assess the likelihood of vaccination success in an individual and help decide between different possible actions such as passive or active immunization or adjustment of immunosuppressive medication. To date, no such decision support system is available.
For this reason, we aim to develop a classification model to predict serological response to third and fourth SARS-CoV-2 vaccinations in previously seronegative, COVID-19-naïve KTR. The model's implementation objective is to identify patients that will likely not respond to an additional dose of vaccine, even with changes in immunosuppressive medication, and thus benefit most from passive immunization strategies. Using our previously reported data of vaccination outcomes in KTR, we develop and compare a set of prediction models based on classical statistical methods as well as machine learning. After selecting the most promising models, we validate the resulting prediction models in four independent validation cohorts, and make the result available as an online calculator.

Development cohort
Data from KTR at Charité-Universitätsmedizin Berlin, Germany, were used to form the development cohort. Details of the underlying patient population, as well as the assays and cutoffs used have been previously reported (5). Briefly, KTR received up to five doses of SARS-CoV-2 vaccine in case of sustained lack of sufficient serological response to vaccination at our institution, combined with either maintenance, reduction or pausing mycophenolic acid (MPA) for fourth and fifth vaccination. For the enzyme-linked immunosorbent assays (ELISA) for the detection of IgG antibodies against the S1 domain of the SARS-CoV-2 spike (S) protein in serum (Anti-SARS-CoV-2-ELISA (IgG), EUROIMMUN Medizinische Labordiagnostika AG, Lübeck, Germany), samples with a cutoff index ≥ 1.1 (in comparison to the previously obtained cut-off value of the calibrator) were considered positive, samples with a cutoff index ≥ 0.8, and < 1.1 were considered low positive, and samples with a cutoff index <0.8 were considered negative, as suggested by the manufacturer.
Alternatively, for the electrochemiluminescence immunoassay (ECLIA) (Elecsys, Anti-SARS-CoV-2, Roche Diagnostics GmbH, Mannheim, Germany) detecting human immunoglobulins, including IgG, IgA and IgM against the spike protein receptor binding domain (RBD), samples with ≥ 264 U/ml were considered to be positive as recommended by Caillard et al. (8,9) Any non-zero antibody level below this cutoff was considered low positive, with limit of detection (LoD) being 0.4 U/mL. For predictive modeling, we included data on third and fourth vaccination, since basic immunization has most likely been performed in most KTR patients already, and since only few patients received fifth vaccination so far.

Validation cohorts
We used four independent, international validation cohorts from outpatient transplant centers at University Hospital Düsseldorf, Germany (191 vaccinations in 137 KTR) (10,11), Medical University Vienna, Austria (184 vaccinations in 184 KTR) (12), Strasbourg University Hospital, France (254 vaccinations in 229 KTR) (13,14), Hotel Dieu Nantes, France (323 vaccinations in 269 KTR) (15). Detailed information about the validation cohorts are presented in Items S1-Item S4 and patient selection including outcome frequencies are summarized separately for each validation cohort in Figures S1-S4. No sample size calculation was applicable for this post-hoc analysis.

Outcome and predictors
The single outcome variable was a positive serological response defined by the maximum anti-SARS-CoV-2 spike (S) IgG or antibody level, measured at least 14 days after vaccination and before any further immunization event such as SARS-CoV-2 infection, passive or active immunization. Since different assays were used at different sites, details on the tests and the respective cutoffs used are provided for each validation cohort in Item S1-S4, which are summarized in Table 2. Generally, IgG or antibody positivity was determined based on the local laboratory's positivity cutoff, mostly the one provided by the manufacturer. Especially for the ECLIA Elecsys assay different cutoffs were available and used. We chose to assess model performance for two cutoffs for this specific assay. First, we used the 0.8 U/mL cutoff provided by the manufacturer, yielding highest sensitivity in detecting patients with previous COVID-19. Second, a cutoff of 15 U/mL, which was initially suggested by the manufacturer to exhibit a positive predictive value of more than 99% for presence of neutralizing antibodies against the wild-type virus, was used (12). Contrary to the manufacturer's designated use, our intention was to provide an alternative positivity cutoff, below which no neutralization against omicron variant occurs, but that is not as close to the LoD (0.4 U/mL) as the positivity cutoff provided by the manufacturer (0.8 U/mL). This alternative positivity cutoff definition was needed to test the hypothesis that the absence or low number of "low-positive" antibody levels before vaccination (below the positivity cutoff, but above the LoD) for this assay led to low performance in validation sets 2 and 4. While the cutoff of 15 U/ mL is somewhat arbitrary, it meets both needs. First, it increases the percentage of low positive patients in validation set 4, and second, patients with antibody levels <50 U/mL in this assay show no neutralization against omicron BA.1, which most likely applies to omicron BA.2 as well (16,17). Hence, adjusting the  Frontiers in Immunology frontiersin.org cutoff to 15 U/mL is compatible with the objective to identify patients without serological response to an additional vaccine dose corresponding best with a lack of neutralizing antibodies. Predictor variables in the data sets comprised 20 variables: four vaccination-specific, three demographic, one comorbidity, three transplantation-specific, five encoding medication, and four biomarkers (Table S1). From the initial 27 candidate predictor variables, seven were excluded during revision for the following reasons: treatment with azathioprine, mTOR inhibitor or rituximab in the last year were removed as predictor variable, since each variable was present in less than 5 subjects in the development dataset. Donor-specific anti-HLA antibodies were removed since they highly depend on mismatch status and medication adherence in the past. Anti-HB-S antibodies were removed since they depend on hepatitis B vaccination status, which was not available for most patients in the development cohort. Treatment with mycophenolic acid (MPA) was removed as predictor variable, since MPA dose contained the same information and was already used. White blood cell count was removed, since it is presumably less suitable to predict vaccination response than lymphocyte count, which was already a predictor variable (Table S2).

Missing data/imputation
For the development dataset, preliminary analysis showed that neither using data from patients without lymphocyte count, Patient flow diagram of the development cohort.
Osmanodja et al. 10.3389/fimmu.2022.997343 Frontiers in Immunology frontiersin.org which was the most common missing laboratory value, nor imputation of missing laboratory values by multiple imputation (both of which yielded higher sample size) did add predictive accuracy for logistic regression and was therefore not followed for the main analysis ( Figure S5; Table S3). After applying all exclusion criteria shown in Table 1, no missing values were present in the development dataset and no imputation methods were necessary. For each validation set, we excluded vaccinations with missing data on serological response, missing information about the SARS-CoV-2 spike IgG or antibody assay used, missing immunosuppressive medication data, or missing estimated glomerular filtration rate (eGFR), lymphocyte count, or hemoglobin. We imputed the remaining variables to reduce the number of omitted cases due to missing values. Instead of multiple imputation, we used a more pragmatic approach and imputed either the most frequent value of the respective variable in the development dataset in case of binary or categorical variables, or the median (or mean) of the respective variable in the development dataset in case of numerical variables, as summarized in Table 3. This is the way a clinician would handle a missing value when using the online risk calculator, since those values are used as presets in the online calculator. In the validation cohorts, no data originating from a time after the respective vaccination was included to make predictions.
Alternatively, we performed multiple imputations by chained equations employing the R package mice after pooling all validation datasets. Performing multiple imputation separately for each dataset was unfeasible, since for validation set 2, no data on BMI, time on dialysis and diabetes status were present. Pooling all validation sets and performing multiple imputation hereafter was one possibility to avoid this problem.
To compare median/mean imputation to other possibilities to deal with missing data, we additionally performed complete case analysis for validation sets 1, 3, and 4.

Development and internal validation
Using the development cohort, we evaluated five models during internal validation. To perform model validation within the development cohort, a resampling approach was used by assigning 590 vaccinations randomly 100 times into training and test sets of 413 and 177 each (70:30 split). Each time, hyperparameter tuning, if applicable, and model fitting was performed on the respective training set, and performance metrics were assessed on the respective test set.
First, as baseline, we fit a logistic regression model with all candidate variables using the R function glm.
Second, we fit 2 logistic regression models with least absolute shrinkage and selection operator (LASSO) regularization using the packages caret and glmnet in R. The LASSO hyperparameter l, which adjusts the tradeoff between model fit and model sparsity, was optimized for each training cohort with respect to the area under the receiver operating curve (AUC-ROC) using inner 5-fold cross-validation. We chose 2 different l optimization criteria yielding 2 different models for each training cohort: (1) maximizing AUC-ROC (termed LASSO-Min model), and (2) penalty maximization while keeping the AUC-ROC within one standard error of the maximum AUC-ROC (termed LASSO-1SE model).
Third, we fit a random forest regression model using the package randomForest in R. We optimized the hyperparameter mtry by evaluating 15 random parameter combinations during two repeated 5-fold cross-validations within the training set. The value of mtry yielding the highest accuracy during crossvalidation was used to fit the random forest on the respective training data.
We calculated median and mean performance during resampling for those five developed models. To evaluate the performance of the binary classification, we used Area Under the Curve of the Receiver Operator Characteristic (AUC-ROC), and confidence intervals (CI) in the resampling approach were determined from the empirical 2.5% and 97.5% quantiles of the performance on the 100 different test sets. Based on the threshold determined by the optimization criterion "closest.topleft" as provided in R package pROC (point with the least distance to [0,1] on the ROC-curve) during ROCanalysis, we calculated models' sensitivity, specificity, accuracy, positive predictive value, and negative predictive value for each resampling step, again yielding median and empirical 95% CI.

External validation and implementation
We chose LASSO-Min and LASSO-1SE for estimation of model coefficients in the entire development cohort, which were then used for external validation. The relationship between the hyperparameter l that controls model sparsity and the AUC-ROC during inner 5-fold cross-validation is shown in Figure S6. We assessed the decision thresholds for classification by determining the "closest.topleft" threshold on the entire development cohort, each for the final 10-variable and the 20variable model. These were used for classification during external validation, and are also provided in the online risk calculator after transforming them into risk probabilities according to the formula: For external validation, we calculated the aforementioned performance metrics on each validation cohort separately. Furthermore, 95% CIs in the external validation cohorts were determined by performing 1000-fold ordinary nonparametric percentile bootstrap, as the empirical 2.5%, and 97.5% quantiles of AUC, sensitivity, specificity, accuracy, positive predictive value, and negative predictive value based on the thresholds determined within the development cohort.
Additionally, we fitted LR, RF and GBRT on the development dataset and performed external validation after pooling all validation datasets. Decision thresholds for LR and GBRT were determined within the development cohort as described above, and decision threshold for RF was 0.5.
To make the prediction models publicly available, we created an online tool implementing the LASSO logistic regression models used for external validation, which can be assessed at https://www.tx-vaccine.com. For patients who meet one or more of the exclusion criteria, the risk calculator should not be used.
Statistical analysis was performed using R studio v.1.2.5042 and R version 4.1.2 (2021-11-01). The underlying code was made available at https://github.com/BilginOsmanodja/txvaccine. The datasets can be made available on request from the corresponding author.
This article was prepared according to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement and we provided a checklist in the supplement (18).

Results
In total, 590 vaccinations (411 third vaccinations, and 179 fourth vaccinations) were used for development and internal validation, which is summarized together with outcome frequencies and reasons for exclusion in Figure 1.
Baseline characteristics of patients in the development and validation datasets including summary statistics of all variables are shown in Table 3.

Model development and internal validation
Using the resampling approach outlined above, we fit five different models on each training set and evaluated their performance on the respective unseen test set during 100 resampling steps.
A logistic regression model employing all candidate variables served as a baseline. Using the two different l optimization criteria outlined in "Methods", the LASSO-Min and LASSO-1SE models were fitted. Additionally, two tree-based machine-learning approaches were studied -random forest (RF) and gradient boosted regression trees (GBRT).
LASSO logistic regression selected in the majority of resampling runs 20, and 10 out of 20 potential predictors to yield the LASSO-Min and LASSO-1SE models, respectively. The regression coefficients, their variances, and the selection frequency of the predictors are shown in Figure 2 and Figure S7. Figure 3 compares AUC-ROC of the 5 models on the unseen test sets during 100 resampling steps, and Table 4 summarizes mean, and median performance metrics as well as 95% confidence intervals determined from empirical 2.5% and 97.5% quantiles during internal validation. Thresholds for binary classification were determined on the respective test set during each resampling step by performing ROC-analysis. With

Model specification
Final risk equations were obtained by fitting LASSO-Min, and LASSO-1SE models on the complete development dataset, yielding a 20-variable and one 10-variable risk equation respectively. The intercept and regression coefficients of the final LASSO logistic regression models are shown in Table 5. Risk equations are provided in Items S5 and S6, and are implemented as an online tool available at https://www.txvaccine.com.

External validation
After applying all exclusion criteria and performing imputation of missing variables, we evaluated both risk equations in the four independent validation datasets. Since predictive performance during external validation was comparable for both models, in the following we report on the sparser 10-variable model. Results of external validation of the 20-variable model are summarized in Table S4 and Figure S7.  (Figure 4). Sensitivity, specificity, accuracy, positive predictive value, and negative predictive value using the thresholds determined during ROC-analysis in the development dataset are summarized in Table 6. The decision thresholds used for external validation are also provided in the online risk calculator to guide physicians' decision as well in Items S5 and S6.

Implementation and cutoff definition
Performance in the validations sets 2 and 4 was poorer than in the development as well as in the other two validation sets. We suspected the positivity cutoff of 0.8 U/mL provided by the manufacturer for the ECLIA Elecsys assay as one main reason. Since it is close to the LoD (0.4 U/mL), no or small fraction of "low positive" antibody levels (values above the LoD and below positivity cutoff) before vaccination are present in both validations sets (Table 3), which is different to both other validation sets and the development dataset. Since a low positive antibody level before vaccination is an important predictor of serological response (Table 5), we adjusted the positivity cutoff to 15 U/mL arbitrarily for two reasons. First, to test the hypothesis that cutoff definition is a reason for lower performance. Second, to provide data that an implementation of this model is feasible independent of the assay used. Our proposed implementation strategy for the prediction model is to identify patients, who will not respond to an additional vaccine dose, and to offer those patients passive immunization. Hence, using any other cutoff below which no neutralization against omicron occurs, is compatible with this strategy under the circumstances of omicron-dominance. We arbitrarily use an alternative positivity cutoff of 15 U/mL for this respective assay, since it has already been proposed by the manufacturer before.
When adjusting the cutoff to 15 U/mL for the ECLIA Elecsys assay, AUC-ROC increased to 0.741 (0.663 -0.808) for validation set 2, and 0.783 (0.730 -0.828) for validation set 4, yielding an overall AUC-ROC of 0.812 (0.784 -0.836) after merging all validation sets. With the decision threshold assessed in the development dataset, the negative predictive value is 0.75 (0.713 -0.784).
Neither complete case analysis nor multiple imputation in the pooled validation cohort led to relevant differences in predictive performance for the 20-variable and 10-variable LASSO LR models ( Table 6; Table S4).

Tree-based models
Next, we assessed model performance of all 5 models in the pooled validation set, using both the cutoff of 0.8 U/mL and 15 U/  Table 7. Analysis of feature importance for the tree-based models revealed that for the RF model, low positive antibody titer, MPA dose, belatacept treatment, vaccination number, and transplant age were the five most important variables, while diabetes, time since vaccination and sex were the least important variables. For the GBRT model, MPA dose, low positive antibody titer, eGFR, lymphocyte count and transplant age were the most important variables, while diabetes, CNI treatment, and mRNA-based vaccine were the least important variables. When comparing the 10 most important variables from RF or GBRT to the LASSO-1SE model, RF as well as GBRT had 8 out of 10 variables in common with the 10-variable LASSO-1SE model ( Table 8).   To assess how variable selection based on feature importance influences model performance, we selected the 10 most important variables for both RF and GBRT, retrained the models and performed external validation in the pooled validation dataset. Both, the 10-variable RF and the 10-variable GBRT yielded the same AUC-ROC during external validation as the respective 20-variable models -0.823 (0.795 -0.849) for GBRT, and 0.801 (0.771 -0.828) for RF.

Discussion
In this article, we present the development, internal and external validation of a 10-variable LASSO regularized logistic regression model for prediction of serological response to the third and fourth dose of SARS-CoV-2 vaccine in previously seronegative, COVID-19-naïve KTR.
It shows good discrimination of KTR exhibiting serological response both in a rigorous resampling approach in the development cohort and in four independent validation cohorts with an overall AUC-ROC of 0.812, and a negative predictive value of 0.75 based on a decision threshold established within the development dataset. Available online as a risk calculator at https://www.tx-vaccine.com and embedded into the proposed implementation strategy, it can assist physicians in choosing between different immunization strategies, namely, additional vaccination with or without adaption of immunosuppressive therapy, or pre-exposure prophylaxis with monoclonal anti-SARS-CoV-2-(S) antibodies.
While this is the first, online available risk calculator to predict seroconversion in response to third and fourth v accination, t here are al read y models predict ing seroconversion after two vaccine doses.
Frölke et al. describe a sparse 6-variable model, where increased age, lower lymphocyte count, lower estimated glomerular filtration rate (eGFR), shorter time after transplantation, not using steroids and the use of mycophenolate mofetil/mycophenolic acid (MMF/MPA) are predictors of non-seroconversion (with a cutoff of 10 BAU/ mL). This is completely in line with our own findings. The performance in the development (n=215) and rather small validation cohort (n=73) are promising (AUC-ROC 0.83 and 0.84, respectively). In a larger, second validation cohort, for which an adapted model without lymphocyte count was used, the performance drops to AUC-ROC 0.75, which emphasizes that lymphocyte count is an important predictor (19). Still, this model is easy to use, and shows apparently more stable performance than the other model available, which is provided by Alejo et al. They use a gradient boosting algorithm, and identified mycophenolate mofetil (MMF) use, shorter time since transplant, and older age as strongest predictors of nonseroconversion, which is in line with our findings as well. Since the model shows good predictive performance on the development dataset (AUC-ROC 0.79), but poor performance in an external validation cohort (AUC-ROC 0.67), it can be suspected that the model is overfitted (20). This is further supported when using the online tool the authors provide at http://www.transplantmodels.com/covidvaccine/, where small changes, e.g. in patient age, show great changes in vaccine response probability. Another reason for worse performance could be that not only kidney transplant recipients are included. Therefore, not only is the cohort more heterogeneous, but important predictors such as eGFR are missing.
Our own data show that GBRT can achieve comparable performance in internal and external validation, when overfitting is limited by hyperparameter tuning within the development cohort. Nevertheless, since the GBRT model did not substantially outperform LASSO logistic regression models, but is more complex and less transparent, we chose not to implement the GBRT model in the online calculator. These  (21). From a biomedical point of view, serological response is only one half of immune response to vaccination and is complemented by T-cell response. However, neutralizing anti-SARS-CoV-2-(S) antibodies are pathophysiologically and epidemiologically established to offer protection from severe disease (9,22), which is also supported by the protection offered by monoclonal antibodies against SARS-CoV-2 applied for prophylaxis and treatment (23,24).
Yet, after the emergence of the omicron variants, neutralization antibody levels against omicron variant show 25.7-fold to 58.1-fold reduction in sera of healthy vaccinated subjects in comparison to wild-type (25). Consequently, antibody levels that ensure neutralization, increased from >264 U/mL for alpha variant (8,9) to >2000 U/mL for omicron (16), making the interpretation of antibody levels more difficult than before.
Despite these uncertainties, it seems intolerable to leave patients without any humoral protection whatsoever. Hence, in patients without serological response to basic immunization, physicians and patients need to decide between additional active vaccination with or without adapting immunosuppressive medication, and pre-exposure prophylaxis with monoclonal antibodies (26).
Since negative predictive value was above 0.75, when merging all validation sets, we suggest the following implementation strategy: for patients, who are likely not to respond to additional SARS-CoV-2 vaccination according to the prediction model, pre-exposure prophylaxis with monoclonal antibodies exhibiting neutralizing capacity against omicron BA.4/5 should be administered to ensure timely protection (27)(28)(29)(30). In patients, who are likely to respond according to the prediction model, there is still a chance that these patients will not reach antibody levels, which ensure neutralizing capacity against omicron variants. For these patients, both, repeated vaccination and monoclonal antibody prophylaxis are feasible and should in our view be chosen depending on the risk for severe disease course.

Strengths and limitations
We provide a rigorously developed and validated prediction model, which is provided as an online risk calculator to support kidney transplant physicians when deciding upon the immunization strategy for their patients. Especially, the estimated effects of adaptions in immunosuppressive medication can be evaluated, e.g. reducing or pausing MPA dose, or switching from belatacept to calcineurin inhibitor. While belatacept treatment and MPA dose have been reported as negative predictors of serological vaccine response throughout the literature (10), it is still unclear, whether modulation of immunosuppression, and especially pausing MPA can increase serological response to SARS-CoV-2 vaccination, since most data originate from observational or small non-randomized controlled trials (5,(31)(32)(33).
Since the data from the development cohort indicate that pausing MPA increases serological response to fourth vaccination, this hypothetical benefit is introduced into the Additionally, predictions can be suspected to be less accurate in cohorts, where no modulation of immunosuppression are performed around fourth vaccination.
Regarding the general sparsity of data on vaccine response to third and fourth dose in KTR, we analyze extensive datasets for  development and validation, and hereby provide representative evaluation of real-life model performance. While performance of the 20-variable model was slightly better, it is also more impractical due to more variables, which is why we chose to report mainly on the sparser 10-variable model. The same applies for the 20-variable GBRT model, which was not implemented in the online calculator. When compared to other models reported for this purpose, it is the first to predict serological response to third and fourth vaccination, and shows the most promising performance during external validation. Several limitations have to be considered: first, this model only predicts serological response and does not include information about T-cell response. However, we have shown before that more than 85% of KTR have SARS-CoV-2 specific CD4+ T-cell response after three vaccinations, which was not increased by fourth vaccination, while serological response rates increase with Predictive performance of the 10-variable, 20-variable LASSO logistic regression, logistic regression (LR), random forest (RF) and gradient boosted regression tree (GBRT) models on the pooled validation set comprised of all 4 validation sets with cutoff 0.8 U/mL and 15 U/mL for the Elecsys assay. 10-var -10-variable LASSO LR model, 20-var -20-variable LASSO LR model, allall validation sets pooled.  additional vaccinations (4,5). This is the rationale, why especially serological response rate can and should be increased to improve protection from SARS-CoV-2 infection in KTR.
Since evidence of antibody level cutoffs that ensure neutralization of or protection from omicron is sparse, we chose not to make any predictions for this endpoint. Instead, we provide an implementation strategy that makes best use of the model's prediction without making far-reaching assumptions about protective antibody levels against omicron.
Still, one major limitation becomes evident for validation sets 2 and 4, where predictive performance was only moderate when using the positivity cutoff of 0.8 U/mL for the ECLIA Elecsys assay. As outlined above, increasing the positivity cutoff to 15 U/mL is compatible with the proposed implementation, and leads to improved performance for two reasons.
First, since the cutoffs in the development dataset were based on protective levels against alpha variant for the ECLIA Elecsys assay (264 U/mL), predictive performance is expectably poorer when predicting positivity with a 0.8 U/mL cutoff. Second, with a cutoff of 0.8 U/mL for the ECLIA Elecsys assay, only few patients have low positive antibody levels before vaccination (above LoD, but below the positivity cutoff). Since this is an important predictor in both models and provides important information about the actual immunological status, loss of performance can be expected when this information is missing. This is further supported by the fact that after adapting the cutoff from 0.8 U/mL to 15 U/mL, in validation set 2, where all pre-vaccination antibody levels were below 0.4 U/mL, the performance only increased slightly (AUC-ROC 0.719 to 0.741), but in validation set 4, where the percentage of low positive patients increased from 14.6% (most of which were due to the other assays used in this dataset) to 33.1%, the performance increased markedly (AUC-ROC 0.696 to 0.783).
On the contrary, since the percentage of low positive antibody levels in the development dataset is 6.8%, performance in cohorts, where this rate is close to 100%, could potentially worsen as well.
Other possible reasons for different performance are the study design of validation set 2, which was a randomized clinical trial, with outcome assessment between days 29 and 42, whereas in the development and other validation sets, the maximum antibody level after the respective vaccination was chosen, independent of the time passed. Additionally, validation set 2 was the one with the highest proportion of adenoviral vaccine, which however, did not show any difference in serological response in the respective trial (12).
validation is comparable to real-life performance of the risk calculator provided. We show that neither performing complete case analysis nor multiple imputation in the validation cohorts substantially changes predictive performance. Other limitations arise from the different immunization strategies used, which lead to different seroconversion rates and have influence on model performance as well.
Last, some immunosuppressive regimens have low frequency below 1% in the development cohort (such as rituximab, mTOR inhibitor, and azathioprine treatment), which limits applicability for these patients.

Conclusion
We provide the first, online available calculator to predict vaccine response to third or fourth vaccination in previously seronegative, COVID-19-naïve KTR. It can guide decisions whether to modulate immunosuppressive therapy before additional active vaccination, or to perform passive immunization to improve protection against COVID-19.

Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement
The studies involving human participants were reviewed and approved by Ethics Comittee of CharitéUniversitätsmedizin Berlin. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions
BO and SR conceived of the presented idea. BO performed data analysis and implemented the risk calculator. SR performed data visualization and project coordination. FG and MM assisted in data preparation of the development dataset. JS, MK, LCR, AH, RR-S, RO, IB, SC, CM, CK, and GB provided validation datasets. BO and SR wrote the manuscript. KB, ES, JS, MK, AH, RR-S, IB, SC, and CM provided significant intellectual input during the conception and development of the article. All authors commented and reviewed the final manuscript.