Multimodal behavioral phenotyping for depressive-spectrum classification and severity estimation using eye tracking, facial behavior, and transcript-derived language

Abstract

Introduction:

Depression assessment remains largely dependent on symptom reports and clinician judgment, while objective tools for depressive-spectrum stratification and severity estimation remain limited. Existing digital and multimodal depression-detection studies often focus on binary case-control classification, handle missing modalities incompletely, provide limited calibration assessment, and rarely combine depressive-spectrum classification with continuous symptom-severity estimation. We therefore developed a quality-aware multimodal framework integrating eye tracking, facial behavior, and transcript-derived language for classification across normal control (NC), subthreshold depression (SD), and major depressive disorder (MDD), together with prediction of 17-item Hamilton Depression Rating Scale (HAMD-17) severity.

Methods:

A total of 186 participants completed a controlled task battery including interview, emotional reading, free viewing with verbal description, fixation, gaze orienting, smooth pursuit, prosaccade, and antisaccade tasks. Eye-tracking, facial-video, and transcript-derived language data were converted into modality-specific features. Baseline-3 combined modality-specific encoders, quality-aware gated fusion, and joint classification-regression learning under a nested repeated-resampling framework with explicit missing-modality handling. Baseline-3+ further incorporated Transformer-based cross-modal interaction and uncertainty-based dynamic task weighting. Performance was evaluated on held-out outer-loop test sets after temperature scaling. Interpretability analyses included gate profiling, selective prediction, SHAP, Integrated Gradients, and counterfactual analysis.

Results:

Baseline-3+ showed the most favorable classification and calibration profile, with accuracy, balanced accuracy, and F1-macro approaching 0.90 across both classification routes and lower expected calibration error than Baseline-3. For severity estimation, the improvement was route-dependent and mainly reduced the regression disadvantage observed under the hierarchical route. Misclassification was concentrated near the SD boundary. Interpretability analyses showed stable quality-aware modality reweighting, with facial features providing the dominant signal, complemented by eye tracking and smaller but meaningful language contributions.

Discussion:

This framework addresses key limitations of prior binary and incompletely calibrated depression-detection models by jointly supporting depressive-spectrum classification, severity estimation, missing-modality handling, calibrated prediction, and individual-level interpretation. Its most plausible role is to augment clinical assessment, particularly for boundary states such as SD.

1 Introduction

Major depressive disorder (MDD) is a leading cause of disability worldwide and remains a major contributor to population disease burden (1). Despite its clinical importance, diagnosis still relies largely on symptom report and clinician judgment, and objective tools for screening, stratification, and longitudinal monitoring remain limited (2–6). Recent work in digital psychiatry and computational phenotyping has therefore focused on behavioral and physiological signals that may complement conventional assessment, although clinically transferable biomarkers for depression remain difficult to establish (7, 8). This challenge is especially relevant across the depressive spectrum. Subthreshold depression (SD) is more prevalent than syndromal MDD in many populations, is associated with functional impairment and elevated risk of progression, and is not adequately captured by symptom thresholds alone (9–12). Available evidence suggests that SD is not simply a milder form of MDD, but a heterogeneous intermediate state with variable symptom burden, psychosocial dysfunction, and transition risk. This heterogeneity also contributes to diagnostic ambiguity in data-driven models.

Behavioral markers used in depression-related modeling differ substantially in their biological and clinical meaning. Eye-tracking studies have linked depressive symptoms to altered visual attention, abnormal allocation to emotional stimuli, disturbed fixation, and impaired oculomotor control across fixation, saccade, and pursuit paradigms (13–19). These measures provide objective indices of attentional bias and cognitive control, although many studies remain focused on single-modality group discrimination or symptom association. Facial-behavior studies have used facial landmarks, head pose, and action-unit dynamics to quantify reduced expressivity, altered affective display, and impaired emotional flexibility (20–25). These features are clinically relevant because facial behavior forms part of routine mental-status assessment, but previous models have often emphasized binary depression detection or abnormalities in facial emotion processing rather than depressive-spectrum stratification. Language-based studies have examined transcript-derived text, acoustic speech, or audiovisual signals and have identified changes in fluency, self-reference, negation, affective wording, semantic structure, and speech characteristics (26–32). However, these studies differ in whether language is represented as text, audio, or combined speech-language information, and many focus on detecting depression status rather than jointly estimating clinical severity. Multimodal artificial intelligence has advanced automated depression assessment by integrating text, audio, facial behavior, eye tracking, and other behavioral signals (33–45). Existing models commonly use early fusion, late fusion, attention-based fusion, or deep representation learning to improve diagnostic discrimination, most often for binary depression detection or symptom-level prediction rather than simultaneous depressive-spectrum classification and severity estimation. These approaches have shown encouraging performance, but several limitations remain important for clinical translation. Many studies retain a binary case-control formulation, whereas fewer address intermediate depressive-spectrum states such as subthreshold depression. Missing or degraded modalities are often handled implicitly or by complete-case analysis, and probability calibration is rarely evaluated in detail. In addition, relatively few frameworks jointly model categorical depressive-spectrum status and continuous symptom severity while also providing individual-level interpretation. These gaps are particularly relevant when the target task extends beyond binary depression detection to spectrum-level stratification and symptom-severity estimation.

Against this background, we developed and evaluated a quality-aware multimodal framework that integrates eye tracking, facial behavior, and transcript-derived language acquired during a controlled behavioral task battery. The study was designed to move beyond conventional binary depression detection by modeling the depressive spectrum across normal control (NC), SD, and MDD, while also estimating continuous symptom severity using the 17-item Hamilton Depression Rating Scale (HAMD-17). Methodologically, the framework combines modality-specific representation learning, explicit modeling of modality availability and missingness, and quality-aware gated fusion, with an extended model incorporating Transformer-based cross-modal interaction and uncertainty-based dynamic task weighting. The evaluation further incorporated a nested repeated-resampling framework, post hoc probability calibration, component-ablation analyses, and individual-level interpretability. By linking spectrum-level classification, continuous severity estimation, missing-modality handling, calibrated prediction, and case-level explanation within a single framework, this study aimed to provide a more clinically interpretable approach for multimodal depressive-spectrum assessment, particularly for boundary states such as subthreshold depression. The framework was therefore evaluated as a calibrated and interpretable research prototype for depressive-spectrum stratification.

2 Materials and methods

2.1 Participants

Participants were recruited from Suzhou Municipal Hospital and Suyuan Community Health Service Station in Wuzhong District, Suzhou, China. The cohort comprised normal control (NC), subthreshold depression (SD), and major depressive disorder (MDD) groups. Diagnostic assessment was performed by licensed psychiatrists. MDD was diagnosed according to the International Classification of Diseases, 11th Revision (ICD-11), whereas SD and NC were defined using prespecified operational criteria supported by standardized rating scales. Clinical assessment and multimodal data acquisition were completed on the same day. The study was approved by the Ethics Committee of Suzhou Municipal Hospital (K-2025-278-K01). Written informed consent was obtained from all participants before enrollment. Detailed eligibility criteria and operational group definitions are provided in Appendix A.

2.2 Experimental procedures

All assessments were conducted in a controlled laboratory setting. Participants were seated in a quiet room, instructed to maintain a natural posture, and asked to minimize large head movements. Eye movements were recorded using a Tobii Pro Nano eye tracker at 60 Hz, mounted below a 14.5-inch laptop display (1920 × 1080 pixels), with a viewing distance of approximately 60–70 cm. Facial behavior was recorded using an external high-definition camera at 1080p and 30 frames/s. Spoken responses were transcribed for downstream language-feature extraction and data-quality control; acoustic speech features were not modeled as an independent modality. Before formal testing, a nine-point eye-tracking calibration was completed and repeated if necessary. Standardized instructions were provided before each task, and brief practice trials were given when required. A 30-s rest interval was arranged between adjacent tasks. The task battery included a sociodemographic questionnaire, a semi-structured interview, emotional text reading, emotional free viewing with verbal description, fixation stability, lateral gaze orienting, smooth pursuit, prosaccade, and antisaccade paradigms. Horizontal and sinusoidal pursuit conditions were both included. The interview and free-viewing tasks yielded transcript-derived language, whereas emotional text reading contributed eye-tracking and facial-behavior responses but was not used as an independent source for language modeling. Detailed task procedures are provided in Appendix B.

2.3 Multimodal preprocessing and feature engineering

Raw eye-tracking signals were processed using the velocity-threshold identification algorithm to classify fixations, saccades, and blinks. Samples outside the display region were discarded, and trials with a tracking ratio below 50% were excluded. A total of 723 participant-level eye-tracking features were extracted, covering fixation, visit, gaze transition, saccadic control, and higher-order cognitive and affective indices. Facial videos were analyzed frame by frame using OpenFace 2.0 to derive facial landmarks, head-pose measures, and facial action unit features. Frames with confidence below 0.7 or failed tracking were removed, and segments with a valid-frame ratio below 50% were excluded. Summary statistics and composite indices were then generated to quantify facial movement, affective expression, and task-related emotional regulation. The language modality was derived from offline transcription of spoken responses using Whisper large-v3 deployed through an OpenVINO-based workflow. Text preprocessing included normalization, sentence segmentation, identification of short segments, and conservative token filtering. Features were computed at the segment level and aggregated at the participant level within each task and emotional condition. The final language feature space included completeness and quality measures, structural and fluency indices, clinically relevant linguistic markers, and affective lexical features.

Across modalities, preprocessing, feature filtering, standardization, and feature selection were performed strictly within the training data of each resampling split to prevent information leakage. In the present study, modalities were defined by distinct data representations and processing pipelines, namely oculomotor behavior, facial behavioral dynamics, and transcript-derived linguistic features, rather than by whether they originated from separate acquisition sessions. Further details are provided in Appendix C. The outer-test data were transformed using parameters and feature sets learned exclusively from the corresponding training partition. To improve reproducibility and provide an auditable basis for subsequent modeling, the three modality-specific participant-level feature tables were first harmonized into a single raw multimodal master table before resampling-based model development. Eye-tracking, facial-behavior, and transcript-derived language features were prefixed by modality and aligned using a unique participant identifier. Diagnostic group labels and HAMD-17 scores were cross-checked across the three source tables before consensus labels were assigned. No imputation, winsorization, standardization, feature filtering, feature selection, calibration, or model training was performed during master-table construction. These procedures were deliberately deferred to the training data within each resampling split to prevent information leakage. The master table also retained modality-availability indicators and raw modality-specific missingness ratios for subsequent missing-modality modeling. Additional details on data alignment, master-table construction, missing-modality handling, and resampling-based model development are provided in Appendix D.

2.4 Multimodal modeling

Tabular features from the eye-tracking, facial, and language modalities were aligned by unique participant identifier. The classification target was depressive-status group membership (NC, SD, or MDD), and the regression target was HAMD-17. To accommodate incomplete multimodal acquisition, modality-level meta-features were constructed, including availability indicators and modality-specific missing ratios. Entire feature blocks for unavailable modalities were set to zero after fold-specific preprocessing. Model development followed a nested repeated-resampling framework. The three diagnostic groups were balanced by design, with 62 participants in each group. In the outer loop, data were repeatedly divided into training and held-out test sets using stratified 85:15 random partitioning over five repeats. Within each outer-training partition, five-fold stratified inner cross-validation was used for hyperparameter selection and early stopping. This design preserved class distribution across training and test partitions. In addition to ACC, we reported BACC, F1-macro, and class-level recall to avoid overinterpreting overall accuracy. Within each inner-loop training fold, modality-specific preprocessing included missingness filtering, median imputation, winsorization, low-variance filtering, redundancy reduction based on Spearman correlation, and z-score normalization. Supervised feature selection was then performed using repeated subsampling stability selection with elastic-net multinomial logistic regression. Because the downstream framework jointly modeled depressive-spectrum classification and HAMD-17 regression, feature selection was anchored to the classification structure to retain stable disease-relevant signals, while continuous severity estimation was learned jointly within the subsequent multimodal model.

The primary multimodal model, Baseline-3, combined modality-specific multilayer perceptron encoders with quality-aware gated fusion and joint classification-regression learning. For each modality m, where , the preprocessed input vector x_m was mapped to a latent modality representation by a modality-specific encoder, . Quality metadata q_m, including modality-availability indicators and modality-specific missingness ratios, were used together with modality embeddings to generate normalized gating weights α_m. The Baseline-3 fused representation was then defined as , and was passed to separate classification and HAMD-17 regression heads. Baseline-3+ extended this architecture by treating modality embeddings as modality tokens and processing them through a lightweight Transformer encoder to obtain an interaction-aware shared representation. The multitask objective jointly optimized depressive-spectrum classification and HAMD-17 regression. In Baseline-3+, uncertainty-based dynamic task weighting assigned learnable task-uncertainty parameters to the classification and regression losses, allowing the relative contribution of each task to be adjusted during optimization. PCGrad was not included in the full Baseline-3+ model and was evaluated separately as a sensitivity arm under fixed task weighting. The full mathematical formulations of modality-specific encoding, quality-aware gated fusion, Route A and Route B probability reconstruction, Transformer-based interaction, dynamic task weighting, temperature scaling, expected calibration error, and ablation settings are provided in Supplementary Appendix E.

Two classification routes were implemented. Route A directly predicted NC, SD, and MDD. Route B used a hierarchical two-stage design that first separated NC from the depressive spectrum and then distinguished SD from MDD within spectrum-positive samples. Training incorporated modality dropout, AdamW optimization, and gradient clipping. Modality dropout was applied only during training; during each training iteration, available modality embeddings could be randomly masked according to the hyperparameter selected within the inner cross-validation loop. Validation and outer-test evaluation used the true observed modality-availability pattern without artificial dropout. Temperature-scaling parameters were fitted on validation predictions derived exclusively from the training portion of each outer split and were then applied unchanged to the corresponding outer-test predictions. The overall architecture of the proposed quality-aware multimodal framework, including fold-wise preprocessing, modality-specific encoding, quality-aware gated fusion, Baseline-3+ extensions, multitask outputs, calibration, and interpretability analysis, is summarized in Figure 1.

Figure 1

2.5 Evaluation and interpretability

Classification performance was assessed using accuracy (ACC), balanced accuracy (BACC), one-vs-rest macro-averaged area under the receiver operating characteristic curve (AUC-OVR), macro-averaged F1 score (F1-macro), and log loss. Regression performance was assessed using mean absolute error (MAE), root mean squared error (RMSE), and coefficient of determination (R²). Metrics were calculated on the outer-loop test sets and summarized across repeated splits as mean ± standard deviation. Interpretability was evaluated at both the modality and individual levels. Gating-weight distributions were examined overall, by diagnostic group, and by missingness burden. Individual-level explanations were derived using Integrated Gradients (IG) and counterfactual analysis under modality-availability constraints. To characterize factors associated with classification failure and regression error, surrogate models based on uncertainty, calibration, missingness, gating, and attention features were additionally analyzed using SHapley Additive exPlanations (SHAP). Full details of the fusion models, calibration, evaluation metrics, and interpretability analyses are provided in Supplementary Appendix E. Case-level interpretability analyses were conducted on the held-out outer-test set of one representative Baseline-3+ Route B outer repeat selected for illustrative analysis. This test set comprised 28 participants. To further address calibration and statistical robustness, we re-analyzed the original outer-test prediction files after temperature scaling. Expected calibration error was calculated using the maximum calibrated class probability and empirical correctness across 10 equal-width confidence bins. Metric values were summarized across the five outer-loop test repeats as mean with 95% confidence intervals. Ninety-five percent confidence intervals were calculated across the five outer repeats using the t distribution and were interpreted as descriptive resampling uncertainty summaries rather than as population-level inferential intervals. Paired comparisons between Baseline-3 and Baseline-3+ were performed across matched outer repeats using sign-flip permutation tests. Because only five outer repeats were available, these tests were interpreted as exploratory repeat-consistency analyses rather than confirmatory hypothesis tests.

Component-level ablation analyses were conducted to clarify the contribution of the main Baseline-3+ extensions. The full Baseline-3 and full Baseline-3+ models were retained as reference settings. Additional ablation settings selectively removed uncertainty-based dynamic task weighting or Transformer-based cross-modal interaction, and a PCGrad sensitivity arm was evaluated under fixed task weighting. Ablation results were summarized using the same outer-test discrimination, calibration, and HAMD-17 regression metrics as the full models. To provide a representative conventional reference aligned with common early-fusion tabular modeling strategies, we additionally evaluated regularized linear benchmark models within the same nested repeated-resampling framework. Elastic-net logistic regression was used for depressive-spectrum classification, and Ridge regression was used for HAMD-17 severity estimation. These models were selected a priori because they are interpretable and appropriate for high-dimensional small-sample tabular data. The benchmark analyses were performed using stability-selected single-modality features and early-fusion features across all modalities. These conventional models were interpreted as reference analyses of the extracted feature space rather than as replacements for the proposed quality-aware multitask framework, because they do not model modality quality, joint classification-regression learning, calibrated uncertainty, or individual-level multimodal explanation.

3 Results

3.1 Participant characteristics

A total of 186 participants were included, comprising 62 NC, 62 individuals with SD, and 62 patients with MDD. The reproducible multimodal master table contained 723 eye-tracking features, 2021 facial-behavior features, and 1029 transcript-derived language features. Cross-table verification identified no inconsistencies in diagnosis labels or HAMD-17 scores. Complete three-modality data were available for 168 participants, whereas 18 participants had at least one unavailable modality, supporting the use of modality-availability encoding and missingness-aware fusion. The master-table construction and modality-availability patterns are summarized in Supplementary Table 1 and S2. Baseline demographic and clinical characteristics are presented in Table 1. The three groups were broadly comparable in age, sex, education, marital status, offspring status, living arrangement, and eye-tracking acquisition quality. Occupational distribution and body mass index showed significant omnibus differences, although no adjusted pairwise comparison remained significant after multiple-testing correction. Several lifestyle and psychosocial variables differed across groups. Satisfaction with current income, sleep status, dietary pattern, physical exercise, cognitive activity, social participation, and psychological disclosure all showed significant between-group differences, generally shifting from more favorable profiles in NC toward less favorable profiles in SD and MDD. By contrast, smoking history, alcohol use, and household-work involvement did not differ significantly. Clinical symptom burden also differed markedly. Scores on the Center for Epidemiologic Studies Depression Scale, the 17-item Hamilton Depression Rating Scale (HAMD-17), the 7-item Generalized Anxiety Disorder scale, and the 14-item Hamilton Anxiety Rating Scale increased stepwise from NC to SD to MDD, with significant pairwise differences throughout. Anxiety-category distribution showed the same gradient. Together, these findings were consistent with the expected clinical ordering of the three groups and indicated increasing depressive and anxiety burden across the spectrum.

3.2 Multimodal fusion models

3.2.1 Training dynamics and validation performance

Across both classification routes, Baseline-3 and Baseline-3+ showed rapid improvement in validation F1-macro across outer repeats during the early training phase, followed by a stable plateau, as shown in Supplementary Figure 1. Most gains occurred within approximately 5 to 10 epochs, indicating efficient convergence. Within Baseline-3, both Route A and Route B reached competitive performance quickly, although split-to-split variation remained visible. By contrast, Baseline-3+ showed faster early convergence, a higher plateau, and less fluctuation across outer repeats in both routes, suggesting that cross-modal interaction and training-stabilization strategies improved optimization stability.

3.2.2 Overall outer-test performance

After temperature scaling, the outer-test results showed a clear performance structure across model families and classification routes, as shown in Figure 2.

Figure 2

For classification, Baseline-3+ generally improved threshold-dependent discrimination metrics, including ACC, BACC, and F1-macro, relative to Baseline-3. Within Baseline-3, Route B outperformed Route A across the main discrimination metrics. ACC improved from 0.829 ± 0.030 in Route A to 0.864 ± 0.030 in Route B. BACC improved from 0.827 ± 0.029 to 0.861 ± 0.031, and F1-macro improved from 0.819 ± 0.027 to 0.857 ± 0.030. Log loss was also lower in Route B, indicating better-calibrated class-probability fitting. Baseline-3+ further increased classification performance in both routes, with ACC, BACC, and F1-macro all approaching 0.90. The two routes were similar in point estimates, although Route B retained a lower log loss than Route A, suggesting an additional advantage in probabilistic output quality.

For regression, the pattern differed. Within Baseline-3, Route A provided more accurate continuous prediction of HAMD-17 severity than Route B, with lower MAE, lower RMSE, and higher R². Within the original gated-fusion framework, the direct three-class organization appeared more favorable for continuous symptom estimation, whereas the hierarchical route preferentially benefited categorical discrimination. In Baseline-3+, this route-dependent difference became much smaller. MAE, RMSE, and R² were closely similar between Route A and Route B, indicating that the introduction of cross-modal interaction and training-stabilization strategies attenuated the effect of classification-head structure on regression performance and partly compensated for the regression disadvantage previously seen in Route B. When classification, calibration, and regression metrics were considered jointly, Baseline-3+ showed the most balanced overall profile, with improved categorical performance and lower expected calibration error. The regression benefit was route-dependent and was most evident in reducing the regression disadvantage of the hierarchical route.

3.2.3 Calibration and component-ablation analyses

Additional calibration analyses showed that Baseline-3+ achieved lower expected calibration error than Baseline-3 in both classification routes. Expected calibration error decreased from 0.149 to 0.069 in Route A and from 0.122 to 0.060 in Route B, indicating more reliable temperature-scaled class-probability estimates. Mean values with 95% confidence intervals and component-ablation results are provided in Supplementary Table 3, and the calibration summary is shown in Supplementary Figure 2. Full conventional benchmark results are provided in Supplementary Table 4. Exploratory paired outer-repeat sign-flip comparisons between Baseline-3+ and Baseline-3 are reported in Supplementary Table 5. These tests showed directionally favorable changes in the main classification metrics and expected calibration error, although improvements were not uniform across all metrics and most comparisons did not reach conventional statistical significance because only five matched outer repeats were available. Therefore, the paired tests were interpreted descriptively as repeat-consistency checks rather than confirmatory hypothesis tests. The component-ablation analysis further showed that the added Baseline-3+ components did not contribute uniformly to all outcomes. Removal of uncertainty-based dynamic task weighting produced the clearest deterioration in the regression component, especially under Route B. In contrast, removal of Transformer-based cross-modal interaction did not uniformly reduce categorical discrimination, suggesting that the quality-aware gated fusion module already captured substantial diagnostic information in the present dataset. The PCGrad sensitivity arm preserved a relatively favorable regression profile but did not improve categorical discrimination. Performance-oriented ablation effects relative to the full Baseline-3+ model are summarized in Supplementary Figure 3.

As prespecified regularized conventional benchmarks, Elastic-net logistic regression and Ridge regression provided tabular reference comparisons using single-modality and early-fusion feature sets. Early-fusion Elastic-net logistic regression achieved an ACC of 0.882, F1-macro of 0.884, and AUC-OVR of 0.958 for depressive-spectrum classification. Early-fusion Ridge regression achieved an MAE of 2.277, RMSE of 2.993, and R² of 0.843 for HAMD-17 estimation. Notably, the face-only Elastic-net classifier also showed strong performance, consistent with the dominant contribution of facial behavior observed in the gating-weight and individual-attribution analyses. These benchmark results indicate that the extracted multimodal feature space, especially the facial-behavior feature block, contained substantial diagnostic and severity-related information. However, the conventional benchmarks should be interpreted as reference analyses of the extracted feature space rather than as replacements for the proposed quality-aware multitask framework, because they do not explicitly model modality quality, joint classification-regression learning, calibrated uncertainty profiling, or individual-level multimodal interpretation.

3.2.4 Classification error patterns and class-level performance

Confusion-matrix analysis showed that most classification errors occurred around the SD boundary, as shown in Figure 3. In Baseline-3, Route A classified the two extreme groups relatively well but showed weaker separation of SD. Mean class recall across outer-test repeats was approximately 93.3% for NC, 58.7% for SD, and 96.0% for MDD. Route B reduced this weakness: mean recall was approximately 91.1% for NC, 67.1% for SD, and 100% for MDD. This pattern was consistent with the higher ACC, BACC, and F1-macro of Route B, suggesting that the hierarchical design improved discrimination of the intermediate phenotype without compromising recognition of the extreme classes. In Baseline-3+, SD-related confusion was further reduced. Residual errors remained concentrated near the SD boundary, whereas recall for NC and MDD remained high under both routes. The remaining misclassifications were mainly limited to a small number of SD cases classified as NC or MDD, which is clinically plausible given the transitional and heterogeneous nature of SD.

Figure 3

3.2.5 Continuous severity prediction

Regression scatterplots showed generally good agreement between predicted and observed HAMD-17 scores across models and routes, as shown in Figure 4. Within Baseline-3, Route A provided the best overall fit. The fitted relationship was closely aligned with the identity line, with a slope near 1 and an intercept close to 0, indicating limited systematic bias and favorable calibration. Route B also preserved an approximately linear relationship but showed larger overall error and a positive intercept, suggesting mild overestimation in some score ranges. Within Baseline-3+, the regression difference between the two routes narrowed substantially. Both routes showed slopes close to 1 and only modest positive intercepts, indicating broadly comparable calibration. In both models, the 95% prediction interval widened at higher HAMD-17 scores, suggesting greater uncertainty in the high-severity range, consistent with greater clinical heterogeneity and lower sample density among individuals with more severe symptoms.

Figure 4

3.3 Model interpretability

3.3.1 Quality-aware gating and cross-modal interaction

The gating-weight analysis showed that the multimodal fusion network learned stable modality-contribution profiles and adaptively reweighted modalities according to data completeness and signal quality, as shown in Supplementary Figure 4 and S5. In Baseline-3, Route A relied more strongly on the facial modality, followed by eye tracking, with language receiving the smallest weight overall. Route B showed a more balanced allocation between facial and eye-tracking inputs, while language remained relatively weakly weighted. After stratification by diagnostic group, the general weighting pattern was preserved across NC, SD, and MDD, indicating that the learned modality preferences reflected broadly shared information value rather than a single class. After stratification by overall missingness tertiles, the high-missingness subgroup showed an expected shift toward more stable and more available modalities. In Baseline-3+, a similar overall gate-weight structure was observed, with facial signals remaining dominant and eye tracking ranking second. However, Transformer attention from the classification token to modality tokens showed relatively stronger allocation to language than was suggested by gate weights alone, indicating that language contributed to shared representation building through cross-modal interaction pathways.

3.3.2 Confidence and selective prediction

Selective-prediction analysis showed that model confidence was clinically informative, as shown in Supplementary Figure 6. When only the highest-confidence cases were retained, classification accuracy remained high. As coverage increased and progressively lower-confidence cases were included, accuracy declined gradually. Although some fluctuation was observed at very low coverage because of the small retained sample size, the overall confidence-stratification pattern remained clear. Compared with Baseline-3, Baseline-3+ produced smoother coverage-accuracy curves across a wider coverage range, suggesting more reliable probability ranking.

3.3.3 Drivers of classification errors

To identify factors associated with classification failure, an interpretable surrogate classifier was trained using prediction entropy, calibrated confidence, gate weights, modality missingness, availability indicators, and, for Baseline-3+, attention-derived features. In Baseline-3, these variables showed good ability to separate correct from incorrect predictions, with area under the receiver operating characteristic curve (AUC) values of 0.88 for Route A and 0.83 for Route B, as shown in Supplementary Figure 7 and S8. These variables captured a coherent profile of error-prone cases. Across models, SHAP analyses consistently identified prediction entropy and calibrated confidence as the dominant error-related variables. Higher entropy was associated with increased error risk, whereas higher calibrated confidence was associated with reduced error risk. Beyond these uncertainty measures, language gate weight, eye-tracking missingness, and gate weights from the eye-tracking and facial modalities also contributed to classification-error risk, indicating that prediction instability increased when key modalities were degraded or incomplete. In Baseline-3+, surrogate discrimination was lower, with AUC values of 0.65 for Route A and 0.56 for Route B. After performance improvement, the remaining errors appeared fewer, more dispersed, or less well captured by simple uncertainty and quality descriptors. Nevertheless, SHAP ranking still highlighted uncertainty and calibrated confidence as the leading determinants, while attention-related variables, especially those involving language, also entered the important-feature set, suggesting a contribution of cross-modal interaction patterns to residual classification errors.

3.3.4 Drivers of regression absolute error

A surrogate regression model was further used to explain absolute error in HAMD-17 prediction using the same family of uncertainty, confidence, gating, attention, and missingness variables, as shown in Supplementary Figure 9 and S10. The overall explanatory power of these surrogate models was limited, indicating that variation in regression error was not fully captured by these quality and fusion descriptors alone. Even so, SHAP ranking showed a consistent pattern in which greater predictive uncertainty, lower calibrated confidence, and degradation of key modality quality were associated with larger absolute prediction errors.

3.4 Individual-level interpretability and case-based presentation

This section focuses on one representative Baseline-3+ Route B outer repeat selected for illustrative case-level interpretation. Although Baseline-3+ Route A showed a favorable overall metric profile, Route B was selected for detailed case-level interpretation because its hierarchical structure provided more clinically informative discrimination around the SD boundary, which represented the principal region of residual classification ambiguity. For illustrative case-level analysis, one representative outer repeat from Route B was examined in detail; this analysis was intended to demonstrate the interpretability workflow rather than to provide a population-level summary.

3.4.1 Distribution of individual IG attributions

Across the held-out outer-test set of the representative Route B repeat, facial features dominated the attribution profile in 27 of 28 individuals, indicating that individual-level diagnostic decisions were driven primarily by facial dynamics and related behavioral signatures. Eye tracking often provided a secondary evidentiary channel, whereas language generally contributed smaller and more auxiliary signals. Among the most frequently selected high-attribution features, facial evidence was concentrated in action-unit-related intensity, duration, and transition measures. Recurrently important eye-tracking features included saccadic velocity and amplitude, fixation behavior, and area-of-interest (AOI) stability. Language features entered the top-ranking sets less frequently and with smaller absolute contributions, most often through length-related, distributional, or filler-density descriptors.

3.4.2 Counterfactual analysis

Counterfactual analysis was performed with NC as the target class to evaluate local decision stability and reversibility under minimal perturbation. Overall, the target class was reached in 15 of 28 cases, corresponding to an achievement rate of 53.6%. However, success varied markedly by initial prediction. All cases initially predicted as NC already satisfied the target condition and required only minimal perturbation. Among cases initially predicted as SD, the achievement rate was 50.0%. Among cases initially predicted as MDD, no case could be shifted to NC within the imposed perturbation constraint. In cases where the target remained unattained, the target-class probability remained very low, particularly for MDD and a subset of SD cases, indicating greater local stability of the decision boundary around more severe phenotypes. Across cases, the largest counterfactual perturbation was usually observed in the facial modality, followed by eye tracking, with language showing the smallest magnitude of change. This pattern closely matched the IG results.

3.4.3 Structured clinical summaries and representative cases

For case-based presentation, class probabilities, HAMD-17 prediction, IG evidence, and counterfactual reversibility were integrated into structured case summaries. Within this 28-participant representative outer-test set, 11 cases showed low-risk profiles leaning toward NC, 8 showed intermediate-to-high-risk depressive-spectrum features, and 9 leaned toward MDD. Three representative cases illustrated low-risk, boundary, and high-risk profiles. The NC case showed high posterior probability for NC, a very low predicted HAMD-17 score, and near-zero counterfactual perturbation, indicating a stable low-risk classification. The SD case showed high spectrum-level probability but low conditional probability of MDD within the depressive spectrum, consistent with an intermediate position; the counterfactual target was not achieved within the perturbation limit. The MDD case showed high probability at both stages of the hierarchical route, a high predicted HAMD-17 score, and failure of counterfactual conversion to NC, indicating strong local stability of the high-risk classification. Representative visualizations are shown in Figure 5 and Supplementary Figure 11 and S12.

Figure 5

4 Discussion

In this single-cohort study, a quality-aware multimodal framework integrating eye tracking, facial behavior, and transcript-derived language showed favorable discrimination across NC, SD, and MDD while also supporting HAMD-17 severity estimation. The main contribution of this work is not limited to point-estimate performance. Instead, the framework integrates eye tracking, facial behavior, and transcript-derived language within a controlled task battery, explicitly models modality availability and missingness, combines quality-aware gated fusion with multitask learning, and evaluates calibrated prediction, component ablation, selective prediction, and case-level interpretability. Baseline-3+ showed the most favorable classification and calibration profile, whereas the hierarchical route most clearly improved discrimination of the intermediate phenotype, particularly SD. For HAMD-17 severity estimation, the benefit of Baseline-3+ was route-dependent and was most evident in reducing the regression disadvantage of the hierarchical route. Facial features contributed the largest share of explanatory signal, with complementary input from eye tracking and smaller but meaningful contributions from text through both direct and cross-modal pathways.

The concentration of misclassification around SD is clinically informative. This pattern is consistent with epidemiologic and clinical evidence that SD occupies an intermediate and heterogeneous position between health and syndromal depression (9–12). In our data, NC and MDD were classified more reliably than SD across model settings, and most residual errors involved SD being assigned to NC or MDD. The main challenge therefore appeared to lie not in global model instability but in ambiguity concentrated at the depressive-spectrum boundary. The improvement in SD recall under the hierarchical route suggests that a staged decision process is better aligned with the latent structure of depressive-spectrum phenotypes than a flat three-class decision. A clinically useful model may first need to distinguish depressive-spectrum status from nondepressed status and then refine subgroup discrimination within the spectrum. This structure may be particularly appropriate when the target phenotype is defined by partial symptom expression rather than by a sharply bounded syndrome.

The divergence between Route A and Route B in Baseline-3 further supports this interpretation. Within the original gated-fusion architecture, Route B improved classification, whereas Route A performed better for HAMD-17 regression. This suggests that shared representations optimized for boundary-sensitive categorical discrimination are not identical to those that best preserve continuous symptom gradients. After introducing cross-modal interaction and uncertainty-based dynamic task weighting in Baseline-3+, this route-dependent divergence became much smaller. These results suggest that the extended architecture improved compatibility between the classification and regression objectives, particularly by reducing the route-dependent regression disadvantage observed under the hierarchical setting.

The dominance of facial information is consistent with current evidence linking depression to reduced expressivity, altered facial affect, impaired emotional flexibility, and abnormalities in emotion recognition (20–25). In the present study, facial features were dominant in both gate-weight summaries and individual-level attributions. The convergence across interpretability levels suggests that facial behavior contributed distinct discriminative information rather than simply covarying with other modalities. One likely explanation lies in the structure of the task battery. Semi-structured interview and emotion-related description tasks place demands on spontaneous expression, interpersonal signaling, and behavioral regulation, domains in which depressive psychopathology is often clinically visible but difficult to quantify with conventional scales. These findings support the view that automated facial analysis can operationalize behavior that is familiar to clinicians but insufficiently standardized in routine assessment.

Eye tracking contributed differently. It was generally secondary to facial behavior in the fusion analyses, yet its contribution remained stable and became relatively more relevant in the hierarchical route. This pattern is compatible with literature linking depression to altered attentional allocation, affective bias, and oculomotor control across fixation, saccade, and pursuit paradigms (13–19). In our framework, eye-tracking signals may have contributed less to coarse separation of the extreme groups and more to resolving intermediate states. This interpretation is plausible because attentional bias and inhibitory control may vary gradually across the depressive spectrum and become especially informative near decision boundaries. The recurrent presence of eye-tracking features in individual attribution profiles further supports the view that they provided clinically meaningful, nonredundant information even when they were not the dominant modality.

The contribution of language differed from that of the other modalities. In the gate-weight analyses, language was consistently assigned lower weight than facial behavior or eye tracking. However, in Baseline-3+, attention analyses suggested a greater role for language in cross-modal interaction than gate weights alone would imply. This indicates that language functioned less as a primary standalone signal and more as a contextual or relational signal informing shared representation learning. Such a pattern is consistent with the present design. We modeled transcript-derived features rather than acoustic speech features, and the extracted variables were mainly lexical, structural, and task-conditioned. Under these conditions, language may provide only moderate standalone discrimination but still contribute meaningful disambiguation when integrated with behavioral channels. This interpretation is also consistent with the error analyses, in which language-related features remained relevant despite lower direct gate weight.

The observed advantages of Baseline-3+ likely reflected a combination of cross-modal interaction modeling, task-balance control, and improved calibration, rather than model capacity alone. The addition of explicit cross-modal interaction and uncertainty-based dynamic task weighting was associated with a smaller route-dependent difference in severity regression and improved the stability of outer-loop validation trajectories. The PCGrad sensitivity arm was used only to examine gradient-conflict mitigation under fixed task weighting and was not treated as part of the defining full Baseline-3+ setting (33–45). At the same time, our results indicate that such improvements are not uniform across tasks. In Baseline-3, Route A favored regression and Route B favored classification; in Baseline-3+, this tradeoff became less pronounced. This suggests that classification structure and shared representation learning interact in clinically relevant ways in multitask depressive-spectrum modeling.

The additional calibration and ablation analyses refine the interpretation of Baseline-3+. The extended architecture should not be viewed as uniformly superior across every metric and route. Its main advantage was reflected in improved classification, markedly lower expected calibration error, and a more balanced classification-regression profile compared with Baseline-3. Uncertainty-based dynamic task weighting appeared particularly relevant for preserving HAMD-17 regression performance under the hierarchical route, whereas Transformer-based cross-modal interaction did not provide uniform incremental benefit in this modest-sized cohort. This pattern is methodologically plausible because the quality-aware gated fusion module already captured strong modality-level signals, especially from facial behavior and eye tracking. Accordingly, the value of Baseline-3+ lies not only in point-estimate performance, but also in calibrated confidence, multitask balance, and richer modeling of multimodal interactions.

The prespecified conventional benchmarks further contextualize the proposed framework in relation to common early-fusion tabular modeling strategies used in depression-related machine learning. Their performance confirmed that the extracted multimodal feature space contained diagnostic and severity-related information. However, these models should be interpreted as reference analyses rather than competing replacements for the proposed framework. Regularized early-fusion models provide an interpretable estimate of the discriminative value of the feature space, but they do not explicitly model modality quality, do not jointly optimize depressive-spectrum classification and HAMD-17 regression, and do not provide calibrated uncertainty profiling or individual-level multimodal interpretation. In this sense, the contribution of Baseline-3+ is not limited to point-estimate performance. Its main value lies in linking incomplete multimodal evidence, multitask prediction, calibrated probability output, and case-level explanation within a single framework. This distinction is clinically relevant because the intended use case is not simple binary case detection, but stratification across NC, SD, and MDD with simultaneous severity estimation and interpretable evidence for boundary cases such as SD.

The uncertainty analyses are relevant to clinical translation because they identify cases in which model output should be interpreted with caution. Prediction entropy and calibrated confidence were the strongest determinants of classification error across models, and selective-prediction curves showed that confidence ranking enriched for more reliable predictions. In clinical settings, model utility depends not only on discrimination but also on the ability to identify cases in which output should be interpreted cautiously (3, 36–38, 42–44). The smoother coverage-accuracy curves in Baseline-3+ suggest that the extended model improved not only classification performance but also the internal ordering of predictive uncertainty. This is especially relevant for borderline cases, triage contexts, and workflows in which model output is intended to support rather than replace clinical judgment.

The surrogate error models help clarify what changed after performance improvement in Baseline-3+. In Baseline-3, uncertainty, confidence, and modality-quality descriptors separated correct from incorrect predictions relatively well. In Baseline-3+, this surrogate discrimination weakened. This is unlikely to mean that these variables became unimportant. Rather, once more systematic errors had been reduced, the remaining failures appeared fewer, more dispersed, and less easily summarized by a simple quality-and-uncertainty profile. This interpretation is consistent with the continued prominence of uncertainty and calibrated confidence in the SHAP rankings even after surrogate discrimination declined. It also suggests that future gains may depend less on general quality-aware fusion and more on targeted modeling of the residual boundary cases.

The individual-level explanation results were consistent with the overall model structure. In nearly all interpreted cases, facial features provided the dominant evidence, eye tracking contributed secondary evidence, and text served an auxiliary role. Counterfactual analysis further showed a gradient in local decision stability: low-risk NC cases were usually easy to preserve, MDD cases were much harder to move toward NC under constrained perturbation, and SD cases were intermediate. This pattern aligns with the group-level confusion structure and with the conceptual position of SD as a transition-prone but heterogeneous state. This pattern argues against a purely redundant fusion effect and supports clinically plausible organization of multimodal evidence.

Several limitations should be acknowledged. First, the sample was recruited from a single regional setting, and the cross-sectional design precludes direct inference about longitudinal transition from SD to MDD. The balanced group sizes facilitated model comparison but do not reflect clinical prevalence. External validation was not available, and future studies should evaluate prevalence-weighted performance in independent and naturalistic clinical cohorts. Acoustic speech features were not modeled as an independent modality, which likely narrowed the representational range of the language-related channel. Facial analysis depended on video quality and automated tracking, and language-feature extraction depended on automatic speech recognition, both of which may introduce systematic measurement error.

Second, HAMD-17 thresholds contributed to the operational definition of depressive-spectrum groups, and the HAMD-17 total score was also used as the regression target. The classification and severity-estimation tasks were therefore clinically related rather than statistically independent. The regression results should be interpreted as within-cohort estimation of the criterion symptom-severity scale, rather than as validation against an independent clinical endpoint. In addition, confidence intervals and paired tests were calculated across five outer repeats and should be viewed as descriptive resampling summaries rather than definitive inferential evidence. The Transformer extension increased model complexity and did not provide uniform incremental benefit across all outcomes, suggesting that simpler quality-aware gated-fusion models may remain preferable in settings where computational efficiency, interpretability, or deployment feasibility is prioritized. Finally, the interpretability analyses were intended to characterize model behavior rather than establish causal mechanisms of depressive psychopathology. These limitations indicate that the present framework should be viewed as a calibrated and interpretable research prototype for depressive-spectrum stratification rather than as a deployable diagnostic system.

5 Conclusions

Overall, a controlled multimodal task battery combined with explicit missingness modeling, calibrated prediction, and structured interpretation may support clinically meaningful stratification across the depressive spectrum. The most plausible role of this framework is to augment clinical assessment, particularly in intermediate or ambiguous cases such as SD, rather than to replace clinician judgment. Future work should prioritize external validation, longitudinal follow-up, integration of acoustic speech with transcript-derived language, and prospective evaluation in real clinical workflows.

References

• 1

  CuiLLiSWangSWuXLiuYYuWet al. Major depressive disorder: hypothesis, mechanism, prevention and treatment. Signal Transduct Target Ther. (2024) 9:30. doi: 10.1038/s41392-024-01738-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 2

  ZiererCBehrendtCLepach-EngelhardtAC. Digital biomarkers in depression: a systematic review and call for standardization and harmonization of feature engineering. J Affect Disord. (2024) 356:438–49. doi: 10.1016/j.jad.2024.03.163

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3

  OrtizAMulsantBH. Beyond step count: are we ready to use digital phenotyping to make actionable individual predictions in psychiatry? J Med Internet Res. (2024) 26:e59826. doi: 10.2196/59826

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4

  ChoiAOoiALottridgeD. Digital phenotyping for stress, anxiety, and mild depression: systematic literature review. JMIR Mhealth Uhealth. (2024) 12:e40689. doi: 10.2196/40689

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5

  LeaningIEIkaniNSavageHSLeowABeckmannCRuhéHGet al. From smartphone data to clinically relevant predictions: a systematic review of digital phenotyping methods in depression. Neurosci Biobehav Rev. (2024) 158:105541. doi: 10.1016/j.neubiorev.2024.105541

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6

  WinterNRBlankeJLeeningsRErnstingJFischLSarinkKet al. A systematic evaluation of machine learning-based biomarkers for major depressive disorder. JAMA Psychiatry. (2024) 81:386–95. doi: 10.1001/jamapsychiatry.2023.5083

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 7

  MaoKWuYChenJ. A systematic review on automated clinical depression diagnosis. NPJ Ment Health Res. (2023) 2:20. doi: 10.1038/s44184-023-00040-z

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8

  Abd-AlrazaqAAlSaadRShuweihdiFAhmedAAzizSSheikhJ. Systematic review and meta-analysis of performance of wearable artificial intelligence in detecting and predicting depression. NPJ Digit Med. (2023) 6:84. doi: 10.1038/s41746-023-00828-5

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9

  LiXZhangHHanXGuoLCebanFLiaoYet al. Predictive potential of somatic symptoms for the identification of subthreshold depression and major depressive disorder in primary care settings. Front Psychiatry. (2023) 14:999047. doi: 10.3389/fpsyt.2023.999047

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10

  HaoXJiaYChenJZouCJiangC. Subthreshold depression: a systematic review and network meta-analysis of non-pharmacological interventions. Neuropsychiatr Dis Treat. (2023) 19:2149–69. doi: 10.2147/ndt.S425509

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11

  ZhangRPengXSongXLongJWangCZhangCet al. The prevalence and risk of developing major depression among individuals with subthreshold depression in the general population. Psychol Med. (2023) 53:3611–20. doi: 10.1017/s0033291722000241

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12

  VolzHPStirnweißJKasperSMöllerHJSeifritzE. Subthreshold depression - concept, operationalisation and epidemiological data. A scoping review. Int J Psychiatry Clin Pract. (2023) 27:92–106. doi: 10.1080/13651501.2022.2087530

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13

  StolicynASteeleJDSerièsP. Prediction of depression symptoms in individual subjects with face and eye movement tracking. Psychol Med. (2022) 52:1784–92. doi: 10.1017/s0033291720003608

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14

  MonégerJHarika-GermaneauGJaafariNDoolubDWarckLSelimbegovićLet al. Depressive self-focus bias following failure: an eye-tracking study among individuals with clinical depression. Front Psychiatry. (2024) 15:1459831. doi: 10.3389/fpsyt.2024.1459831

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15

  LuYSongLHuangCFanTHuangJZhangLet al. The association between eye movement characteristics and cognitive function in adolescents with major depressive disorder. Psychiatry Res Neuroimaging. (2024) 345:111914. doi: 10.1016/j.pscychresns.2024.111914

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16

  ImbertLNeigeCMoirandRPivaGBediouBValletWet al. Eye-tracking evidence of a relationship between attentional bias for emotional faces and depression severity in patients with treatment-resistant depression. Sci Rep. (2024) 14:12000. doi: 10.1038/s41598-024-62251-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 17

  SuslowTHoepfelDKerstingABodenschatzCM. Depressive symptoms and visual attention to others' eyes in healthy individuals. BMC Psychiatry. (2024) 24:184. doi: 10.1186/s12888-024-05633-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 18

  FeldmannLZsigoCMörtlIBartlingJWachingerCOortFet al. Emotion regulation in adolescents with major depression - evidence from a combined EEG and eye-tracking study. J Affect Disord. (2023) 340:899–906. doi: 10.1016/j.jad.2023.08.087

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 19

  BodenschatzCMCzepluchFKerstingASuslowT. Efficient visual search for facial emotions in patients with major depression. BMC Psychiatry. (2021) 21:92. doi: 10.1186/s12888-021-03093-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 20

  SlonimDAYehezkelIPazABar-KalifaEWolffMDarAet al. Facing change: using automated facial expression analysis to examine emotional flexibility in the treatment of depression. Adm Policy Ment Health. (2024) 51:501–8. doi: 10.1007/s10488-023-01310-w

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21

  LacerdaKCDSouzaFCOAraújoCRVMotaBEFMuñozPMGBergerWet al. High depressive symptomatology reduces emotional reactions to pictures of social interaction. Sci Rep. (2024) 14:1266. doi: 10.1038/s41598-024-51813-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22

  TernovoySUstyuzhaninDShariyaMBeliaevskaiaARoldan-ValadezEShishorinRet al. Recognition of facial emotion expressions in patients with depressive disorders: a functional MRI study. Tomography. (2023) 9:529–40. doi: 10.3390/tomography9020043

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 23

  MoFGuJZhaoKFuX. Confusion effects of facial expression recognition in patients with major depressive disorder and healthy controls. Front Psychol. (2021) 12:703888. doi: 10.3389/fpsyg.2021.703888

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 24

  AkhapkinRVVolelBAShishorinRMUstyuzhaninDVPetelinDS. Recognition of facial emotion expressions in patients with depressive disorders: a prospective, observational study. Neurol Ther. (2021) 10:225–34. doi: 10.1007/s40120-021-00231-w

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 25

  KrauseFCLinardatosEFrescoDMMooreMT. Facial emotion recognition in major depressive disorder: a meta-analytic review. J Affect Disord. (2021) 293:320–8. doi: 10.1016/j.jad.2021.06.053

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 26

  LiYKumbaleSChenYSuranaTChngESGuanC. Automated depression detection from text and audio: a systematic review. IEEE J BioMed Health Inform. (2025) 29:7498–513. doi: 10.1109/jbhi.2025.3570900

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27

  MaranPLBraquehaisMDVlaicAAlonzo-CastilloMTVendrell-SerresJRamos-QuirogaJAet al. Performance of automatic speech analysis in detecting depression: systematic review and meta-analysis. JMIR Ment Health. (2025) 12:e67802. doi: 10.2196/67802

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 28

  LiuLLiuLWafaHATydemanFXieWWangY. Diagnostic accuracy of deep learning using speech samples in depression: a systematic review and meta-analysis. J Am Med Inform Assoc. (2024) 31:2394–404. doi: 10.1093/jamia/ocae189

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 29

  MangalikSEichstaedtJCGiorgiSMunJAhmedFGillGet al. Robust language-based mental health assessments in time and space through social media. NPJ Digit Med. (2024) 7:109. doi: 10.1038/s41746-024-01100-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 30

  TrifuRNNemeșBHertaDCBodea-HateganCTalașDAComanH. Linguistic markers for major depressive disorder: a cross-sectional study using an automated procedure. Front Psychol. (2024) 15:1355734. doi: 10.3389/fpsyg.2024.1355734

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 31

  TeferraBGRuedaAPangHValenzanoRSamaviRKrishnanSet al. Screening for depression using natural language processing: literature review. Interact J Med Res. (2024) 13:e55067. doi: 10.2196/55067

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 32

  MenneFDörrFSchräderJTrögerJHabelUKönigAet al. The voice of depression: speech features as biomarkers for major depressive disorder. BMC Psychiatry. (2024) 24:794. doi: 10.1186/s12888-024-06253-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 33

  WeberJWeberMLopez AlcarazJM. Depression diagnosis from patient interviews using multimodal machine learning. Front Psychiatry. (2025) 16:1694762. doi: 10.3389/fpsyt.2025.1694762

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 34

  JinNYeRLiP. Diagnosis of depression based on facial multimodal data. Front Psychiatry. (2025) 16:1508772. doi: 10.3389/fpsyt.2025.1508772

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 35

  XuZGaoYWangFZhangLZhangLWangJet al. Depression detection methods based on multimodal fusion of voice and text. Sci Rep. (2025) 15:21907. doi: 10.1038/s41598-025-03524-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 36

  TrellesARuizTFRojoAP. Systematic review and meta-analysis of explainable machine learning models for clinical depression detection. Behav Sci (Basel). (2025) 15:1476. doi: 10.3390/bs15111476

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 37

  AhmedIBrahmacharimayumAAliRHKhanTAAhmadMO. Explainable AI for depression detection and severity classification from activity data: development and evaluation study of an interpretable framework. JMIR Ment Health. (2025) 12:e72038. doi: 10.2196/72038

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 38

  GohYSSeeQRVongsirimasNKlanin-YobasP. Artificial intelligence in diagnosing depression through behavioural cues: a diagnostic accuracy systematic review and meta-analysis. J Clin Nurs. (2025). doi: 10.1111/jocn.17694

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 39

  LeeDYKimNParkCGanSSonSJParkRWet al. Explainable multimodal prediction of treatment-resistance in patients with depression leveraging brain morphometry and natural language processing. Psychiatry Res. (2024) 334:115817. doi: 10.1016/j.psychres.2024.115817

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 40

  ZhangWMaoKChenJ. A multimodal approach for detection and assessment of depression using text, audio and video. Phenomics. (2024) 4:234–49. doi: 10.1007/s43657-023-00152-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 41

  SadeghiMRicherREggerBSchindler-GmelchLRuppLHRahimiFet al. Harnessing multimodal approaches for depression detection using large language models and facial expressions. NPJ Ment Health Res. (2024) 3:66. doi: 10.1038/s44184-024-00112-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 42

  WangLWangCLiCMuraiTBaiYSongZet al. AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis. NPJ Digit Med. (2025) 8:523. doi: 10.1038/s41746-025-01933-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 43

  KaczmarczykRWilhelmTIMartinRRoosJ. Evaluating multimodal AI in medical diagnostics. NPJ Digit Med. (2024) 7:205. doi: 10.1038/s41746-024-01208-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 44

  KlineAWangHLiYDennisSHutchMXuZet al. Multimodal machine learning in precision health: a scoping review. NPJ Digit Med. (2022) 5:171. doi: 10.1038/s41746-022-00712-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 45

  HeLNiuMTiwariPMarttinenPSuRJiangJet al. Deep learning for depression recognition with audiovisual cues: a review. Inf Fusion. (2022) 80:56–86. doi: 10.1016/j.inffus.2021.10.012

  • Pubmed Abstract
  • CrossRef
  • Google Scholar