A Cross-Domain Framework for Emotion and Stress Detection Using WESAD, SCIENTISST-MOVE, and DREAMER Datasets

Ahmad Almadhor^1*Stephen Ojo²Thomas Nathaniel³Kingsley Ukpong⁴Shtwai Alsubai⁵Abdullah Al Hejaili⁶

• ¹Jouf University, Sakakah, Saudi Arabia
• ²Anderson University, Anderson, United States
• ³University of South Carolina, Columbia, United States
• ⁴Federal University Oye-Ekiti, Oye, Nigeria
• ⁵Prince Sattam bin Abdulaziz University, Al Kharj, Saudi Arabia
• ⁶University of Tabuk, Tabuk, Saudi Arabia

Emotional and stress-related disorders pose a growing threat to global mental health, emphasizing the critical need for accurate, robust, and interpretable emotion recognition systems. Despite advances in affective computing, existing models often lack generalizability across diverse physiological and behavioral datasets, limiting their practical deployment. This research addresses this gap by proposing a cross-dataset deep learning framework that leverages transferable feature representations and temporal modeling to enhance emotion, stress, and activity state recognition. This study presents a dual deep learning-based framework for mental health monitoring and activity monitoring. The first approach introduces a framework for stress classification that uses a 1D-CNN model trained on the WESAD dataset. This model is then fine-tuned using the ScientISST-MOVE dataset to detect daily life activities based on motion signals, and it is used as transfer learning for a downstream task. An explainable AI technique is used to interpret the model's predictions, while class imbalance is addressed using focal loss and class weighting. The second approach employs a temporal conformer architecture combining CNN and transformer components to model temporal dependencies in continuous affective ratings of emotional states based on valence, arousal, and dominance (VAD) using the DREAMER dataset. This method incorporates feature engineering techniques and models temporal dependencies in ECG signals. Both approaches are supported by additional modules such as clustering and rule-based emotion interpretation for richer analysis. The deep learning classifier trained on WESAD biosignal data Almadhor et al. Running Title achieved 98% accuracy across three classes, demonstrating highly reliable stress classification. The transfer learning model, evaluated on the ScientISST-MOVE dataset, achieved an overall accuracy of 82% across 4 activity states, with good precision and recall for high-support classes. However, the explanations produced by Grad-CAM appear uninformative and do not clearly indicate which parts of the signals influence the prediction. The conformer model achieved an R2 score of 0.78 and a rounded accuracy of 87.59% across all three dimensions, highlighting its robustness in multi-dimensional emotion prediction. The framework demonstrates strong performance, interpretability, and real-time applicability in personalized affective computing.