Multimodal AI for depression detection: integrating facial expressions and semantic analysis in outpatient care
M.-F. Lin1, S.p. Konara mudiyanselage2, H.-H. Tseng2,3
1The National Cheng Kung University, Department of Nursing, Tainan, Taiwan, 2The National Cheng Kung University, Institute of Behavioral Medicine, Tainan, Taiwan, 3The National Cheng Kung University, Department of Psychiatry, Tainan, Taiwan
Background: Depression remains underdetected in outpatient settings due to reliance on self-report measures, stigma, and time constraints in routine clinical care. Advances in artificial intelligence (AI) enable objective, scalable approaches for identifying depressive symptoms using behavioral and linguistic markers. However, evidence for the clinical utility of multimodal AI systems in real-world outpatient populations remains limited.
Aims & Objectives: This study aimed to evaluate the predictive performance and clinical relevance of two AI-driven depression detection systems—iSeeME (facial-expression analysis) and EDDTW-V2 (semantic narrative analysis)—in identifying depressive symptoms among high-risk outpatients, and to examine their complementarity with standard clinical assessments.
Method: A cross-sectional study was conducted with 62 outpatients recruited from psychiatric and surgical oncology clinics at a tertiary medical center in Taiwan. Participants completed standardized clinician-rated and self-report measures, including the Hamilton Depression Rating Scale (HDRS-17), Beck Depression Inventory-II (BDI-II), and Patient Health Questionnaire-9 (PHQ-9). Facial-expression data from video-recorded narratives were analyzed using the iSeeME system, while transcribed narratives were processed using EDDTW-V2. Predictive validity was assessed using accuracy metrics, correlations with clinical scales, and k-means cluster analysis to identify distress phenotypes.
Results: Of the 62 participants, 39 met criteria for clinical depression (HDRS ≥7). The iSeeME system demonstrated superior predictive performance (accuracy = 0.77, precision = 0.76, recall = 0.85, F1-score = 0.81) compared with EDDTW-V2 (accuracy = 0.71, F1-score = 0.73). iSeeME showed stronger correlations with BDI-II (r = 0.44, p < 0.01) and PHQ-9 (r = 0.34, p < 0.01), reflecting sensitivity to affective-behavioral symptoms. EDDTW-V2 demonstrated moderate associations with depression scales and captured complementary cognitive-linguistic features. Cluster analysis identified a high-psychological-distress subgroup characterized by younger age, single status, fewer children, and predominance of psychiatric outpatients.
Discussion & Conclusions: AI-driven facial and semantic analysis tools show promise as complementary approaches to traditional depression assessments in outpatient care. Facial-expression–based models may be particularly sensitive to observable affective symptoms, while semantic systems capture cognitive and linguistic markers that may be underreported. Integrating multimodal AI tools into outpatient workflows could enhance early detection, triage, and monitoring of depression, especially in high-volume clinical settings. Further longitudinal and cross-cultural validation is warranted.
In sessions
