This article presents an innovative approach to predicting depression relapse in adolescents. Adolescentsíntensive use of video and voice-based smartphone apps presents a rich, multimodal dataset that can be utilized for this purpose. This work uses a dataset from the Depression Early Warning study conducted at the Center for Addiction and Mental Health. After using a pre-trained Inception ResNet to generate embeddings of video frames, the proposed framework integrates this with synchronized speech data. These embeddings are fused with audio features, resulting in a multimodal dataset. The combined features are processed through a Long Short-Term Memory model and a fully connected network to predict relapse of depression. An average accuracy of 0.80 highlights the efectiveness of the proposed multimodal approach and underscores its potential to efectively predict depression relapse in adolescents.
CEUR ceur-ws.org
Depression is a worldwide, prevalent mental health
disorder among adolescents. The recognition and treatment
of adolescent depression hold paramount significance
due to its association with substantial risks, notably
suicide, which stands as the fourth leading cause of death
within this demographic [
veys that may be subject to recall bias. This approach also
does not allow for timely interventions [
The use of speech and video analysis for depression prediction represents an innovative and promising approach in mental health research. Analyzing speech patterns and facial expressions can provide valuable insights into an individual’s emotional and mental state. Below is a review on the use of speech and video for depression prediction.
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License information about the mental state of a person and vocal
Attribution 4.0 International (CC BY 4.0).
source features can be used as biomarkers of depres- in [11]. The framework combined spatial information
sion severity [
Behavioral analysis of facial expressions has been stud- The model’s performance was further analyzed based on ied as a source for eliciting the underlying emotional diferent modalities where the model trained with audio state [10]. Computer vision methods have been used to features performed better than the model trained with analyze facial expressions and gestures to predict the un- visual features. The best performance was achieved by derlying mental health state of users [11]. A framework the model trained on the audiovisual features. The work for estimating depression levels from video data using a of Othmani et.al. in [15] used deep learning techniques two-stream deep spatiotemporal network was introduced to recognize depression and predict relapse from audio and visual cues extracted from videos of clinical inter- interviewed by the coordinator during their initial visit views. It involves a correlation-based anomaly detection and followup sessions. During recorded Zoom sessions, framework that compares the audiovisual patterns of the coordinator asked them 10 open-ended questions depression-free subjects to those of depressed individu- about their past activities and mood, resulting in 2-10 als. The correlation between the audiovisual encoding minutes of video data per session. This dataset was colof a test subject and a deep audiovisual representation lected as part of an ongoing research study at CAMH and of depression is computed to monitor depressed subjects is unavailable to the public. and predict relapse. The approach achieves promising results, with an accuracy of 80.99% and 82.55% for relapse 4.2. Definition of Relapse depression prediction on the DAIC-Woz dataset.
The existing landscape of research on adolescent de- While there are many definitions of relapse in depression, pression has made significant strides in understanding a commonly accepted one is given by [16] which defines the onset and symptoms of depression in this age group. a relapse in adolescents as observing a CDRS score of at However, there is a notable gap in the ability to efec- most 28 during at least 12 weeks of treatment followed by tively predict depression relapse from audio and video an increase in CDRS to at least 40 for at least two weeks. modalities. By incorporating synchronized video and The first period of 12 weeks corresponds to a remission speech data, this research captures a broader spectrum of stage where the depressed adolescent does not exhibit behavioral and emotional cues that might signify impend- symptoms but has not yet completed treatment. The ing relapse in adolescents. The synchronization ensures period of two weeks corresponds to a depressed episode. that both modalities are aligned, allowing for a detailed In this study, there can be at least a three month break examination of facial expressions, body language, and between followup visits. Hence, the timing aspect of Kenspeech patterns simultaneously. nard’s definition must be accordingly modified to adhere to the provided data. This work proposes a definition of relapse as a period of at least one visit with a CDRS score 3. Problem Formulation of at most 40 followed by one visit with a CDRS score of at least 40.
There are three main stages that make up the methods of this pipeline. The first consists of preprocessing the video and audio data and organizing them such that the two modalities are aligned and the labels are balanced. The second involves training models on random subsets of the training data. In the final stage, the final model that was trained on the training set is evaluated on multiple test sets, and the performance metrics are averaged across each set. A diagram summarizing the pipeline is shown in Figure 1. 4.3.1. Stage 1: Data Preparation
only the participant is speaking. Since the interviews are conducted via Zoom, the videos are also cropped such that only the participant’s face is visible. Several spectral features are extracted from the audio data using the Python package libRosa [17]. They include the MFCCs, fundamental frequency, chromagrams, power spectral density, and spectral rollof. These features are computed over a rolling window that is applied across the video. The amount of overlap is chosen such that the number of windows matches that of the video frames and are evenly spread out across the video.
Our work aims to classify fused video and speech features for the classification of data that is measured before a relapse event. This entails a binary classification task where the two classes include “relapse sometime in the future” and “non-relapse”. This problem is significantly diferent from detecting the presence of depression or predicting a certain depression rating scale score. The problem of relapse prediction is more complex since it involves the direct prediction of a clinical event within a population of adolescents who are already diagnosed with Major Depressive Disorder.
depression early warning study that was run in the Centre for Addiction and Mental Health (CAMH). It includes 80 video interviews collected from 52 adolescents aged 12-21 who were all diagnosed with Major Depressive Disorder.
up to 7 followup visits, each spaced apart by 3-12 months.
During each visit, participants were assessed by a trained research coordinator and psychiatrist, providing psychiatric evaluations of their depressive states via the Children’s Depression Rating Scale (CDRS). Participants were
Spectral Feature Extraction
Train Fold 1 Train Fold 2 Train Fold N Test Fold Inception ResNet LSTM
In the provided dataset, there is a significant class ing an LSTM and a fully connected network. The LSTM imbalance where the non-relapse data is heavily over- is used to process 16 consecutive frames of features at a represented (96.25% non-relapse). In order to not bias time, and the resulting hidden state is then fed into the the training of the models and the evaluation metrics (de- fully connected network to be classified as either relapse scribed in the next two sections), several training folds are or non-relapse. During the training process, random segprepared alongside a test fold. The folds are constructed ments of 16 frames are sampled from the training video by first randomly selecting a proportion of relapse video clips in order to not bias the training of the network clips to use in the test fold. This proportion is chosen towards a certain class. to be 30%, and it is computed based on the number of The training process is applied to each train fold, and frames within each video clip. A random selection of within a given fold, it is repeated for eight epochs. Afnon-relapse clips are chosen to match the number of ter the AudioVisual Network is trained on a given fold, frames of the relapse ones (rounded to the nearest whole its saved parameters are used to continue training the number of clips). This completes the test fold which network on a new fold. This is repeated until all train is reserved for Stage 3 of the pipeline. The rest of the folds are exhausted. This allows the network to train on relapse subjects are assigned to be used by train folds the entire training dataset while still keeping the classes in Stage 2. Non-relapse video clips are randomly sam- relatively balanced. pled without replacement where the number of clips is selected to match the number of frames of the relapse 4.3.3. Stage 3: Evaluation of Models subjects. Each random sample of non-relapse clips makes up another train fold, and this process is repeated until all non-relapse clips are used.
ture (+InceptionResNet), is evaluated on the test fold that was reserved in Stage 1. A receiver operating character4.3.2. Stage 2: Training of Models istic (ROC) analysis is carried out on the predictions and ground truth labels. The optimal threshold of the ROC The video frames are fed into an InceptionResNet model curve is selected by choosing the point that maximizes that was pre-trained on VGGFace2 [18], a large-scale face the diference between the true and false positive rats. dataset. The resulting embeddings from this network are This threshold is used to compute the MAE. then fused with the spectral features of the audio data. The entire process of training the models and evaluatThe resulting fused features are then fed into a neural ing the final one on a test fold is carried out for 10 sets network module (named AudioVisual Network) contain- of folds. This is to ensure that the reported metrics are not biased toward a certain set of subjects. The resulting performance metrics are averaged across all of the test folds.
Table 1 shows the results of evaluating the trained model on the 10 test folds. Each accuracy and MAE measure was reported after finding the optimal threshold of the ROC curve.
An average accuracy of 0.80 shows that video and speech data are relatively promising in the prediction of relapse in adolescent depression. In previous work by Othmani et al. [15], the authors also predicted relapse of depression using video and speech data. Similar to our work, they also yielded accuracies at around 0.8. To the best of our knowledge, this is the only other work that used video and speech to predict relapse of depression. Our work diferentiates from Othmani et al. in two significant ways: 1) our study focuses on adolescents and 2) the source of our data is from non-clinical interviews. These interviews allow for more conversational topics that may better mimic a real-life situation in an adolescent’s everyday life. While the target population for this work includes adolescents, this framework can be extended to other depressed populations. of gender-based analysis is a notable limitation, potentially overlooking important nuances in how depression manifests across diferent genders.
Future work in predicting depression from audiovisual features will prioritize the development of gender and context aware models. Moreover, given the longitudinal nature of the study, a promising avenue for future work is to exploit the temporal nature of data to track changes in audiovisual features over long periods of time. Employing an overarching time series model could enhance the understanding of the dynamic nature of depression, allowing for the development of more adaptive and personalized prediction models.
Another way to extend this work is to combine other objective sources of data that can be collected simultaneously with video and speech. One such modality includes wearable technologies, and there have been several studies on using them for the prediction of depression [19]. Using similar techniques, it may be possible to fuse audiovisual features and those derived from wearables to create a more robust predictor of adolescent depression relapse. Finally, we intend to evaluate our work using publicly available audio/video depression datasets such as AVEC. Fold 1 2 3 4 5 6 7 8 9 10 Average
MAE 0.077 0.28 0.21 0.23 0.12 0.26 0.17 0.23 0.17 0.27 0.21
Accuracy 0.92 0.72 0.79 0.77 0.88 0.74 0.83 0.77 0.83 0.73 0.80
Predicting depression from audiovisual features encounters various challenges. The subjectivity of depression labels and the heterogeneous nature of this condition make it dificult to develop a universally applicable model. Additionally, there may be ethnic and cultural biases in the data that may have impacted the model’s generalizability. This work did not consider the context within which interviews were conducted. Furthermore, the exclusion