Regular monitoring is essential to efectively track mood fluctuations and assess ongoing treatment needs for mood disorders (e.g., identifying early signs of relapse, adjusting therapeutic interventions, and improving longterm outcomes). The current ongoing work aims at assessing the relationships between language and symptom severity in people with bipolar disorders, thus investigating potential mHealth mood detection mechanisms based on speech patterns. Acoustic features included conversational measures for nonverbal language and statistics for prosodic cues. Preliminary results, combining acoustic features and natural language processing (NLP) scores, were promising, somehow discriminating clinical conditions of people with BD when assessing their mood states. This approach may ofer potential benefits for individualized mental health care and early intervention approaches in real-world scenarios.
Bipolar disorder (BD) is a lifelong episodic illness resulting in reduced psychosocial functioning. The
majority of BD cases onset in early adulthood and it is among the leading causes of disability in
workingage adults [
Traditionally, this evaluation has heavily relied on clinical interviews, including an analysis of thoughts and their manifestation in language of people with BD. Focusing on meaning and communication, language has a central role for diagnosis and treatment in BD with speech patterns being LGOBE (G. Carrà)
CEUR
ceur-ws.org
crucial when assessing current experiences, emotions, and thoughts. For instance, pressure of speech
encompassing a high number of words during phonation and a small number of pauses is likely to
be a sign of underlying manic symptoms. Conversely, mood states in depression are characterized by
poverty of speech and a monotone pitch [
Progress in Machine Learning (ML) and Natural Language Processing (NLP) techniques may support
the development of automated systems assessing speech patterns as objective markers of mood states
[
However, there are several barriers in the implementation of mood detection systems in real-world
applications, including high degree of heterogeneity between studies and the use of non-standardized
metrics reporting [
Referring to the process of examining and interpreting the characteristics of spoken output, speech signal analysis was used to identify key patterns from acoustic signals generated during speech. Speech signal analysis was proven efective to characterize mood states of people with BD, thus contributing to individualized approaches including the estimate of various speech features based on signal’s frequency and energy/amplitude [20]. Remarkably, deep learning techniques may significantly advance speech signal processing, enabling more accurate recognition, analysis, and interpretation of individuals’ language, especially for mood detection. Indeed, deep learning encompasses ML techniques that can automatically learn hierarchical representations from data. Core models embrace neural networks (NN) including multiple interconnected layers of nodes (mimicking the structure and functioning of the human brain) as well as convolutional neural networks (CNN), in which the nodes of each layer are clustered [21]. In addition, considering spectrograms in speech analysis systems, the latter may foster automatically learning representations from image data according to a benchmark performance in image classification [ 19]. 3.1. Proposed approach Assessing mood fluctuations based on gold-standard assessments of mania and depression in BD, the proposed approach aimed at exploiting deep learning algorithms for mood states prediction. Eligible participants involved subjects with a diagnosis of BD, aged between 18 and 65 years old from both inpatient and outpatient services. They were approached by specifically trained staf. Subjects unable to provide informed consent, those with vocal or hearing issues were excluded. Speech data were collected and processed through a mobile app that study participants accessed on their smartphones using password-protected access to self-administer verbal performance tasks. In particular, the system embedded in the smartphone was grounded on a cloud-based architecture hosting the system database, the Representational State Transfer (REST) Application Programming Interfaces (API), and the backend processing modules.
Considering mood states variability and relevant speech signal segmentation, we aimed to combine two diferent approaches for speech analysis. First, highlighting speaking segments from raw audio data, speech was automatically processed through speech recognition and quantities representing voice characteristics (i.e., acoustic features) were estimated. By leveraging Parselmouth module as a bridge to speech-to-text preprocessing and related Praat’s built-in functions, basic acoustic features were computed (e.g., fundamental frequency, harmonics-to-noise ratio, jitter and shimmer). Speech rate, verbal task duration, and phonation duration were also considered. In addition, acoustic features were integrated with both standard and novel NLP scores for linguistic components of speech according to distributional semantic models (e.g., estimating information on processing speed and capturing both lexical overlap and semantic similarity in the spoken output). As a whole, previous studies showed an enhanced performance considering combined features [22]. With predictive accuracy as the primary goal rather than understanding the exact contributions of individual features (both speech signal-derived and NLP-extracted features), the models’ architecture was based on a feedforward neural network with fully connected layers (Rectified Linear Unit -ReLU- activation in two hidden layers and sigmoid activation in the output layer). The Adaptive Moment Estimation (adam) optimizer was used with the model seeking to minimize cross-entropy loss. The model stops the training if the validation loss does not improve for 10 epochs. A 5-fold cross-validation was used with each fold providing metrics that are averaged for the final evaluation based on 80% of the data for training and 20% for testing.
On the other hand, by considering the feasibility of performing acoustic signalling analysis as a function of frequency and time, this ongoing work aimed to focus on potential speech segments from spectrograms for image data classification. Higher energy against low energy regions (e.g., pauses) in spectrograms may be distinguished by darker/lighter colours. Consistently, spectrograms may be able to display the properties of a changing signal through a series of snapshots according to segment length with speech corpora capturing tones, emotions, rhythms among signals beyond content of speech.
Based on the caseload of the ASST Nord Milano Mental Health Care Trust, 37 subjects with BD were involved, while enrolment is still active. Involved participants mainly lived alone or with family, were globally educated, but unemployed. They were likely to report severe depressive symptoms (MADRS score ≥19, 46%) and just a few had severe manic features (YMRS score ≥20; 24%). Therefore, we preliminarily focused on MADRS assessment (Table 1), taking into account sex-specific discriminating ability of NLP-based and acoustic features for mood states prediction.
appearance or not. Includes low spirits, despondency or the feeling of being beyond help and without hope. Rated according to intensity, duration and the extent to which the mood is reported to be influenced by events.
tension mounting to either panic, dread or anguish. Rated according to intensity, frequency, duration and the extent of reassurance called for.
Model performance was comprehensively evaluated according to accuracy and Receiver Operating Characteristic (ROC) Area Under the Curve (AUC) estimates to assess the ability of the models to correctly classify mood states. In particular, two classes for symptom severity were considered for classification (i.e., severe/not severe). Neural network models developed -including diferent sets of speech features and considering a chance level of 0.5 - showed varying levels of performance (Table 2). Notably, NLP features, such as mean intraword time and semantic similarity between words, provided satisfactory results as compared to models relying on acoustic features only (e.g., fundamental frequency, and jitter- and shimmer-related features). However, further analysis revealed that sex-based diferences influenced the models’ ability to accurately discriminate between mood states, thus suggesting that sex may modulate the expression of mood in both linguistic and acoustic features.
Figure 1 shows sample speech spectrograms of two study participants with diferent levels of symptom severity. While sampled spectrograms might have relatively limited representativeness, related visual quality -based on the identification of key features- can indicate how well this captures the underlying patterns in the data, thus providing a useful representation of the signal for further analyses. Indeed, visual inspection of relevant spectrograms of people with BD revealed likely distinct acoustic patterns when assessing mood states, possibly reflecting symptom severity. Specifically, sample patterns from verbal tasks of participants were likely to exhibit a diferent number and duration of pauses with varying speech rate and mean intraword time (i.e., presence/absence of pressure of speech) as well as potential diferences in signal frequency and intensity. However, according to existing evidence, no standardized feature framework is available. Therefore, considering the uncertainty about which features should be extracted as well as the risk of bias due to potentially missing information, these results suggested the need to focus on speech segments more in detail, by pre-processing and analyse image data directly from speech spectrograms for image data classification purposes, possibly corroborating the role of speech features as digital markers of mood states in people with BD.
The current work explored the use of speech signal analysis to map symptom severity in people with BD when assessing mood states using neural networks. Preliminary results showed relatively adequate accuracy for prediction, though with varying model performance according both to features selected and subgroups (e.g., sex). As a whole, combining acoustic signals and NLP can be a feasible, clinically useful, application in mental healthcare with acoustic features representing novel markers for mood states.
Future work will focus on capturing a higher degree of complexity of the underlying data distribution, by extending in a larger, more diverse sample of people with BD, accounting for potential confounders, and exploring speech segments more in detail based on speech spectrograms. This would enable a better understanding of how these methods can operate in real-world settings, particularly with regard to their potential for integration into clinical practice. Indeed, feature information may be complemented by feeding the spectrograms directly into the models as input data, using short voice segments to develop deep learning algorithms from speech spectrograms for classification purposes (e.g., CNN). In addition, speech signals are often mixed with other signals and both frequency and amplitude are likely to change over time resulting in non-stationary and non-linear signals. Therefore, empirical mode decomposition (EMD) related approaches, by breaking the signal down into components that reflect relevant changes, could ofer valuable insights [ 23, 24, 25]. Consistently, smartphone-based approaches for speech processing show potential for real-time monitoring (or detection) of mood states in BD likely relying on ecological momentary assessments with NLP and artificial intelligence (AI) being promising for smart mental healthcare over time [26, 27, 28]. Based on mHealth technologies, this approach would help devising human-centered mood remote monitoring based on symptom patterns from speech possibly with significant clinical impact. Ecologically valid long-term mood monitoring of individuals with bipolar disorder using speech, in: Proc. IEEE Int. Conf. Acoust. Speech Signal Process, 2014, pp. 4858–4862. doi:10.1109/ICASSP. 2014.6854525. [18] S. Kapoor, T. Kumar, Fusing traditionally extracted features with deep learned features from the speech spectrogram for anger and stress detection using convolution neural network, Multimed Tools Appl 81 (2022) 31107–31128. doi:10.1007/s11042-022-12886-0. [19] H. Meng, T. Yan, F. Yuan, H. Wei, Speech emotion recognition from 3D log-mel spectrograms with deep learning network, IEEE Access 7 (2019) 125868–125881. doi:10.1109/ACCESS.2019.2938007. [20] A. Z. Antosik-Wójcińska, M. Dominiak, M. Chojnacka, K. Kaczmarek-Majer, K. R. Opara, W. Radziszewska, A. Olwert, Ł. Święcicki, Smartphone as a monitoring tool for bipolar disorder: a systematic review including data analysis, machine learning algorithms and predictive modelling, Int J Med Inform 138 (2020). doi:10.1016/j.ijmedinf.2020.104131. [21] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (2015) 436–444. doi:10.1038/ nature14539. [22] H. Naderi, B. H. Soleimani, S. Matwin, Multimodal deep learning for mental disorders prediction from audio speech samples, in: 33rd Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada, 2019, pp. 4858–4862. arXiv 1909.01067v5. [23] P. Marti-Puig, E. Gallego-Jutglà, G. Masferrer, J. Solé-Casals, A New Algorithm for Speech Enhancement Based on Multivariate Empirical Mode Decomposition, Artificial Intelligence Research and Development, IOS Press., 2018, pp. 247–255. doi:10.3233/978-1-61499-918-8-247. [24] C. Sun, H. Li, L. Ma, Speech emotion recognition based on improved masking EMD and convolutional recurrent neural network, Front Psychol 13 (2023) 1075624. doi:10.3389/fpsyg.2022. 1075624. [25] U. Souza, J. Escola, T. Vedovatto, L. Brito, R. Lemos, Bidirectional EMD-RLS: Performance analysis for denoising in speech signal, Journal of Computational Science 74 (2023) 102181. doi:10.1016/j. jocs.2023.102181. [26] E. Kerz, S. Zanwar, Y. Qiao, D. Wiechmann, Toward explainable AI (XAI) for mental health detection based on language behavior, Front Psychiatry 14 (2023) 1219479. doi:10.3389/fpsyt. 2023.1219479. [27] B. Zhou, G. Yang, Z. Shi, S. Ma, Natural language processing for smart healthcare, IEEE Rev
Biomed Eng 17 (2024) 4–18. doi:10.1109/RBME.2022.3210270. [28] W. Hinzen, L. Palaniyappan, The ’L-factor’: Language as a transdiagnostic dimension in psychopathology, Prog Neuropsychopharmacol Biol Psychiatry 131 (2024) 110952. doi:10.1016/j. pnpbp.2024.110952.