The assessment of symptom severity plays a crucial role in the clinical management of mental disorders, being pivotal in diagnosing and monitoring the mental well-being of patients [1]. Traditionally, this evaluation has heavily relied on clinical experience, sometimes supported by questionnaires and rating scales during in-person visits. However, advancements in Machine Learning (ML) and Natural Language Processing (NLP) techniques offer the potential for automated systems that can support in assessing measures of symptom severity in dialogues between psychiatrists and the growing number of patients. In particular, the evolving landscape of prompt engineering techniques applied to Large Language Models (LLMs) presents a novel avenue for developing such kind of systems, to better support psychiatric assessment practices in the future. This study, in particular, embarks on the task of automatically mapping psychiatrist-patient dialogue content to the Montgomery-Åsberg Depression Rating Scale (MADRS) [2], a widely accepted instrument for evaluating depression severity, through the potential of recently developed generative Artificial Intelligence (AI) models [3]. To establish a foundation, a manual mapping process performed by clinical experts is employed to establish connections between question-answers from some psychiatrist-patient dialogues and the corresponding items of the MADRS questionnaire, together with the corresponding scores (both at the individual item level and the global level). This manual mapping serves as a benchmark for subsequent comparison with results obtained from the considered AI-based approaches.
In a first approach, distinct prompt engineering techniques applied to LLMs are leveraged to compute depression severity scores for each MADRS item. Each item is devoted to assessing a different symptom domain, such as sadness, inner tension, reduced sleep, etc., rated on a scale from 0 to 6, with higher scores indicating more severe depressive symptoms. The computed scores are then further aggregated to provide an overall assessment, ranging from 0 to 60, with higher scores indicating more severe depression. In a second approach, we evaluate the effectiveness of using prompts to directly compute the overall depression severity score.
This study serves as a preliminary step to explore the feasibility, in the future, of creating an advanced conversational system that generates questions and analyses responses to automatically assess symptom severity levels. The obtained results illustrate that the proposed approaches and the best models tested have an accuracy of about 70% in making the mapping between conversation and MADRS scores, with a pretty high correlation. While not optimal, this result appears encouraging in the belief that refinements on the models (via fine-tuning) and prompts could lead to higher results and pursuit of the goal of developing a fully automated system.
The urgent need for innovation around access and quality of mental health care has become clear in the last few years [4]. More and more mental health-related digital strategies for therapeutic approaches have been offered via ML and, in general, AI models, thus contributing to the development of detection systems for mental disorders, e.g., [5,6,7].
However, although significant progress has been made in the field, there are several barriers in the implementation of detection systems in real-world applications, including a need for increased transparency and replication [8]. Moreover, the literature is sparse with a high degree of heterogeneity between studies and the use of nonstandardized metrics reporting [9]. In addition, several areas remain understudied, including the use of these approaches among people suffering from mental disorders such as depression. Nonetheless, a few studies analyzed automated approaches for evaluating depression.
A recent study trained ML models to diagnose depression from spontaneous responses of 113 outpatients using interviews by experienced physicians that were first audio-recorded and transcribed verbatim. The study showed automated depression diagnosis based on interviews as a feasible approach [10]. The use of transcribed autobiographical memory interviews was also considered for patients with treatment-resistant depression treated with psilocybin [11]. Quantitative speech measures were computed using the interview data from 17 patients and 18 untreated age-matched healthy control subjects, and an ML algorithm was developed to classify between controls and patients and predict treatment response. Results showed that speech analytics and ML successfully differentiated individuals with depression from healthy controls and identified treatment responders from nonresponders with a significant level of accuracy and precision. More generally, question-based computational language assessment, based on self-reported and freely generated word responses, analyzed with AI, has been shown as a potential tool that may complement rating scales and evaluate mental health issues in clinical settings [12]. A recent systematic review highlighted preliminary favorable evidence about the use of conversational agents (i.e., tools providing feedback to user input related to wellbeing and mental health queries) and their promising role in screening, assessment, diagnosis, and treatment of mental disorders, including the effective identification of people with depressive symptoms [8,13,14]. For instance, discreet text interfaces possibly allowed participants to feel more comfortable using conversational agents in public [15].
Although these approaches appear to ensure optimal control over conversation flow and topics benefiting users and providers, a pre-defined response range may decrease usability in a diverse range of clinical settings with different risks such as possibly disrupting the therapeutic alliance [15]. Indeed, a feasible option for developing a mass screening integrated approach for early detection of depression is intended as a means of assisting with automation and concealed communication with verified scoring systems rather than replacing clinical interviews [16]. Moreover, the diversity of outcomes and the choice of outcome measurement instruments employed in studies on conversational agents for mental health point to the need for an established minimum core outcome set and greater use of validated instruments [17]. Therefore, an enhanced personalization of conversational agents leveraging the interdisciplinary use of NLP techniques to better understand the context of the conversation about vulnerable experiences related to depressive symptoms -with a more human-like approach -appears desirable [18].
LLMs are advanced AI systems [19], which possess the capability to generate human-like text across a wide range of topics, and thus seem to be the most suitable tool for solving the literature problem enunciated above. However, to accomplish a particular task, there is the need for a process for crafting specific instructions or prompts to guide these models; such a process is known as prompt engineering [20], and is gauging importance in recent years in medicine [21].
The This means designing appropriate prompt templates for each prompting technique with respect to each approach.
We ask LLMs appropriately guided by prompts to generate a score for each item of the MADRS. Such items and their descriptions are illustrated in Figure 1, while ZS and CoT prompt templates are detailed in the following. Zero-Shot Learning. The model is simply asked to generate a score for each item of the MADRS. These items are specified in the template, as follows:
Given the following document containing a conversation between a physician and a patient, denoted by M and P respectively, following the Montgomery-Åsberg Depression Rating Scale (MADRS), answer me with the severity score, from a minimum of 0 (symptom absent) to a maximum of 6 (extremely severe), for the following item only: [item title, description]. Answer me only with a value between a minimum of 0 and a maximum of 6 related only to the described label. Below is the document to be analyzed:
This template is repeated for each of the 10 items of MADRS, and [item title, description] contains the title and description shown in Figure 1 for each item, for example: Reduced sleep, representing the experience of reduced duration or depth of sleep compared to the subject's own normal pattern when well. Once the scores for each item are obtained, they are simply added together to obtain the overall score.
CoT Learning. In this preliminary work, the CoT approach is based on simply asking the model to provide a motivation before performing the task. This helps the model make a more informed decision than the ZS scenario. Therefore, the CoT template used is as follows:
[ZS "local" template] + Provide the rationale before answering.
Also in this case, the scores for each item are summed up to obtain the overall score.
Here, LLMs are appropriately guided to directly generate the overall depression score with respect to MADRS.
Zero-Shot Learning. The ZS template employed in this global approach to computation is as follows:
CoT Learning. CoT learning in the global approach uses the ZS "global" template in which reasoning is required before providing the answer:
[ZS "global" template] + Provide the rationale before answering.
In this section, we present the results of the comparative evaluation of the local and global approaches, in relation to the various proposed prompt engineering techniques (and thus, regarding the different templates used). Firstly, we introduce the dataset employed in the evaluations and the technical characteristics of the implemented models.
It is well understood, especially in such a delicate field as psychiatry, that dealing with patient data is rather complex and ethically sensitive. For this reason, for this preliminary study, a team of medical experts generated a small dataset in which clinicians took on the roles of both the doctor and the patient. This was done to create typical conversations regarding various levels of depression severity, namely: severe depression, moderate depression, mild depression, and absence of depression. In total, 10 doctor-patient conversations were generated in Italian, with at least 3 conversations for the first three previously outlined severity levels. Clinicians also labeled the questions and answers against the corresponding items of the MADRS and provided both item-level and global scores for the entire conversation.1
To assess the effectiveness of generative models in addressing the considered problem, various LLMs were tested. These models were trained on diverse datasets, tailored for a multilingual context, given that our psychiatrist-patient conversations are in Italian. In particular, the following models were used: GPT-3.5: GPT-3.5-turbo-0613, it is an iteration of the Generative Pretrained Transformer (GPT) model developed by OpenAI. It is an advanced version of its predecessor, GPT-3, with improvements in various aspects such as model architecture, training data, and fine-tuning techniques. GPT-4: GPT-4-0613, it is a large multimodal model (accepting image and text inputs, emitting text outputs).2 Mistral: Mistral-7B-Instruct-v0.2, it is an instruct fine-tuned 7B LLM, trained mainly on English data, but also acquainted with Italian during its pretraining phase [22]. Mixtral: Mixtral-8x7B-Instruct-v0.1, it is a pretrained generative Sparse Mixture of Experts model, trained mainly on 5 languages including Italian. It has 46.7B total parameters but only uses 12.9B parameters per token.3 Dante: DanteLLM_instruct_7b-v0.2-boosted, it is a recent state-of-the-art Italian LLM based on the 7B Mistral model. 4 Hermes: Hermes7b_ITA, it is a 7B LLM trained on a 120K instruction/answer dataset in Italian. It is based on Nous-Hermes-llama-2-7b LLM, a version of meta/Llama-2-7b fine-tuned to follow instructions. 5
The results obtained measure the effectiveness of the above-mentioned models, in conjunction with the appropriate prompting templates, in correctly predicting the item-level scores and overall score of each conversation compared with those assigned by the medical experts. They are illustrated in terms of accuracy (Acc.), Pearson (P.), and Spearman (S.) correlation coefficients.
Tables 1 and 2 show some results of the prompts and LLMs models applied to the local computation approach. It can be seen that from the results in Table 1, especially in terms of accuracy, the local approach does not provide satisfactory overall results. However, a substantial improvement can be appreciated when models are asked to explain the reasons for their choices (CoT), and in particular for the Hermes model. Regarding the correlation coefficients of Person and Spearman, we can observe how these are globally quite high, improving in the CoT scenario for models trained on larger amounts of data and decreasing on smaller ones.
Correlation results for each MADRS item in the local CoT scenario. GPT-3.5 0.61 0.80 0.35 0.24 0.48 0.56 0.73 0.81 0.74 0.79 0.60 0.66 0.54 0.58 0.17 0.24 0.31 0.41 0.83 0.87 GPT-4 0.65 0.51 0.61 0.50 0.70 0.67 0.89 0.79 0.90 0.89 0.18 0.36 0.83 0.76 0.47 0.37 0.84 0.83 0.95 0.96 Mistral 0.15 0.20 0.64 0.78 0.53 0.21 0.71 0.79 0.21 0.20 0.40 0.54 -0.34 -0.37 0.31 0.31 0.82 0.82 0.94 0.93 Mixtral 0.46 0.48 0.91 0.88 0.73 0.43 0.76 0.69 0.84 0.90 0.21 0.35 0.72 0.64 -0.52 -0.36 0.36 0.39 0.83 0.87 Dante -0.49 0.49 0.66 0.68 0.75 0.47 0.50 -0.78 -0.76 -0.08 -0.08 -0.25 -0.05 -0.04 0.09 0.11 0.11 0.24 0.25 Hermes 0.57 0.56 -0.25 -0.61 0.06 0.24 0.07 0.01 -0.16 -0.22 -0.25 -0.32 0.30 0.17 0.24 0.16 0.18 0.29 -0.02 0.22
Table 3 shows the results of the prompts and LLMs models applied to the global computation approach. The results in this case show that an accuracy of around 70% can be achieved. It is particularly interesting to note how the best models are the GPT-based in the ZS case, while it is Dante in the CoT case, which instead turns out to be one of the worst using a ZS technique. Person and Spearman correlation coefficient results illustrate a significant increase in correlation in the smaller models in the CoT scenario, with variable fluctuations in the case of the larger models.
Compared to the approaches, prompt engineering techniques, and LLMs considered, it is clear that the use of the global approach is superior to the local one. This would seem to suggest that LLMs have a greater chance of success with respect to the task considered when the conversation is considered to produce the global MADRS score, without the model being asked to generate MADRS item-based scores to be later aggregated. However, we operated in a context in which we did not provide specific examples of the model according to a Few-Shot strategy, which need to be investigated in the future.
As it emerges from Table 2, referring to the local computation approach in the CoT scenario, the correlation with respect to the scores predicted in the individual items is generally not very high, although it is objectively better in some specific items such as #4 (i.e., reduced sleep, for the models trained on more data), #10 (i.e., suicidal thoughts, again for larger models). The smaller, Italianspecific models do not correlate well on this task.
Concerning Figure 2, illustrating the confusion matrix referring to the global computation approach for the Dante model performed in the CoT scenario, we can observe how the model does not confuse depression severity classes that are too distant from each other.
This study explored the utilization of generative Artificial Intelligence (AI) models for automatically mapping psychiatrist-patient dialogue content to the Montgomery-Åsberg Depression Rating Scale (MADRS). Two distinct approaches were investigated: the application of prompt engineering techniques to compute symptom severity scores for each MADRS item, and the direct calculation of the overall depression severity score. The results demonstrated that the proposed approaches, coupled with the best-performing models, achieved an accuracy of approximately 70% in mapping conversations to MADRS scores.
Though the current accuracy shows promise, there is room for improvement. Future studies could refine models, improve prompt techniques, explore new methods, and use more data sources. This could lead to an automated system that generates questions and evaluates symptom severity from dialogue analysis.