The recent advancements in deep learning have led to a myriad of approaches for medical diagnosis and assistance. Some topics such as data labelling, data curation, human-in-the-loop, explainability or privacy-preserving methodologies are hot topics for applied machine learning in the healthcare context. In this domain, we normally expect a three-way interaction (doctor-system-patient), so that interfaces play a crucial role. Remotely managed VR (Virtual Reality) systems help us to enhance communication and feedback between doctors and patients in situations where in-person assistance is not feasible. Moreover, the recent breakthroughs with LLMs (Large Language Models) enable us to use natural language as additional interface, considering NLU (Natural Language Understanding) for intent recognition, ASR (Automatic Speech Recognition) and TTS (Text-To-Speech). In the context of the CEL.IA network, we present IVAMED (Intelligent Virtual Assistant for MEdical Diagnosis), a chatbot-oriented application in a VR environment for remote medical assistance. We tackle the situation in which face-to-face assistance is not possible. We provide the tools for remote interaction and guided diagnosis. We propose the evaluation of the MoCA (Montreal Cognitive Assessment) test for early detection of MCI (Mild Cognitive Impairment) and the BDI (Beck Depression Inventory) test for measuring characteristic attitudes and symptoms of depression.
cedures and preventive measures, which can empower patients to take better care of themselves [9, 10]. Chatbots In recent years, there has been a significant rise in the can also assist healthcare professionals by automating use of chatbots in the healthcare industry [1, 2, 3, 4]. administrative tasks, such as scheduling appointments Healthcare chatbots ofer a convenient and eficient way and managing patient records, allowing them to focus on to interact with patients, providing them with person- providing quality care. The healthcare industry has seen alised and immediate attention. These chatbots can be significant benefits from chatbots, including reduced wait integrated into websites or mobile applications, provid- times, improved patient outcomes, and reduced costs. ing round-the-clock assistance to patients. Additionally, There are several standardised cognitive assessment the recent advancements in NLP (Natural Language Pro- tools used in healthcare, such as the MoCA (Montreal cessing) have led to the development of LLMs (Large Cognitive Assessment) [11] test and the BDI (Beck DeLanguage Models), which can understand the nuances of pression Inventory) [12] test. These tests are used to human language and generate responses that are contex- evaluate cognitive and emotional functions, which can tually appropriate. aid in diagnosing and treating various conditions. MoCA
Healthcare chatbots [5, 6] can be used for tasks such is a widely used cognitive screening tool that assesses as symptom assessment or medication reminders [7, 8]. various cognitive domains, such as attention, memory, They can also provide information on healthcare pro- language, visuospatial abilities, and executive functions. In contrast, BDI is a self-reported questionnaire used to measure the severity of depression symptoms.
VR (Virtual Reality) has been used in healthcare to improve patient outcomes [13], including pain management and rehabilitation [14, 15]. VR can provide a realistic and immersive environment for patients to distract them from their pain and facilitate relaxation. It can also provide a safe and controlled environment for rehabilitation exercises. Chatbots can be integrated with VR technology to provide personalized assistance during their VR sessions.
In Section 1.1, we present our motivation to carry out this project. In Section 1.2 we enumerate our main contributions. In Section 2, we depict the architecture and methodology followed for this project, describing our results in Section 3. Finally, we conclude with some limitations in Section 4 and future work in Section 5.
we need to adjust and control the chatbot output thoroughly. The chatbot-oriented LLMs fit in AGI (Artificial General Intelligence) contexts and they can be somewhat “fine-tuned” with prompt engineering [ 16, 17], but this is not enough for guided and ad hoc ADI (Artificial Domain Intelligence) systems. Another important point of our project is the contribution in the development of remote medical diagnosis tools with VR integration. This enables us to enhance communication and feedback between doctors and patients in situations where in-person assistance is not feasible.
In the context of the CEL.IA network1, we propose an IVA (Intelligent Virtual Assistant) integrated in a VR environment for remote medical diagnosis. We mixed the concepts of NLU using the Rasa chatbot framework with both ASR and TTS modules. All these components are part of the VR system, which enables the users to perform the BDI and the MoCA tests. In this paper, we describe our first demo version, including the dialogue system in a chatbot-oriented application.
For this IVA project, our contributions are as follows: • We integrated a chatbot-oriented application for remote medical diagnosis in a VR environment, providing visual interaction with selection and manipulation strategies to perform the medical tests (MoCA and BDI). • We implemented a domain-specific IVA system with Rasa framework for NLU and dialogue management. • We provide both voice and text interfaces to better communicate with the chatbot, so that we can use spoken language in Spanish (ASR), as well as listen to the answers (TTS).
We designed and implemented the following modules:
• NLU subsystem: In charge of the intent
recognition NLP task. Rasa projects follow a data-driven
approach, providing several files with text
samples for each intent and configuration files to
adjust the pipelines for model training.
1https://www.redcelia.es/
• Dialogue module: Relies on a combination of a
rule-based system and a “user stories” mechanism
to infer the next action in the conversation. These
actions can be (
We used some of the Rasa components to train the model [18]. We configured diferent pipelines, replacing the feature extractor component and preserving intact the rest of them: 1. Tokenizer: Rasa component in charge of splitting each sentence into tokens or words. We used the simple WhiteSpaceTokenizer to get the tokens splitting using white spaces. 2. Feature extractor: Rasa component in charge of feature engineering to transform the corresponding tokens into numerical vector representations. We made use of several feature extractors. We provide all the details in the experimental Section 3. 3. Intent classifier : Rasa component in charge of the intent recognition NLP task. We used the DIET classifier from the Rasa framework authors [19]. DIET is a multi-task modular transformer architecture that handles both intent classification and entity recognition together. It provides the ability to plug and play various pre-trained embeddings like BERT (and variants), GloVe, ConveRT, among others [20]. 4. Fallback classifier : This Rasa component is in charge of triggering the default intent when the
models/stt_es_citrinet_512 3https://github.com/synesthesiam/opentts#voices intent prediction from the DIET classifier does not have a confidence above a pre-specified threshold. If so, the default intent will diverge from the regular conversation, following a custom action for managing the unexpected input.
Nowadays, the most amount of NLP approaches are end-to-end and avoid the feature extraction step [21]. However, we are following a more classical approach since we do not have enough data for an end-toend fine-tuning process with transformer models. Moreover, relying on multilingual deep word embeddings as the input features for the DIET classifier has some advantages, considering that this model outperforms fine-tuned BERT and is about six times faster [20].
We can highlight the following two main features: 1. VR environment for medical diagnosis and cognitive tests: The interface for the cognitive tests (MoCA and BDI tests), as well as the other generic UI (User Interface) elements, make full use of the Babylon.js framework, which provides support for the WebXR standard to develop immersive web experiences, deployable on dedicated immersive hardware or web standard. The majority of the UI elements have been implemented as virtual objects within a 3D environment running inside an HTML canvas on the web page. This approach facilitates the switch between a traditional 3D web application with WebGL and an immersive environment with WebXR. Additionally, the scene includes the avatar for the IVA system with a recording button through which audio can be sent to communicate with the conversational IVA and answer to the cognitive test questions (see Figure 2). 2. Remote assistance service: The diagnosis system integrates a service that the healthcare professionals can remotely control to evaluate the cognitive tests the patient is carrying out. This service has been built on the Colyseus framework4, an open source project that provides tools for the implementation of multi-user online experiences. This functionality allows the healthcare professional to create a shared online environment, where a 3D scene of a virtual clinic is loaded. From this virtual space, the doctor has access to a menu where he/she can select the cognitive test to be loaded on the scene and to be performed by the patient. Then, the professional can use this virtual space to visualise patient’s progress within each test in real-time. For instance, when the patient is asked to draw a clock, the doctor can observe the drawing’s progression in real-time (see Figure 3).
We conducted tests with diferent configurations
for feature extraction in the NLU pipeline, following
two diferent approaches:
1. Traditional feature extractors: We made use
of (
sifier [ 20]. We compared the results obtained following the two aforementioned feature extraction approaches, defining 5 diferent configurations: BERT, DistilBERT, GPT-2 and XLM-RoBERTa for the pre-trained language models feature extractors and the “No model” approximation for the traditional feature extractors. We followed a 4-fold cross-validation. We computed the macro average for the f1-score, which summarises the recall and precision metrics of the intent classifier.
Intuitively, feature extraction using transformers and their deep word embeddings should be better. However, looking at the Figure 4, this is not the case. The highest macro f1-score value was achieved by the “No model” approach, which uses the traditional feature extractors. The classic feature extractors behave better with short and simple sentences, where the global context is not relevant and the vocabulary is limited (MoCA and BDI tests for medical diagnosis). Among the transformer-based extractors, the worst results were achieved by GPT-2, since this is a decoder-only transformer model optimised for NLG (Natural Language Generation) tasks rather than for NLU classification tasks. It is worth mentioning that the best transformer approach was DistilBERT, the light-weight encoder-only model needing less computing resources, even outperforming the XLM-RoBERTa model for feature extraction.
Some problems could arise when using our chatbot implementation: • Rasa Knowledge base: Due to the data-driven nature of Rasa, any new example must be manually added to the knowledge base and, consequently, it is required to retrain a new model including the changes in the NLU pipeline. • Traditional feature extraction vs deep word embeddings: As it was exposed in the Section 3, the simplicity of the sentences expected by our chatbot favours traditional feature extractors over transformers and deep embeddings. Using a language model limits the chatbot performance if we are not considering an end-to-end approach with a fine-tuned model for the NLU task [21]. • Similar intent classification : Although this limitation was mitigated implementing similarity scores (in order to match each user input to the most reasonable answer for each medical test slot), the chatbot still occasionally struggles with classifying intents when the sentence is almost identical to an example from another intent.
IVA application for medical diagnosis in a VR environment with ASR, TTS and web interfaces. We included a dialogue system capable of guiding the user to perform each of the medical tests and get a result. We made use of the Rasa framework and proposed several NLU pipelines for intent classification.
In the near future, we want to explore some NLU improvements following an end-to-end approach with a fine-tuned intent classifier based on transformerencoders. In this way, we will avoid the current feature extraction step in the NLU pipelines. We are also exploring data augmentation techniques to increase the number of samples for each intent, leveraging LLMs and cross-language translation models.
This project belongs to the CEL.IA network initiative5, which is supported by the Ministerio de Ciencia e Innovación through the CDTI (Centro para el Desarrollo Tecnológico Industrial) (grant CER-20211022).
Lim, et al., A discussion of virtual reality as a new [16] S. Qiao, Y. Ou, N. Zhang, X. Chen, et al., Reatool for training healthcare professionals, Frontiers soning with Language Model Prompting: A Surin public health 6 (2018) 44. vey, 2022. URL: https://arxiv.org/abs/2212.09597. [5] N. Bhirud, S. Tataale, S. Randive, S. Nahar, A lit- doi:10.48550/ARXIV.2212.09597. erature review on chatbots in healthcare domain, [17] Q. Dong, L. Li, D. Dai, C. Zheng, et al., A Survey International journal of scientific & technology re- on In-context Learning, 2023. URL: https://arxiv. search 8 (2019) 225–231. org/abs/2301.00234. doi:10.48550/ARXIV.2301. [6] S. Laumer, C. Maier, F. T. Gubler, Chatbot Accep- 00234.
tance in Healthcare: Explaining User Adoption of [18] Rasa, Rasa Components, 2023. URL: https://rasa. Conversational Agents for disease Diagnosis, in: com/docs/rasa/components/.
J. vom Brocke, S. Gregor, O. Müller (Eds.), 27th Eu- [19] T. Bunk, D. Varshneya, V. Vlasov, A. Nichol, ropean Conference on Information Systems - Infor- DIET: Lightweight Language Understanding for Dimation Systems for a Sharing Society, ECIS 2019, alogue Systems, 2020. URL: https://arxiv.org/abs/ Stockholm and Uppsala, Sweden, June 8-14, 2019, 2004.09936. doi:10.48550/ARXIV.2004.09936. 2019. URL: https://aisel.aisnet.org/ecis2019_rp/88. [20] M. Mantha, Introducing DIET: state-of-the[7] T. Nadarzynski, O. Miles, A. Cowie, D. Ridge, Ac- art architecture that outperforms fine-tuning ceptability of artificial intelligence (AI)-led chat- BERT and is 6X faster to train, 2020. URL: bot services in healthcare: A mixed-methods study, https://rasa.com/blog/introducing-dual-intentDigital health 5 (2019) 2055207619871808. and-entity-transformer-diet-state-of-the-art[8] L. Athota, V. K. Shukla, N. Pandey, A. Rana, Chatbot performance-on-a-lightweight-architecture/. for Healthcare System Using Artificial Intelligence, [21] A. Rahali, M. A. Akhloufi, End-to-End Transformerin: 2020 8th International Conference on Reliability, Based Models in Textual-Based NLP, AI 4 (2023) 54– Infocom Technologies and Optimization (Trends 110. URL: https://www.mdpi.com/2673-2688/4/1/4. and Future Directions) (ICRITO), 2020, pp. 619–622. doi:10.3390/ai4010004.
doi:10.1109/ICRITO48877.2020.9197833. [22] Y. Zhang, R. Jin, Z.-H. Zhou, Understanding bag-of[9] K.-J. Oh, D. Lee, B. Ko, H.-J. Choi, A Chatbot for Psy- words model: a statistical framework, International chiatric Counseling in Mental Healthcare Service journal of machine learning and cybernetics 1 (2010) Based on Emotional Dialogue Analysis and Sen- 43–52. tence Generation, in: 2017 18th IEEE International [23] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: Conference on Mobile Data Management (MDM), Pre-training of Deep Bidirectional Transformers 2017, pp. 371–375. doi:10.1109/MDM.2017.64. for Language Understanding, in: J. Burstein, C. Do[10] K. Chung, R. C. Park, Chatbot-based heathcare ran, T. Solorio (Eds.), Proceedings of the 2019 Conservice with a knowledge base for cloud computing, ference of the North American Chapter of the Cluster Computing 22 (2019) 1925–1937. Association for Computational Linguistics: Hu[11] Z. S. Nasreddine, N. A. Phillips, V. Bédirian, S. Char- man Language Technologies, NAACL-HLT 2019, bonneau, et al., The Montreal Cognitive Assess- Minneapolis, MN, USA, June 2-7, 2019, Volume 1 ment, MoCA: a brief screening tool for mild cogni- (Long and Short Papers), Association for Computative impairment, Journal of the American Geriatrics tional Linguistics, 2019, pp. 4171–4186. URL: https:// Society 53 (2005) 695–699. doi.org/10.18653/v1/n19-1423. doi:10.18653/v1/ [12] A. T. Beck, C. H. Ward, M. Mendelson, J. Mock, n19-1423.
J. Erbaugh, An inventory for measuring depression, [24] HuggingFace, How to use DistilBERT, Archives of general psychiatry 4 (1961) 561–571. 2023. URL: https://github.com/huggingface/ [13] D. Freeman, S. Reeve, A. Robinson, A. Ehlers, et al., transformers/tree/main/examples/research_ Virtual reality in the assessment, understanding, projects/distillation#how-to-use-distilbert. and treatment of mental health disorders, Psycho- [25] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudlogical medicine 47 (2017) 2393–2400. hary, et al., Unsupervised Cross-lingual Repre[14] G. Tao, B. Garrett, T. Taverner, E. Cordingley, C. Sun, sentation Learning at Scale, CoRR abs/1911.02116 Immersive virtual reality health games: a narrative (2019). URL: http://arxiv.org/abs/1911.02116. review of game design, Journal of NeuroEngineer- arXiv:1911.02116.
ing and Rehabilitation 18 (2021) 1–21. [26] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, [15] M. Ma, H. Zheng, Virtual reality and serious games I. Sutskever, Language Models are Unsupervised in healthcare, Advanced computational intelligence Multitask Learners (2019). paradigms in healthcare 6. Virtual reality in psychotherapy, rehabilitation, and assessment (2011) 169–192.