Aphasia is a condition resulting from a stroke or injury to the brain. This condition renders the patient unable to use language eficiently. Machine learning and artificial intelligence automate the assessment and diagnosis of aphasia to a certain extent. However, solutions based on machine learning that are patient-centric, self-paced, and help with therapy are lacking. In this work, we present a rehabilitation framework that can be implemented to address this deficit. As a part of an ongoing project, we aim to implement these ideas to facilitate the rehabilitation of aphasic patients by benefiting from the developments in machine learning. It is also anticipated that these ideas can be extended to help people wanting to acquire a new language.
bilitation framework that is easy to use, patient-centric, self-paced, and attempts to utilize the advances in maThe rapid advancements in machine learning and artifi- chine learning. We aim to design the framework and cial intelligence have made it possible to solve diverse ofer the patients the possibility to practice as often and problems. A particular use case in healthcare that has for as many iterations as they would like. We expect to attracted the attention of researchers is the application eventually reduce the patients’ dependence on therapists of machine learning approaches to manage aphasia. and provide more flexibility during rehabilitation.
Aphasia is an acquired language disorder that can be The subsequent sections in this paper present the
caused by an injury to the brain, a stroke, or a result of related research, discussions regarding the approach
degenerative processes [
Therapy and recovery to some extent are possible in these papers conclude that the majority of the works use
most cases. However, constraints such as a shortage of machine learning for the diagnosis and assessment of
therapists, availability of frequent therapy appointments aphasia. The authors highlight the need for exploring
for sustained recovery, and limited funding of support in- the possibility of utilizing the advancements in machine
frastructure for long-term rehabilitation programs prove learning to improve aphasia rehabilitation as well.
to be a hurdle [
In this work, we discuss ideas as a part of an ongoing ing Bag of Audio Words and Long Short-Term Memory
project, that could be used to rehabilitate People With (LSTM) neural networks to automate the detection and
Aphasia (PWA). Our objective is to implement a reha- classification of aphasia speech. In this work, neural
networks are trained to identify the features corresponding
35th GI-Workshop on Foundations of Databases (Grundlagen von Daten- to aphasic speech and then classify the severity of
disabilbanken), May 22-24, 2024, Herdecke, Germany. ity based on the input audio sample. Similar to this work,
* Corresponding author. are approaches proposed by Hirsch et al., 2023 [
Pierce et al., 2024 [11] explored an application of gen- and facilitate understanding of aphasia. However, a
syserative artificial intelligence to aphasia management. In tematic framework that is easy to use and focuses on
this work, the authors investigate the possibility of us- helping aphasic patients recover functionally in a
usering DALL-E, a powerful text-to-image model to create centric and self-paced manner using machine learning
images using text prompts. The authors aim to use the approaches is lacking [
Research by Vong et al., 2024 [12] explored the possibility of training a neural network using multi-modal data 3.1. Language learning framework (visual and audio streams) to learn the word-referent map- We propose incorporating the tasks and activities [17] pings. The data is collected from a camera mounted on that PWAs accomplish during their traditional therapy, the head of a single child (aged between 6 to 25 months) into an online framework. To this end, we use the Flutter thus providing approximately 61 hours of input streams toolkit based on the Dart programming language to imfrom a first-person perspective to the neural network. plement this framework. This afords the possibility to Though not directly related to aphasia management, this develop applications that run across multiple platforms. approach presents interesting insights into how humans We intend to implement this framework as a gamified acquire language and how this process can be adapted to platform. The basic Graphical User Interface (GUI) dethe realm of machines. pends on the task being presented to the patients. The
The work by Palmer et al., 2017 [13] attempts to man- GUIs for two tasks (of the four) are presented in Figure age aphasia rehabilitation by addressing the following 1 and Figure 2. The figures show a tablet emulator with questions: (i) What do people with aphasia want to talk the interface visible to the patients. These figures show about? and (ii) What words and topics are selected for prac- a virtual therapist who serves as a guide to the patients tice by aphasic patients? The authors select 100 patients throughout this therapy experience. The tasks are prerepresenting diferent geographical locations across the sented as questions that the patient answers to progress. UK, age brackets, genders, and exhibiting varying de- The gamified aspect of the framework is implemented grees of capabilities on the Comprehensive Aphasia Test by including a computerized rival that aids in demon[14]. The patients are then requested to select 100 words strating the workings of the game and also serves as a each that they identify as important to them in their benchmark for the patients to compare their performance daily lives. The words are then categorized by the topics against. We have implemented additional components of and their frequency of occurrence. The authors claim uncertainty and randomness into the framework. These that such quantitative analysis could minimize the time include elements such as a random probability generator needed for preparing patient-specific therapy material to determine whether to award points for correctly anand might improve the eficiency of the rehabilitation. swering a task. Including these aspects should make the
Gu et al., 2020 [15] explored the possibility of inte- rehabilitation experience more engaging. grating the behavioral and brain variables obtained from post-stroke aphasia patients with machine learning algorithms to categorize and predict the responsiveness to
volved in the traditional therapy for aphasia. These tasks focus on improving individual aspects of language ranging from single-word level to sentence-level complexity. These tasks also focus on improving skills such as retrieving a desired word, coherent speech, and fluency. An overview of the planned tasks is provided here: • Comprehension:
This activity helps the patient at a single-word level and strengthens the word-object mapping. The patient is provided with two images and an audio prompt. The patient has to select the image corresponding to the spoken word prompt. The GUI for this exercise is depicted in Figure 1. As a demonstration, the computerized rival exhibits how to answer this task with several audio prompts. As the computerized rival is implemented to select the correct options with a preset probability, it is also possible for the rival to commit errors. This uncertainty aids in exhibiting to the patients, how not to solve the task. Success and failure of the rival are appropriately indicated by providing video (via gifs) and audio (via notiifcation tone) feedback. After this demonstration, the patient is provided the audio prompt to solve the task. Identical video and audio feedback are provided to the patient to inform them how they perform the task. • Repetition:
This activity helps improve the patient’s ability to
understand spoken words and produce coherent
speech. The patient is provided with an audio
input in this task. The patient has to then
reproduce that particular word by speaking. In this
task, the patient might answer in several ways.
The patient might repeat the name correctly. The
patient can also incorrectly repeat the given word.
It is also possible that the patient might hesitate
before answering or produce incoherent speech.
It is therefore imperative to use a speech
recognition system and corresponding evaluation metric
that handles the listed cases robustly. We plan to
record the patient’s speech and use speech
recognition systems similar to [
This task aims to improve the patient’s ability to retrieve words from their vocabulary and produce coherent speech. In this task, the patient is provided a visual prompt as seen in Figure 2. The patient has to speak the name of the object. As the patient output modality is similar to the previous task (audio/speech), it would be logical to use a similar processing methodology. The speech systems we select to use should be able to determine if the patient’s speech corresponds to the given prompt or not. • Multi-word comprehension:
This task helps to improve the patient’s ability to understand the context of a given situation, formulate their thoughts, and articulate them in words. In this task, the patient is given an input image similar to the one shown in Figure 3 and is asked questions about this image. The patient should then answer the questions based on what they observe in this image. For instance, for the depicted image, the patient might be asked to answer questions such as: (i) What do you see in this image? (ii) How many people are in this image? (iii) What is kept on the table? (iv) Could you point to the {lamp, windows, table, chair}?
pend on the type of question being asked. For instance, for (i) and (iii), the answer might be descriptive, whereas it might primarily be numeric for (ii). Also, it is possible that for (iv), the patient might either answer verbally, with gestures, or point to the relevant entity in the image. This necessitates a robust evaluation strategy for this task.
corporate the principle of adaptive dificulty adjustment into our framework. This idea has been explored in video games to tune the dificulty level of the game based on the skill and performance of the player [18, 19, 20, 21, 22, 23]. If the player finds the game too dificult, the game dificulty is lowered. However, if the player finds the game too easy, the dificulty level is increased. This concept has been visualized in Figure 4. The optimum zone of operation where the player is engaged and upgrades their skill is in the flow zone . Anywhere other than this zone would make the game either too dificult/too easy or not substantially rewarding for the player.
This principle can be utilized in aphasia rehabilitation
tasks as well [
The framework discussed in previous sections can theoretically be used by healthy individuals who want to learn a new language. Starting from a single-word level complexity, the subject aiming to learn a new language can acquire the ability to understand the situational context with the aid of multi-modal data such as images, audio, and videos. To this end, the framework must be expanded to support multiple languages. This process will involve acquiring audio data of spoken words, and a description of visual data in the target language among other upgrades. These modifications can be implemented by harnessing the power of generative AI as is explored in the works of [11].
In this section, we discuss the state of implementation of the ideas discussed in this paper. Additionally, we provide the potential research directions to be explored. • Implementation of tasks: The discussions concluded above provide a high-level overview of the task description. The implementation of these tasks is in progress. The first iteration of the comprehension task is implemented and the subsequent tasks will be targeted in the future. • Evaluation criteria: For the comprehension task, the accuracy of the patient response can be used as the evaluation metric. The patient’s response can be either correct or incorrect and the accuracy can be quantified by calculating the number of correct responses to the total number of responses. For the subsequent tasks, we have to identify suitable metrics. As we discussed earlier, the patient’s response for these tasks might also include hesitation, long pauses, and incoherent words. We plan on categorizing the patient responses into two categories: correct and incorrect. For initial iterations, we will follow a broad definition of correctness to allow patients to familiarize themselves. The patient responses will not be strictly evaluated and deviation from the ground truth will be allowed. However, as the therapy progresses, this definition of correctness will be made stricter, and fewer errors will be allowed. The selection of evaluation metrics will be based on the speech recognition system we use to implement these tasks. • Adaptive dificulty adjustment: This aspect will be implemented in the future. As discussed, we have yet to identify a suitable algorithm to integrate into our framework. Additionally, we have to select the necessary patient parameters that can be used to tune the dificulty level.
Additional aspects can be implemented to improve the framework in the future. As explored in [11], the image training material for rehabilitation tasks can be generated using a text-to-image model. This will vastly reduce the time needed to procure the data and be a lowcost alternative. Another possibility can be to explore deep learning models that can generate speech from text. This will further aid in the creation of the rehabilitation material.
Advances in machine learning and generative artificial intelligence can be used to create training material for rehabilitation. Also, implementing concepts from the domain of game design such as adaptive dificulty adjustment algorithms into our proposed therapy framework can make it engaging and rewarding. Finally, the scope of the rehabilitation framework can be extended with upgrades for additional languages. This will make the framework useful for healthy individuals who want to acquire a new language.
Proceedings of the 34st conference Elektronische Kim, Diversifying dynamic dificulty adjustment Sprachsignalverarbeitung, 2023, pp. 134–141. agent by integrating player state models into monte[9] D. S. Barbera, M. Huckvale, V. Fleming, E. Upton, carlo tree search, Expert Systems with Applications H. Coley-Fisher, C. Doogan, I. Shaw, W. Latham, 205 (2022) 117677.
A. P. Lef, J. Crinion, Nuva: a naming utterance [22] G. K. Sepulveda, F. Besoain, N. A. Barriga, Explorverifier for aphasia treatment, Computer Speech & ing dynamic dificulty adjustment in videogames, Language 69 (2021) 101221. in: 2019 IEEE CHILEAN Conference on Electrical, [10] S. S. Mahmoud, A. Kumar, Y. Li, Y. Tang, Q. Fang, Electronics Engineering, Information and CommuPerformance evaluation of machine learning frame- nication Technologies (CHILECON), IEEE, 2019, pp. works for aphasia assessment, Sensors 21 (2021) 1–6.
2582. [23] M. Zohaib, et al., Dynamic dificulty adjustment [11] J. E. Pierce, Ai-generated images for speech pathol- (dda) in computer games: A review, Advances in ogy—an exploratory application to aphasia assess- Human-Computer Interaction 2018 (2018). ment and intervention materials, American Journal [24] J. C. Lopes, R. P. Lopes, A review of dynamic difof Speech-Language Pathology 33 (2024) 443–451. ifculty adjustment methods for serious games, in: [12] W. K. Vong, W. Wang, A. E. Orhan, B. M. Lake, International Conference on Optimization, LearnGrounded language acquisition through the eyes ing Algorithms, and Applications, Springer, 2022, and ears of a single child, Science 383 (2024) 504– pp. 144–159.
511. [13] R. Palmer, H. Hughes, T. Chater, What do people with aphasia want to be able to say? a content anal- A. Online Resources ysis of words identified as personally relevant by people with aphasia, PloS one 12 (2017) e0174065. • AI Hotpot: [14] K. Swinburn, G. Porter, D. Howard, Comprehensive https://hotpot.ai/blog/
aphasia test, London: Psychology Press; (2004). ai-art-and-image-generator-guide [15] Y. Gu, M. Bahrani, A. Billot, S. Lai, E. J. Braun,
M. Varkanitsa, J. Bighetto, B. Rapp, T. B. Parrish, D. Caplan, et al., A machine learning approach for predicting post-stroke aphasia recovery: A pilot study, in: Proceedings of the 13th ACM International Conference on Pervasive Technologies Related to Assistive Environments, 2020, pp. 1–9. [16] A. Das, T. W. Day, V. Kulkarni, A. Buchanan, K. Cottrell, N. W. John, K. Chatterjee, Towards intelligent extended reality in stroke rehabilitation: Application of machine learning and artificial intelligence in rehabilitation, in: Augmenting Neurological Disorder Prediction and Rehabilitation Using Artificial
Intelligence, Elsevier, 2022, pp. 309–329. [17] N. Nozari, A. K. Kittredge, G. S. Dell, M. F. Schwartz,
Naming and repetition in aphasia: Steps, routes, and frequency efects, Journal of memory and language 63 (2010) 541–559. [18] D. Kristan, P. Bessa, R. Costa, C. Vaz de Carvalho,
Creating competitive opponents for serious games through dynamic dificulty adjustment, Information 11 (2020) 156. [19] M. Csikszentmihalyi, The flow experience and its significance for human psychology, Optimal experience: Psychological studies of flow in consciousness 2 (1988) 15–35. [20] E. Lach, J. Lach, W. Suchanek, Dynamic dificulty adjustment of the computer player in computer games, Studia Informatica 38 (2017) 19–30. [21] J. Moon, Y. Choi, T. Park, J. Choi, J.-H. Hong, K.-J.