Speech therapy is the medical field in which speech impairments are treated. It mainly concerns the inability of people to adequately enunciate words, to construct and elaborate appropriate sentences when speaking, and generally lack linguistic skills. This research work presents a software system, called e-SpeechT, supporting speech therapies performed in presence and remotely from both physicians' side and other actors involved in the therapies, i.e., patients, caregivers, and also teachers, schools, etc. The study explores the requirements and functionalities with respect to customization elements and how Artificial Intelligence (AI) can support all parties involved in the process in performing their tasks. The system was designed and developed following the participatory design focusing on the diferent roles, experiences, and demands of the involved end users and inviting them to contribute from diferent perspectives.
Today, medicine is constantly being exposed to innovative technologies, and new products and
services. Many professionals are beginning to trust the use of new instruments to perform
their daily activities, such as administering therapies to their patients. This research work
focuses on speech therapy and how technology can be a powerful instrument to support all
parties involved. More specifically, this case study examines the roles of speech therapists,
caregivers, and patients who are children aged 4 to 7 years. Speech therapy refers to the medical
ifeld that focuses on the study of linguistic impairments that can negatively influence the
personal and professional sphere of an individual, afecting his ability to understand and use
spoken language with others [
In the current scenario, there are systems and mobile applications developed to support patients to perform therapy’s activities and therapists to administer the treatment. For instance, Happi Scrive 1 and KidEWords 2 are two crossword puzzle applications for children used for the development of writing skills through an attractive graphical interface. Another example is Teach and Touch 3 which is an application designed to support the speech therapist when administering the therapy; the child is allowed to perform a personalized rehabilitation path on specific morphosyntactic dificulties.
In this context, a new web application integrating AI and customization functionalities, called e-SpeechT, can be introduced.
The integration of Artificial Intelligence (AI) in this scenario provides new solutions based
on the needs and requirements of patients and therapists by not only providing functionalities
that allow to automate tasks and reduce the cognitive demand of repetitive activities, but also
by supporting the management and evolution of the system [
Customization in e-health has the potential of revolutionizing speech therapies promoting inclusion and the establishment of a symbiosis between humans and machines.
This paper presents and describes a web-application, called the e-SpeechT, created to support the actors involved (e.g. patients, caregivers and therapists) while performing, administering, and managing therapy; through the integration of AI techniques these activities can be automated and the system can be further adaptable to the patient’s performance.
This solution aims to improve the efectiveness and accessibility of speech rehabilitation, showing the requirements and architectural components of e-SpeechT and illustrating how artificial intelligence is integrated into it. 2. e-SpeechT Design e-SpeechT was developed according to the SCRUM framework, as an agile project; the framework proved to be powerful and useful, especially due to its iterative nature and flexibility since it allows one to meet user needs eficiently and efectively [ 8]. Speech therapists were involved throughout the process, employing the Human-Centered Design (HCD), which allowed to obtain rich insights into their needs and preferences to create compliant solutions [9]. The ultimate objective is to create an interactive system that can be customizable and tailorable by end users as domain experts in the design team [10]. 1https://apps.apple.com/it/app/happi-scrive/id464675842 2https://apps.apple.com/it/app/kidewords-by-chocolapps/id879490139 3http://www.teachandtouch.it/
User requirements are fundamental when designing or developing any kind of software; they describe the functionalities the final system will have and provide guidance regarding the priority of tasks and activities to carry out throughout the process.
For e-SpeechT, a number of actors were identified: speech therapist, caregiver, patient, school, teacher, and others to encompass the multi-faceted roles that end users can play, but for sake of simplicity, only the three main actors are here represented. The requirements were grouped in three diferent categories, one for each type of the main actors, and elicited through multiple interviews involving professionals. Through their collaboration in every sprint of the SCRUM process, it was possible to understand their preferences, priorities and needs, as well as those belonging to caregivers and patients. Some of the main requirements and the corresponding functionalities, that were identified for each role, are listed below and are represented in Figure 1.
Speech therapist Speech therapists are professionals who organize, manage and administer therapies to children; they also have the important duty of communicating and helping caregivers, such as parents, to help their children throughout the process [11]. In particular, the application must allow therapists to: • Create and manage diagnoses for their patients. • Share words and exercises with other professionals who use it. • Manage appointments and schedule them, even in their ofice. • Verify the results of a therapy performed by a patient. • Be supported by an automatic correction feature to reduce their workload, while allowing them to change the decisions they made to keep professionals in control. • Send notifications to patients about the status of their therapy. • Automatically classify new patients according to the level of severity of their impairment. • Change the criteria based on which the patients are classified by the system. Caregiver Caregivers are the intermediaries between therapists and patients. This is a case of multi-tiered proxy design problems in which the end user, who must be able to tailor and customize the system, and the caregiver do not coincide with the actual end user of the system, i.e. the child [12]. Caregivers have to follow them and ensure that their children perform all assigned tasks, as well as accompanying them to appointments. In particular, the application must allow caregivers to: • Start and stop the child’s execution of a therapy. • Manage the way the system is presented to the child, in terms of wallpapers, scenarios, and graphical elements.
Patient Patients are the target of therapies and are afected by speech impairments of diferent degrees of severity. Their age ranges from 4 to 7 years old. The requirements collected for their personal area of e-SpeechT state that the system must: • Display playful and gaming aspects to avoid feelings of stress or frustration. • Include gamification, encompassing game mechanics such as rewards and playful feedback. • Allow the child to log in without the aid of a professional or caregiver.
Speech Therapist
Speech System
Create Diagnosis Create Exercises
Correct Exercises Manage
Therapy
Set Thresholds
Create
Therapy
Manage Appointments
Perform Exercises Manage Scenario
Manage Child's Personal
Data End Therapy Start Therapy
Patient Caregiver
After eliciting the requirements, creating the use cases, and carrying out the preliminary phases of the Human-Centered Design (HCD), e-SpeechT was designed and developed. This section illustrates how customization plays a pivotal role in the web application and how it aims to guarantee the user-specific solutions to health problems, carry out the right diagnoses and avoid stress, boredom and frustration. e-SpeechT embodies End-User-Development (EUD) in multiple aspects and functionalities, which difer depending on the type of user involved.
The e-SpeechT application allows diferent types of intervention, from customization to EUD activities, to personalize its behavior. From the Speech Therapist’s side, the application allows to create new examples to evaluate their patients, since each word is characterized by peculiarities that result in diferent outcomes in exercises; it is also possible to assign images to words with no constraints, meaning that therapists can pick images that bring continuity to the work done during on-site appointments. Exercises can be created and designed based on patients’ needs, diagnosis and level of impairment; they belong to one of the three categories - i.e., naming images, minimum pair recognition and repetition of words
In addition, therapists are free to create and modify therapies depending on the patient and their diagnosis by manipulating attributes like exercises to be administered, beginning and end date and date, time and place of the appointment. Therapies can contain one or multiple exercises, which can be administered individually or through series.
e-SpeechT ofers an exercise automatic correction functionality, which exploits machine learning and speech recognition algorithms in order to facilitate the process of correcting exercises from the therapist’s standpoint. They are used to adapt how the system behaves according to specific groups of patients. Thresholds are used to adjust the level of strictness of the system when it comes to automatically correct exercises during the evaluation of the patient’s vocal interaction; three categories of thresholds are available, which depend on three levels of severity of impairments: Moderate, Severe, and Slight, as shown in Figure 2. (a) Customization of the thresholds of the auto
matic correction
User-specific personalization is crucial to ensure that patients (children aged from 4 to 7 years old) feel welcomed, comfortable, and not under examination.
In e-SpeechT, scenarios are used to diferentiate the style, aesthetic and atmosphere of the patient’s page during the therapy sessions, when carrying out exercises. Caregivers, through EUD activities, are able to modify the whole scenarios by choosing among the ones proposed by the system or creating a new one from scratch. Visual familiarity can prevent children from feeling examined or judged.
Another important aspect of the the system lies in setting a graphical password for the patient, which children are able to log in the application with.
This research work presents a web-application, e-SpeechT, that aims at supporting therapists, patients, and caregivers during treatments. It describes the design and development process of the application in question, which was performed in accordance with the Human-Centered Design highlighting its participatory process. In this way, the role of the end users can be further investigated to analyse how they are progressing into active participants during the design and development phases.
In this context, e-health systems that include AI can bring advantages automating tasks performed by therapists and providing patient-specific solutions. Additionally, through the improvement of speech recognition algorithms it would be possible to easily and automatically detect errors in enunciating/repeating words [13]. However, AI can present some limitations such as, for example, the limited context understanding and the risk of overreliance. For this reason, it is necessary to guarantee the right balance between the technology and the human intervention in clinical practice.
Another important aspect is to understand how to guarantee the correct balance between the end users’ contribution and how to guarantee that they will be motivated to contribute to the system’s performance over long periods of time. Therefore, as a starting point, usability studies were carried out to understand the weaknesses and strengths of the system; as a future work, it is intended to conduct a longitudinal study to test the medical feasibility by gathering information on how children respond to therapies when using the system in question and whether all the involved actors continue to be motivated to use e-SpeechT.
Future work also involve conducting tests to evaluate the application with respect to the WCAG (Web Content Accessibility Guidelines)4; this is an important aspect because this application is primarily aimed at individuals possibly afected by disabilities.
Another future work is to expand the application’s environment by integrating smart assistants and Internet of Things (IoT) devices. This feature has the objective of automatizing the process of guiding children in their daily activities when having to carry out exercises at home. This project has already started, but is still in its early development stages.
In conclusion, the intersection between speech therapy and technology can revolutionize the approach to deal with speech disorders. However, it is important to underline that the role of AI in this scenario is not a panacea, but an instrument capable of helping users perform their tasks without replacing them. 4https://www.w3.org/WAI/WCAG22/Understanding/understanding-act-rules.html [5] P. Tagde, S. Tagde, T. Bhattacharya, P. Tagde, H. Chopra, R. Akter, D. Kaushik, M. H. Rahman, Blockchain and artificial intelligence technology in e-Health, Environmental Science and Pollution Research 28 (2021) 52810–52831. URL: https://link.springer.com/10.1007/s11356-021-16223-0. doi:10. 1007/s11356-021-16223-0. [6] F. Cassano, A. Pagano, A. Piccinno, Supporting speech therapies at (smart) home through voice assistance, volume 483, 2022, p. 105 – 113. URL: https://www.scopus.com/inward/ record.uri?eid=2-s2.0-85137986566&doi=10.1007%2f978-3-031-06894-2_10&partnerID=40&md5= de8c5b2f0a3ed3f354cb010281ad4004. doi:10.1007/978-3-031-06894-2_10. [7] V. Barletta, M. Calvano, A. Curci, A. Piccinno, A new interactive paradigm for speech therapy, in: J. Abdelnour Nocera, M. Kristín Lárusdóttir, H. Petrie, A. Piccinno, M. Winckler (Eds.), Humancomputer interaction – INTERACT 2023, Springer Nature Switzerland, Cham, 2023, pp. 380–385. doi:10.1007/978-3-031-42293-5_39. [8] T. Badriyah, E. N. Fadila, I. Syarif, N. Jauari Akhmad, Rapid development of m-health application with the sprint design approach and scrum process : Application development for e-prescribing, in: 2018 International Conference on Applied Science and Technology (iCAST), 2018, pp. 336–342. doi:10.1109/iCAST1.2018.8751603. [9] V. S. Barletta, M. Calvano, A. Curci, A. Piccinno, A. Pagano, Poster: Speech therapies and smart assistants: An interaction paradigm proposal, 2023. URL: https://www.scopus.com/ inward/record.uri?eid=2-s2.0-85173634861&doi=10.1145%2f3605390.3610823&partnerID=40& md5=62e9ad3f964c7c3df3b04931fd4fd68a. doi:10.1145/3605390.3610823, cited by: 0. [10] M. F. Costabile, D. Fogli, A. Marcante, P. Mussio, L. Parasiliti Provenza, A. Piccinno, Designing customized and tailorable visual interactive systems, International Journal of Software Engineering and Knowledge Engineering 18 (2008) 305–325. URL: http://www.worldscientific.com/doi/abs/10. 1142/S0218194008003702. doi:10.1142/S0218194008003702. [11] T. Pring, E. Flood, B. Dodd, V. Jofe, The working practices and clinical experiences of paediatric speech and language therapists: a national uk survey, International Journal of Language & Communication Disorders 47 (2012) 696–708. URL: https://onlinelibrary.wiley.com/doi/abs/10.1111/ j.1460-6984.2012.00177.x. doi:https://doi.org/10.1111/j.1460-6984.2012.00177.x. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1460-6984.2012.00177.x. [12] D. Fogli, A. Piccinno, Co-evolution of End-User Developers and Systems in Multi-tiered Proxy Design Problems, volume 7897 of Lecture Notes in Computer Science, Springer, Berlin Heidelberg, 2013, pp. 153–168. URL: http://dx.doi.org/10.1007/978-3-642-38706-7_12. doi:10.1007/ 978-3-642-38706-7_12. [13] M. Calvano, A. Curci, A. Pagano, A. Piccinno, Speech therapy supported by ai and smart assistants, in: R. Kadgien, A. Jedlitschka, A. Janes, V. Lenarduzzi, X. Li (Eds.), Product-Focused Software Process Improvement, Springer Nature Switzerland, Cham, 2024, pp. 97–104.