ARTIS: a digital interface to promote the rehabiliatation of text comprehension dificulties through Artificial Intelligence

Abstract

This study introduces ARTIS, an AI-powered interface aimed at enhancing text comprehension in children with language and learning disorders and deficits. Leveraging neuro-psycholinguistic models, ARTIS ofers personalized practice at varying dificulty levels, targeting specific linguistic components of text processing. Through AI algorithms, ARTIS autonomously extracts keywords, associates them with pictograms, identifies complex words, generates semantic networks, and proposes exercises on grammatical components. By automating cognitive processes and providing tailored interventions, ARTIS represents a significant advancement in promoting inclusive hybrid speech and language therapy practices, and improving text comprehension skills in children with special educational needs.

Keywords

• eol>Natural Language Processing
• Text comprehension
• Language and learning disorders
• Speech and Language Therapy

1. Introduction

local and global. Local processing involves the understanding of individual sentences and phrases, while global processing focuses on integrating these local meanings into a coherent representation of the entire text. Central to their model is the concept of "coherence," where readers strive to create a cohesive mental representation by linking information within the text and connecting it to their existing knowledge.

As much as the models described may emphasize diferent components as central to the comprehension process, they all converge on the idea that language skills, along with semantic and inferential skills, participate in linking prior knowledge to new information, creating a coherent and more complex representation of new meanings. In designing our interface, ARTIS, we drew inspiration from these foundational theories in psycho-linguistics. a clinical perspective.

2. Pedagogical and

Psycho-linguistics Foundations

Over the years researchers in psycho-linguistics had been trying to formalise how people understand connected text. Text comprehension is defined as a complex cognitive task which involves an active process of meaning construction, dependent not only on the information in the text but also on the information possessed by the reader [ 7 ]. Researchers in psycho-linguistics have strived to formalize the mechanisms underlying this process. As a result, four main theories emerged as the main ones in the psycho-linguistics landscape: 1. The Simple View of Reading by Gough and

Tunmer [ 8 ] where the decoding process and language skills are considered central to text comprehension. The key idea of this model is that 3. Existing Work comprehension, C, is defined as the product of decoding factors, D, defined as fluent word reading In the literature, there is one Italian systems for teleand language comprehension, L [ 8 ]. rehabilitation targeting specific learning disorders and special educational needs, i.e. RIDInet [17]. RIDInet 2. Structure Building Framework by Gerns- focuses on text comprehension skills, particularly semanbacher [ 9, 10, 11 ] where text comprehension is tic and syntactic inferential processes. Neberthless, dedefined with the metaphor of the construction of spite the clinical context, RIDInet lacks the integration a building where one’s must start from the foun- of Natural Language Processing (NLP) techniques. In dations i.e. from the first element contained in English, two primary systems are 3D Readers [18] and the text, for later one integrating the new infor- CACSR [19]. 3D Readers ofer users options for verbal or mation and enriching the departing structure in visual strategies to enhance comprehension, with immea cohesive one. diate feedback provided. CACSR provides personalized 3. Memory-Focus Model of Sanford and Gar- instruction using various techniques like visual images rod [ 12, 13 ] in which is central the process of and summarizing strategies, ofering real-time feedback. co-reference resolution. In fact, the key idea of Although these systems enhance reading comprehenthis theory is that when comprehending a text, sion, they are primarily used for educational contexts a reader must first establish a coherent interpre- and lack focus on clinical categories. Moreover, Open tation of the text comprehended so far and must Book [20] is the only system employing NLP, mainly therefore establish whether a given element had for children with Autism Spectrum Disorder, focusing been discussed previously or not. on text simplification and customization functions [ 21]. 4. Construction-Integration Model of Kintsch Its key features include text simplification through NLP and van Dijk [14, 15, 16] focuses on how infor- techniques and rich customization functions, enabling mation from the text, at the granularity level of users to quickly adapt the document presentation (font, propositions, is connected to and completed by in- font size, line spacing, colors) to their preferences. It formation stored in the long-term memory of the also provides assistive tools such as dictionaries and imreader. Besides literal understanding of the words ages. However, Open Books was designed for a specific and syntax in the text, it’s essential to be able to clinical population, and it focused more on improving gather propositions from the text, their concepts their decoding skills than reading comprehension per and connections, and organise them into intercon- se. Finally, Systems targeting tele-rehabilitation specifinected hierarchical structures. It is only with this cally for Dyslexia exist [22, 23, 24, 25], but few focus on representation of information, integrated with reading comprehension skills. the reader’s preexisting knowledge makes it possible to gain deep access to the content of the text[ 7, 16, 15 ]. To reflect this view, Kintsch and van Dijk identified two main levels of processing: 4. Target Population of ARTIS ercises that focus on co-reference structures. Finally, in a preliminary stage, ARTIS endeavors to develop users’ ability to construct broader mental models of language by integrating textual information with their existing knowledge [31]. Subsequent sections will delve into the functionalities of each level, providing detailed insights into the underlying algorithms. The initial prototype of the interface was extensively described in [32].

ARTIS targets two distinct target populations: students

with specific reading comprehension disorders, and those identified in literature as “poor comprehenders”.

Reading Comprehension Disorder, identified in the DSM-V[26], delineates dificulty in grasping the meaning of text despite proficient decoding skills. This encompasses understanding word sequences, implicit information (inferences), and the deeper meaning of text content. 5.1. Understanding Sentences The distinction between decoding and comprehension abilities is highlighted in the literature [27], which under- The interface initially presents the text at a sentence level lines diferences in cognitive processes, predictive factors, using the Spacy Sentencizer tool 1., allowing users to fodisorder characteristics, and treatment approaches. The cus on individual sentences. To aid comprehension, users functions involved in text comprehension include lexical can listen to the text through speech synthesis powered skills (vocabulary), inferential skills, working memory, by the Google text-to-speech API, improving accessibility attention, and meta-cognitive control, which are distinct for those with reading dificulties. Keywords from each from decoding skills. Even if this disorder therefore often sentence are extracted and displayed using Picture Comintersects with other conditions such as specific language munication Symbols (PCS) on the left side of the screen, disorder, decoding disorder (dyslexia), intellectual dis- facilitating comprehension for individuals with literacy ability, or memory dificulties, it can be said that specific challenges. The extraction process involves using a cusreading comprehension disorder is somehow indepen- tomized version of Keybert [33] and manual verification dent of specific decoding disorder. Despite complexities by speech and language therapists. These keywords are in classification, understanding this disorder is crucial, linked to appropriate pictograms through the Arasaac as it often gets conflated with other clinical profiles. This API 2, replacing real images to ensure suitability for vulis precisely the reason behind the beginning of this work nerable audiences. and the reason why we involved since the beginning of its conception speech and language therapists.

The literature defines “poor readers” or “poor comprehenders” students who experience specific problems in comprehension in the face of decoding skills that are instead within the normal range [28],[29], [30]. Distin- Figure 1: Section on understanding sentences. guishing poor comprehension from a reading or writing learning disability necessitates diagnostic test results. For example, dyslexia manifests as reading efort, character- 5.2. Understanding Words ized by decoding and reading dificulties, error prevalence, and fluency deficits. In contrast, poor readers and writers exhibit milder characteristics, including adequate reading speed with occasional fluency issues. Poor comprehenders’ dificulties often stem from factors such as inadequate reading and writing process automation, environmental stimulation deficiencies, cognitive, memory, or attention issues, rather than neurological causes.

In a following step, the interface ofers a closer exami

nation of the text at the word level. It displays the ten rarest words for each selected text, identified using WordFreq [34], a tool providing frequency estimates across languages. This feature aims to assist individuals with reading comprehension dificulties by addressing their hesitancy towards unfamiliar vocabulary. Upon user selection, the word’s definition, along with a PCS representing its meaning and the original sentence context, 5. Architecture of ARTIS is provided. Definitions, sourced from the Oxford Dictionary API 3, are presented in both written and spoken ARTIS is an online tool designed to enhance reading com- forms through speech synthesis, enhancing accessibility. prehension skills. It comprises four distinct levels aimed 5.3. Understanding Paragraphs at fostering a foundational understanding of language.

The tool assists users in comprehending the lexical and Based on insights from [ 6 ], a co-reference resolution algosyntactic aspects of the text, beginning with understand- rithm was incorporated to aid users in grasping the coning sentences and progressing to words. Furthermore, the nections between diferent entities and pronouns within interface aids users in constructing coherent sequences 1https://spacy.io/api/sentencizer and hierarchical structures within the text through ex- 2https://arasaac.org/ 3https://developer.oxforddictionaries.com/

Network, users explore the interconnections of words, fostering a deeper understanding of their multifaceted meanings.

the text. Pronouns were prioritized due to the challenges associated with their recognition. In this level of the inter- Figure 4: Sectiong on bridging existing and newer knowledge. face, sentences are segmented, and grammatical elements 6. Empirical Evaluation such as common nouns, proper nouns, and pronouns are highlighted with distinct colors based on their Part of A pilot study was conducted at the CRC in Rome, a deSpeech Tags. This color-coded approach assists in imme- velopmental rehabilitation center, from January to June diate diferentiation between elements and suggests the 2023. The study involved 24 Italian-speaking children function of each lexical morpheme, guiding the under- from primary 2 to 5, who were assessed using the platstanding of pronouns by identifying their correct refer- form under the guidance of speech therapists from the ents. Users are presented with propositions and asked CRC. Each child had a clinical diagnosis and an updated to select the correct referent from four alternatives. One functional profile, including IQ, language, and learning is accurately identified by the co-reference resolution profiles. The study included children with impairments algorithm, while two are intentionally misleading, and in text comprehension and language skills, particularly the fourth closely resembles the correct answer. Positive in receptive and expressive vocabulary not aligned with feedback is provided upon selecting the correct answer. their age reference. The participants comprised children Spacy part-of-speech tagger 4 was used for parsing, and diagnosed with Specific Learning Disorder (7 subjects), a fine-tuned version of Coreferee 5 served as the corefer- Primary Language Disorder (8 subjects), Borderline Inence resolution algorithm. tellectual Functioning (7 subjects), and High Functioning Autism (3 subjects).

The children who participated in the feasibility study underwent standardised pre- and post-treatment evaluations, with particular attention in the post-evaluation to data on reading comprehension and language skills.

The speech therapists who followed the children in the trial were also asked to fill in a questionnaire (google Figure 3: Section on understanding paragraphs. form) to investigate usability, functionality and the level 5.4. A first attempt for bridging Textual of perceived efectiveness of ARTIS in its diferent comUnderstanding with Prior Knowledge ponents, in relation to the diferent pathologies, with qualitative questions structured according to a five-point As emphasized by [31], text comprehension extends be- likert scale (Very Little, Slightly, Suficient, Very Much). yond its surface-level representation. To foster deeper un- The data collected focused on the platform usage, work derstanding, Synset Networks were introduced. Each se- area preferences, exercise types, and their correlation lected word is presented alongside its associated Synsets with diferent clinical profiles. The therapists’ feedback from the Merriam Webster thesaurus, facilitating vocab- regarding exercise functionality, including the need for ulary expansion and recall. This approach enhances com- additional support or information beyond AI-generated prehension by elucidating the various meanings a term content, was also evaluated. can encompass within a text and linking it with familiar Results indicated that all participants (24 children) utiwords, thereby aiding in the integration of new infor- lized the platform to visualize sentences and aid commation with existing knowledge. Through the Synset prehension of selected texts. This feature included dis4https://spacy.io/usage/linguistic-features playing relevant images corresponding to individual sen5https://github.com/richardpaulhudson/coreferee tences and the option for speech synthesis reading. No supplementary material or therapist intervention was required for this aspect. Children with more significant language deficits (DPL, FIL, AUTISM) tended to use the vocabulary area (focused on words) more frequently. The inclusion of definitions accompanied by pictures proved highly beneficial. Occasionally, additional information sourced from Google searches was needed to clarify or enhance the meaning of certain words. A small subset of the sample (4 subjects diagnosed with DSA) engaged with the semantic network. Utilizing the semantic network required meta-cognitive and generalization skills, with therapist support in tasks such as image searches, reflection on diferent word uses, and identification of texts or phrases highlighting semantic distinctions.

The results of the pre and post-treatment evaluations show a significant increase in text comprehension and some language skills with particular reference to receptive and expressive vocabulary.

7. Risks and Mitigations

To ensure the development of a trustworthy AI system, extensive involvement of key stakeholders, particularly speech and language pathologists, was maintained throughout the work. Their participation ranged from the initial decision-making phases to the final evaluation of the interface, with continuous exchange of advice and feedback. Moreover, the administration of exercises always occurs under the supervision of practitioners. Before conducting clinical trials involving children, precautionary measures were taken to mitigate risks. Texts were pre-selected and categorized by school age to cater to users’ individual levels. Additionally, manual clinical control was implemented to identify and rectify any potentially unsafe pictograms or misleading keywords extracted by the AI models. A stop button was integrated into the interface to prevent exposure to misleading material. Furthermore, certain functionalities, such as the semantic network, are selectively displayed based on individual needs and clinical judgment, ensuring tailored support for each user.

8. Conclusions & Future Work

In this paper, we described the development of ARTIS, an AI-powered interface designed to enhance text comprehension in children with language and learning disorders. By leveraging neuro-psycholinguistic models and thanks to the integration of AI algorithms, ARTIS autonomously extracts keywords, associates them with pictograms, identifies complex words, generates semantic networks, and proposes exercises on grammatical components. These features aim to improving text comprehension skills in children with special educational needs by promoting inclusive hybrid speech and language therapy practices.

Looking ahead, future work could focus on several aspects. Firstly, conducting an extensive clinical evaluation is essential to assess the efectiveness of the interface in clinical settings. Additionally, we aim to integrate concept-map-based document summaries, as highlighted in [35], to enhance the bridging between knowledge present in the text read and prior knowledge. [13] A. J. Sanford, S. C. Garrod, Selective processing in laka, T. Edirisinghe, S. Thelijjagoda, Alexza: A text understanding. (1994). mobile application for dyslexics utilizing artificial [14] T. A. Van Dijk, W. Kintsch, et al., Strategies of intelligence and machine learning concepts, in: discourse comprehension (1983). 2018 3rd International Conference on Information [15] W. Kintsch, The role of knowledge in dis- Technology Research (ICITR), IEEE, 2018, pp. 1–6. course comprehension: A construction-integration [26] A. P. Association, et al., Diagnostic and statistical model, in: G. Stelmach, P. Vroon (Eds.), Text manual of mental disorders, Text revision (2000). and Text Processing, volume 79 of Advances [27] A. P. Association, et al., Manuale diagnostico e in Psychology, North-Holland, 1991, pp. 107– statistico dei disturbi mentali, Quinta edizione. Raf153. URL: https://www.sciencedirect.com/science/ faello Cortina Editore (2014). article/pii/S0166411508615514. doi:https://doi. [28] D. V. Bishop, M. J. Snowling, Developmental org/10.1016/S0166-4115(08)61551-4. dyslexia and specific language impairment: Same [16] W. Kintsch, How readers construct situation mod- or diferent?, Psychological bulletin 130 (2004) 858. els for stories. The role of syntactic cues and [29] J. Oakhill, K. Cain, Issues of causality in children’s causal inferences, Typological Studies in Language reading comprehension, Reading comprehension 31, John Benjamins., Amsterdam; Philadelphia, strategies: Theories, interventions, and technolo1995. URL: https://www.degruyter.com/database/ gies (2007) 47–71.

COGBIB/entry/cogbib.6551/html, 2010. [30] B. Carretti, R. De Beni, C. Cornoldi, et al., Disturbi [17] RIDInet, https://www.anastasis.it/ridinet/), ???? della comprensione del testo, in: Dificoltà e disturbi [18] 3D Readers Software, https://www.sbir.gov/node/ dell’apprendimento, Il Mulino, 2007, pp. 143–162.

334161), ???? [31] W. Kintsch, T. A. Van Dijk, Toward a model of [19] A.-H. Kim, S. Vaughn, J. K. Klingner, A. L. Woodruf, text comprehension and production., Psychological C. Klein Reutebuch, K. Kouzekanani, Improving review 85 (1978) 363. the reading comprehension of middle school stu- [32] M. Galletti, E. Pasqua, F. Bianchi, M. Calanca, dents with disabilities through computer-assisted F. Padovani, D. Nardi, D. Tomaiuoli, A reading collaborative strategic reading, Remedial and spe- comprehension interface for students with learning cial education 27 (2006) 235–249. disorders, in: Companion Publication of the 25th In[20] E. Barbu, M. T. Martín-Valdivia, L. A. U. Lopez, ternational Conference on Multimodal Interaction, Open book: a tool for helping asd users’ semantic 2023, pp. 282–287. comprehension, in: Proceedings of the Workshop [33] M. Grootendorst, Keybert: Minimal keyword extracon Natural Language Processing for Improving Tex- tion with bert., 2020. URL: https://doi.org/10.5281/ tual Accessibility, 2013, pp. 11–19. zenodo.4461265. doi:10.5281/zenodo.4461265. [21] N. Pavlov, User interface for people with autism [34] R. Speer, rspeer/wordfreq: v3.0, 2022. URL: https: spectrum disorders, Journal of Software Engineer- //doi.org/10.5281/zenodo.7199437. doi:10.5281/ ing and Applications 2014 (2014). zenodo.7199437. [22] M. Wang, B. Muthu, C. Sivaparthipan, Smart as- [35] M. Galletti, M. Anslow, F. Bianchi, M. Calanca, sistance to dyslexia students using artificial intelli- D. Tomaiuoli, R. Van Trijp, E. Pasqua, Interactive gence based augmentative alternative communica- concept-map based summaries for send children, tion, International Journal of Speech Technology 2022, pp. 5236–5243.

(2021) 1–11. [23] A. Zingoni, J. Taborri, V. Panetti, S. Bonechi,

P. Aparicio-Martínez, S. Pinzi, G. Calabrò, Investigating issues and needs of dyslexic students at university: Proof of concept of an artificial intelligence and virtual reality-based supporting platform and preliminary results, Applied Sciences 11 (2021) 4624. [24] P. D. Barua, J. Vicnesh, R. Gururajan, S. L. Oh,

E. Palmer, M. M. Azizan, N. A. Kadri, U. R. Acharya, Artificial intelligence enabled personalised assistive tools to enhance education of children with neurodevelopmental disorders—a review, International Journal of Environmental Research and Public Health 19 (2022) 1192. [25] S. Rajapakse, D. Polwattage, U. Guruge, I. Jayathi

References

[1] J. Oakhill , Instantiation in skilled and less skilled comprehenders , The Quarterly Journal of Experimental Psychology Section A 35 ( 1983 ) 441 - 450 .

[2] K. Nation , M. J. Snowling , Factors influencing syntactic awareness skills in normal readers and poor comprehenders , Applied psycholinguistics 21 ( 2000 ) 229 - 241 .

[3] R. Padovani , La comprensione del testo scritto in età scolare. una rassegna sullo sviluppo normale e atipico , Psicologia clinica dello sviluppo 10 ( 2006 ) 369 - 398 .

[4] K. Nation , M. J. Snowling , Beyond phonological skills: Broader language skills contribute to the development of reading , Journal of research in reading 27 ( 2004 ) 342 - 356 .

[5] F. Arosio , C. Branchini , L. Barbieri , M. T. Guasti, Failure to produce direct object clitic pronouns as a clinical marker of sli in school-aged italian speaking children , Clinical linguistics & phonetics 28 ( 2014 ) 639 - 663 .

[6] A. Zachou , E. Partesana , E. Tenca, M. T. Guasti , et al., Production and comprehension of direct object clitics and definite articles by italian children with developmental dyslexia, Advances in language acquisition ( 2013 ) 464 - 471 .

[7] W. Kintsch , On comprehending stories, Cognitive processes in comprehension ( 1977 ) 33 - 62 .

[8] P. B. Gough , W. E. Tunmer , Decoding, reading, and reading disability, Remedial and special education 7 ( 1986 ) 6 - 10 .

[9] M. A. Gernsbacher , Language comprehension as structure building , Psychology Press, 2013 .

[10] M. A. Gernsbacher , Cognitive processes and mechanisms in language comprehension: The structure building framework, in: Psychology of learning and motivation , volume 27 , Elsevier , 1991 , pp. 217 - 263 .

[11] M. A. Gernsbacher , Two decades of structure building , Discourse processes 23 ( 1997 ) 265 - 304 .

[12] A. J. Sanford , S. C. Garrod , S. Garrod , Understanding written language: Explorations of comprehension beyond the sentence , John Wiley & Sons, 1981 .