Comunicative Challenges and Augmented Reality for ASD Angelo Rega1, Andrea Mennitto2, Salvatore Vita2 and Luigi Iovino2,3 1 UniversitàTelematicaPegaso, Italy 2 Garage94 – Laboratory for learning and technology, Ottaviano, 80044, Italy 3 NeapoliSanit Rehabilitation Center, Ottaviano, 80044, Italy Abstract Background. Communication skills are a major priority for the successful treatment of people with autism, and technology has been in the recent years a powerful aid to rehabilitation professionals. Method. This paper describes a concept of a tablet app called “LIAR”. LIAR, as in Language Interface For Autistic’s Rehabilitation, is a software specifically designed for the treatment of autistic people in developmental age. Our main goal is to build a dynamic PECS system, conjugating a sensory-enriched environment to increase motivation (Technology Enhanced Learning) and assisting communication through symbols. Our methodology, named i-Mand, is based on the Verbal Behavior theories as used by the PECS system. The program consists of five phases, each focusing on helping the children acquire specific communication-related skills. Conclusions. LIAR combines proven and evidence-based methodologies with the latest frontiers of technology. Our future researches will focus on evaluating the efficacy of LIAR in treating children with ASD. Keywords communication, autism, speech generating devices, augmented reality, cloud, tablet, mand1 1. Introduction Communication skills are a major priority for the successful treatment of people with autism in developmental age [21]. According to Osterling, Dawson and McPartland (2001), 25% of children with autism got little to no speech [4]. This lack in the use of speech is often due to restrictive and repetitive repertories of interest, activity and limited interest to the others [22; 23]. Also many children with autism are trying to communicate even though their attempts do not have a typical outcome [15] resulting in misunderstandings and errors [11]. Training children with autism how to communicate can decrease maladaptive behavior (aggressive and auto-aggressive behavior, impulsivity) in frequency and severity, and enhance autoregulation and adaptation skills [13]. Some authors suggested that by improving communication in persons with autism it could be achieved a better social interaction and a decrease in problematic behavior [7]. The origins of augmentative and alternative communication are in strategies like hand gestures and manual signs, that gradually evolved in more advanced tools [6]. In particular, in this phase emerged low tech tools, that had in common the non-electronic nature, like communication boards and graphic symbols [16]. Using these tools, the subject could point to the desired object or exchange a symbol depicting an object or an action. This last strategy is better known as Picture Exchange Communication System (PECS)[16]. As a low-tech methodology, the Picture Exchange Communication System (PECS) is a communication training program system largely used to enhance communication skills in children with autism [2]. This program, divided into multiple individual training moments, can improve the children's communication by using a set of cards depicting everyday objects and actions. While PECS system grew in popularity, some more advanced technology began to emerge, leading to the development of high tech Augmentative and Alternative Communication (AAC) software and Proceedings of the Digital Innovations for Learning and Neurodevelopmental Disorders, May 24–25, 2024, Rome, Italy angelo.rega@unipegaso.it (A. Rega); a.mennitto@neapolisanit.net(A. Mennitto); s.vita@neapolisanit.net(S. Vita); l.iovino@neapolisanit.net (L. Iovino) 0000-0002-0641-7347(A. Rega); 0000-0003-1684-9140(A. Mennitto); 0000-0003-1684-9140(S. Vita); 0009- 0009-6455-2772(L. Iovino) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings hardware. This technological advancement leads to the invention of speech generating devices (SGD), that offered to the user the opportunity to actively communicate what he wanted [16]. Speech Generating Devices (SGD) are currently used to help those children to express their needs by providing an artificial voice output. Hardware boards are very easy to use, but can be quite limited for number of words and interaction. The main drawback in the use of those devices is that they were specifically designed for people with language impairments, and partially adapted to autism only recently. The most important limitations in applying this systems in treating autistic subjects are their high cost, but above all the scarce possibility of personalization [16]. Autism is a spectrum disorder, meaning that every person has different need and for that reason it is unlikely that everyone could benefit in the same way from a static system like SGD devices [3]. On the other hand new technologies, constantly evolving in the last decade, are capable of bringing new potential advantages in terms of both hardware and software features. Specifically, modern tablets are very light and portable, come packed with advanced features (GPS, gyroscope, internet access, etc.) and are part of everyday objects for the majority of the population. This last feature is very useful in order to enhance the children's motivation [1] and to reduce the stigma [14]. In a recent meta-analysis [8], the authors found that the affordability and accessibility of iPods and iPads, in conjunction with the fact that they are socially acceptable objects, where the cause of their popularity. In other words, by using these devices, the person does not feel stigmatized or uncomfortable. In particular it is very effective introducing iPad with adults, because they can already use it and identify the object as an everyday item, unlike other AAC devices, that have to be learned and mastered by the user, who needs to familiarize with menus, navigation tools and new commands [12]. So, even though both PECS systems and iPad require learning and preparation time, the advantages of the iPad are clear: this device needs the least work to be implemented, as opposed to the time needed by the PECS to prepare, store, and move the cards depicting the objects [5]. In light of what has been said, high-tech devices like iPads, and most generally tablets provide a very good opportunity in improving and personalizing the approach to communication skills in autistic subjects. It is for this reason that arises the necessity to start a systematic study on how to design and implement a tablet app specifically created to meet the needs of the professionals working in the field of rehabilitation of communication skills. The starting point has to be a multidisciplinary approach that takes into account the contribution of every expert in the treatment of autism that is part of the team. Recently, a lot of educational apps were designed to interact with the environment, getting and visualizing informations in real time to the end user in order to adapt dynamically. Augmented Reality (AR) apps are applications that combine digital information with the real world. The most popular AR apps are the one that overlay virtual pictures on a feed from the user’s camera. This technology uses device’s sensors to detect the environment and to arrange the overlay visualization in real time. Using the experience of speech therapists, psychologist and medical doctors working in our rehabilitation center with these instruments, we built the first concept of LIAR. 2. Method Our team started to design and build an app which aims to foster communication of children with autism, by working on a multi step program. In the following lines we will describe the concept we came up with. The methodology of LIAR is the i-MAND which is a specific mand training teaching method based on Verbal Behavior theory. According to Skinner (1957), a mand is “a verbal operant in which the response is reinforced by a characteristic consequence and is therefore under the functional control of relevant conditions of deprivation or aversive stimulation”. In other word, a mand is a request for a specific preferred event and is the only verbal operant for which a response is directly evoked by a motivating operation [MO; 9; 17]. Researchers have explored a large number of behavior analytic procedures, that have been demonstrated to be effective in teaching children to vocal mand[10], manual sign language [18], picture exchange communication system (PECS) [19], and speech-generating devices (SGDs)[20]. About this last, however, there is no set of specific procedures for teaching children with ASD how to operate and communicate with them. LIAR is been designed with the purpose of create a clear and systematic procedure to implement it. In In fact, it is composed by a 5-step teaching method that effectively allow children how to use the system, starting from the use of touch-screen (warm-up) until to reach the request with a sentence (mand phase). LIAR is a software leveraging modern, cheap hardware commercially available. The software is written in C#, using the Unity Engine (www.unity3d.com) to ensure broad devices support: in fact, every post-2015 Windows, Android and iOS tablet can run effortlessly our app. The User Interface design uses light colors and simple shapes in order to highlight the user-defined content. The customization menu is hidden to the end user, as the therapist has to press a button on a Bluetooth gamepad to show the content. This solution was adopted in order to allow the therapist to interact with the tablet effortlessly and out of the sight of the child, avoiding the interruption of his environment exploration and the loss of attention. The hardware controller, a universal 4-button Bluetooth device built by BlueBeach (Fig. 1), it’s tiny (6,2 x 5,6 x 1 cm) and light (41 g), offer a long-lasting built-in battery and can be associated with every target Operating System. Figure 1:Bluetooth Controller The graphical interface is optimized for tablet with display sizes from 8 to 10 inches, because every on-screen element can be dynamically adapted. In order to ensure the best user experience, edgeless devices are recommended. LIAR is compatible with ARMv7 (Cortex) CPU or Atom CPU. Is also required full support for OpenGL ES 2.0 or later. The device must have a rear camera to allow the user to add new pictures as photos of his world. Near Field Communication (NFC) and BLE (Bluetooth Low Energy) chip are required to let the device interact with special, RFID-tagged items and the therapist’s controller. The methodology used for the development is based on 4 steps: 1) Analysis of the needs: using the data from 4 focus groups conducted with therapists and psychologists, it was aimed to find the best way to treat children with ASD, discussing the PECS methodology, sharing videos of treatment sessions, gathering ideas and requests. 2) Systematic Review: the most recent tablet applications were tested and analyzed in the Focus Group. Every app has been classified using a rubric from Boyd et al. (2015), outlining advantages, usability, customizability and costs [3]. 3) Development environment and target platform: target of this step was to find the best platform and development environment to target while building the first prototype. Scalability, programming languages, integration of peripherals and capabilities of cross-compilation are been taken into account. 4) Design and development: flow charts were made to describe every functionality to the developers. The team’s work was focused on usability and user interface, with the purpose of smoothing the approach both to the children and the operators. Long and non useful actions were redefined, distracting elements were removed. The whole user experience was optimized to reduce cognitive load and excessive color stimulation. During the development of the application we avoided accidental interactions to limit the frustration level. Through the design phase we also tried to hide the inner technical details and setup settings to the user. All the on-screen objects are designed to be interactive with a single tap, as the size of the items are adequate also for children with difficulties in motor coordination. In the following paragraphs the single phases will be described, from the user interface to the methodology used. 2.1. Main Menu The main menu (Fig. 2) shows six squared buttons, grayed-out by default, that can be unlocked and highlighted by the therapist, allowing the child to focus and interact exclusively with the designated exercise. Figure2:Main Menu White, unique-looking icons were used to differentiate all the activities, giving less visual contrast to locked phases but keeping them on screen, as a visual indicator of the route through the app. 2.2. Warm up This phase is useful to familiarize children with the drag and drop mechanic we will use in the whole app. It is also useful to increase sensomotor coordination: in fact, many children with autism have issues with fine motor skills [12] that can be trained in this phase. Figure 3:“Warm Up” boxes Figure 4:“Warm Up” customization menu On a black screen, two squares are presented: a full and a hollow one (Fig. 3). Our patients have to drag the full one in the hollow, covering at least 80% of the blank space in order to complete the task, triggering voice and visual reward. Both can be customized (Fig. 4) by the therapist, that can choose to give a physical reinforcement too. The therapist can also schedule different reinforcement ratios, change the reinforcement image and increase the size of the squares: this feature, in particular, is useful for younger or movement-impaired patients. The operator can also use a Bluetooth gamepad to autonomously trigger the rewarding sequence, as a way to shape the wanted behavior. In this phase, some customization was needed to best help our target audience: multi touch is disabled and touch sensitivity is toned down. 2.3. First Steps This phase is about gradual exposing the user to the communication interface. The operator has to press a gamepad button in order to spawn a picture. The child can drag this picture onto the white square in order to let the tablet speak the object name and to receive the dragged item in real world as a reinforcement. In the settings' menu it is possible to customize a despawn rate for the pictures and to select multiple images for all the buttons on the gamepad. Figure 5:“First Steps” Figure 6:“First Steps” customization menu 2.4. I want… In this phase, visually identical to the previous one, the operator has to trigger the visualization of a picture by pressing a button on the gamepad. The child can drag and drop the showed pic on a target square, but the tablet does not emit any sound unless the game pad is pressed another time. The operator can define the stimulus to display and has to define with the child a “hand off” behavior (i.e.: giving the tablet to the therapist, looking for eye contact, etc.). 2.5. Mand Discrimination This phase, visually similar to the other ones, keeps the “I Want...” confirmation needs. The new element is that the therapist can choose to show, along with the target, other images, by a randomized list of reinforced and unreinforced items. The child has to discriminate the correct stimulus and drag and drop it on the white box. Figure 7: “Mand Discrimination” customization menu 2.6. Mand Phase This phase extends the software UI, showing more information like picture’s categories, target squares and a play button to start the text to speech function. The child has a library of pictures he can improve and organize within the software, using images from the internet or from the tablet’s camera. This section includes augmented reality technology, as localization and RFID. Localization is useful to discriminate between settings by highlightings the objects used in that particular environment. For example, when the child is at school, the software will show first items like books, pencils, etc. The software can also read RFID tags placed on specific objects used in therapy room for training. Bringing the tablet near a tag, the user can interact with the attached object, watching a video on how to use it and can request it by using the drag and drop UI. Figure 8: “Mand Phase” 2.7. Cloud Database A cloud platform was also deployed to collect anonymized data of every interaction with our software in milliseconds accuracy. Only the therapist, who got physical access to the tablet, can search for personalized data. A growing number of statistics are generated on the fly to report overall and specific trends in the everyday use. The therapist can also export statistics and raw data, in order to give feedback to the family of ask for a supervision. 3. Discussion and Implications LIAR is the result of years of work in the NeapoliSanit rehabilitation center. It is born to combine proven and evidence-based methodologies with the latest frontiers of technology. The research and development team has benefited from the contribution and experience of rehabilitation technicians in order to meet the needs emerging in their daily activities. Our goal is to develop a system capable of supporting people’s communicative exchanges and enriching their environment with interactive objects through the use of Augmented Reality. Those objects can be a powerful stimulus to formulate a request (Mand). Making use of dynamics facilitators during the learning process should result in a very efficient system. Therefore, our future researches will focus on: • Evaluating the efficacy of a communication methodology based on interactive objects and pictures and comparing it to the traditional PECS approach. • Observing the number of self-learning occurrences in a system that encourage environment exploration and communicative exchanges. References [1] Bhatt, S. K., De Leon, N. I., & Al-Jumaily, A. (2014). Augmented reality game therapy for children with autism spectrum disorder. International Journal on Smart Sensing & Intelligent Systems, 7(2). [2] Bondy, A. S., & Frost, L. A. (1994). The picture exchange communication system. Focus on Autistic Behavior, 9(3), 1–19. [3] Boyd, T. K., Hart Barnett, J. E., & More, C. M. (2015). Evaluating iPad technology for enhancing communication skills of children with Autism Spectrum Disorders. Intervention in School and Clinic, 51(1), 19–27. [4] Dawson, G., Osterling, J., Rinaldi, J., Carver, L., & McPartland, J. (2001). Brief report: Recognition memory and stimulus-reward associations: Indirect support for the role of ventromedial prefrontal dysfunction in autism. Journal of Autism and Developmental Disorders, 31(3), 337–341. [5] Flores, M., Musgrove, K., Renner, S., Hinton, V., Strozier, S., Franklin, S., &Hil, D. (2012). A comparison of communication using the Apple iPad and a picture-based system. Augmentative and Alternative Communication, 28(2), 74–84. [6] Frolli A, Ciotola S, Esposito C, Fraschetti S, Ricci MC, Cerciello F, Russo MG. AAC and Autism: ManualSigns and Pecs, a Comparison. Behavioral Sciences. 2022; 12(10):359. [7] Ganz, J. B., Earles-Vollrath, T. L., Heath, A. K., Parker, R. I., Rispoli, M. J., & Duran, J. B. (2012). A meta-analysis of single case research studies on aided augmentative and alternative communication systems with individuals with autism spectrum disorders. Journal of Autism and Developmental Disorders, 42(1), 60–74. [8] Kagohara, D. M., van der Meer, L., Ramdoss, S., O’Reilly, M. F., Lancioni, G. E., Davis, T. N., … Sutherland, D. (2013). Using iPods® and iPads® in teaching programs for individuals with developmental disabilities: A systematic review. Research in Developmental Disabilities, 34(1), 147–156. [9] Laraway, S., Snycerski, S., Michael, J., & Poling, A. (2003). Motivating operations and terms to describe them: Some further refinements. Journal of Applied Behavior Analysis, 36(3), 407–414. [10] Lechago, S. A., Carr, J. E., Grow, L. L., Love, J. R., &Almason, S. M. (2010). Mands for information generalize across establishing operations. Journal of Applied Behavior Analysis, 43(3), 381–395. [11] Marans, W. D., Rubin, E., & Laurent, A. (2005). Addressing social communication skills in individuals with high‐functioning autism and asperger syndrome: Critical priorities in educational programming. Handbook of Autism and Pervasive Developmental Disorders, Volume 2, Third Edition, 977–1002. [12] McNaughton, D., & Light, J. (2013). The iPad and mobile technology revolution: Benefits and challenges for individuals who require augmentative and alternative communication. Taylor & Francis. [13] Mirenda, P. (2003). Toward functional augmentative and alternative communication for students with autism: Manual signs, graphic symbols, and voice output communication aids. Language, Speech, and Hearing Services in Schools, 34(3), 203–216. [14] Mirenda, P., & Iacono, T. (2009). Autism spectrum disorders and AAC. ASHA. [15] Prizant, B. M., & Wetherby, A. M. (2005). Critical issues in enhancing communication abilities for persons with autism spectrum disorders. Handbook of Autism and Pervasive Developmental Disorders, Volume 2, Third Edition, 925–945. [16] Shane, H. C., Laubscher, E. H., Schlosser, R. W., Flynn, S., Sorce, J. F., & Abramson, J. (2012). Applying technology to visually support language and communication in individuals with autism spectrum disorders. Journal of Autism and Developmental Disorders, 42(6), 1228–1235. [17] Skinner, B. F. (1957). Verbal Behavior, Englewood Cliffs, NJ: Prentice Hall. [18] Tincani, M. (2004). Comparing the picture exchange communication system and sign language training for children with autism. Focus on Autism and Other Developmental Disabilities, 19(3), 152–163. [19] Tincani, M., Crozier, S., &Alazetta, L. (2006). The Picture Exchange Communication System: Effects on manding and speech development for school-aged children with autism. Education and Training in Developmental Disabilities, 177–184. [20] van der Meer, L., Sutherland, D., O’Reilly, M. F., Lancioni, G. E., & Sigafoos, J. (2012). A further comparison of manual signing, picture exchange, and speech-generating devices as communication modes for children with autism spectrum disorders. Research in Autism Spectrum Disorders, 6(4), 1247–1257. [21] Frolli, A., Cerciello, F., Esposito, C., Rega, A., Ricci, C., & Moderato, P. (2022).A descriptive survey on Applied Behavioral Analysis: a sample of Italian teachers. [22] Piedimonte, A., Conson, M., Frolli, A., Bari, S., Della Gatta, F., Rabuffetti, M., Keller, R., Berti, A., & Garbarini, F. (2018). Dissociation between executed and imagined bimanual movements in autism spectrum conditions. Autism research : official journal of the International Society for Autism Research, 11(2), 376–384. https://doi.org/10.1002/aur.1902 [23] Caliendo, M., Di Sessa, A., D'Alterio, E., Frolli, A., Verde, D., Iacono, D., Romano, P., Vetri, L., & Carotenuto, M. (2021). Efficacy of Neuro-Psychomotor Approach in Children Affected by Autism Spectrum Disorders: A Multicenter Study in Italian Pediatric Population. Brain sciences, 11(9), 1210. https://doi.org/10.3390/brainsci11091210