=Paper=
{{Paper
|id=Vol-3751/short6
|storemode=property
|title=Comunicative Challenges and Augmented Reality for ASD
|pdfUrl=https://ceur-ws.org/Vol-3751/short6.pdf
|volume=Vol-3751
|authors=Angelo Rega,Andrea Mennitto,Salvatore Vita,Luigi Iovino
|dblpUrl=https://dblp.org/rec/conf/dilend/RegaMVI24
}}
==Comunicative Challenges and Augmented Reality for ASD==
https://ceur-ws.org/Vol-3751/short6.pdf
Comunicative Challenges and Augmented Reality for ASD
Angelo Rega1, Andrea Mennitto2, Salvatore Vita2 and Luigi Iovino2,3
1 UniversitàTelematicaPegaso, Italy
2 Garage94 – Laboratory for learning and technology, Ottaviano, 80044, Italy
3 NeapoliSanit Rehabilitation Center, Ottaviano, 80044, Italy
Abstract
Background. Communication skills are a major priority for the successful treatment of people with autism,
and technology has been in the recent years a powerful aid to rehabilitation professionals.
Method. This paper describes a concept of a tablet app called “LIAR”. LIAR, as in Language Interface For
Autistic’s Rehabilitation, is a software specifically designed for the treatment of autistic people in
developmental age. Our main goal is to build a dynamic PECS system, conjugating a sensory-enriched
environment to increase motivation (Technology Enhanced Learning) and assisting communication
through symbols. Our methodology, named i-Mand, is based on the Verbal Behavior theories as used by
the PECS system. The program consists of five phases, each focusing on helping the children acquire
specific communication-related skills.
Conclusions. LIAR combines proven and evidence-based methodologies with the latest frontiers of
technology. Our future researches will focus on evaluating the efficacy of LIAR in treating children with
ASD.
Keywords
communication, autism, speech generating devices, augmented reality, cloud, tablet, mand1
1. Introduction
Communication skills are a major priority for the successful treatment of people with autism in
developmental age [21]. According to Osterling, Dawson and McPartland (2001), 25% of children
with autism got little to no speech [4]. This lack in the use of speech is often due to restrictive and
repetitive repertories of interest, activity and limited interest to the others [22; 23]. Also many
children with autism are trying to communicate even though their attempts do not have a typical
outcome [15] resulting in misunderstandings and errors [11]. Training children with autism how to
communicate can decrease maladaptive behavior (aggressive and auto-aggressive behavior,
impulsivity) in frequency and severity, and enhance autoregulation and adaptation skills [13]. Some
authors suggested that by improving communication in persons with autism it could be achieved a
better social interaction and a decrease in problematic behavior [7]. The origins of augmentative
and alternative communication are in strategies like hand gestures and manual signs, that gradually
evolved in more advanced tools [6]. In particular, in this phase emerged low tech tools, that had in
common the non-electronic nature, like communication boards and graphic symbols [16]. Using
these tools, the subject could point to the desired object or exchange a symbol depicting an object or
an action. This last strategy is better known as Picture Exchange Communication System
(PECS)[16]. As a low-tech methodology, the Picture Exchange Communication System (PECS) is a
communication training program system largely used to enhance communication skills in children
with autism [2]. This program, divided into multiple individual training moments, can improve the
children's communication by using a set of cards depicting everyday objects and actions. While
PECS system grew in popularity, some more advanced technology began to emerge, leading to the
development of high tech Augmentative and Alternative Communication (AAC) software and
Proceedings of the Digital Innovations for Learning and Neurodevelopmental Disorders, May 24–25, 2024, Rome, Italy
angelo.rega@unipegaso.it (A. Rega); a.mennitto@neapolisanit.net(A. Mennitto); s.vita@neapolisanit.net(S. Vita);
l.iovino@neapolisanit.net (L. Iovino)
0000-0002-0641-7347(A. Rega); 0000-0003-1684-9140(A. Mennitto); 0000-0003-1684-9140(S. Vita); 0009-
0009-6455-2772(L. Iovino)
© 2024 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
hardware. This technological advancement leads to the invention of speech generating devices
(SGD), that offered to the user the opportunity to actively communicate what he wanted [16].
Speech Generating Devices (SGD) are currently used to help those children to express their needs
by providing an artificial voice output. Hardware boards are very easy to use, but can be quite
limited for number of words and interaction. The main drawback in the use of those devices is that
they were specifically designed for people with language impairments, and partially adapted to
autism only recently. The most important limitations in applying this systems in treating autistic
subjects are their high cost, but above all the scarce possibility of personalization [16]. Autism is a
spectrum disorder, meaning that every person has different need and for that reason it is unlikely
that everyone could benefit in the same way from a static system like SGD devices [3]. On the other
hand new technologies, constantly evolving in the last decade, are capable of bringing new
potential advantages in terms of both hardware and software features. Specifically, modern tablets
are very light and portable, come packed with advanced features (GPS, gyroscope, internet access,
etc.) and are part of everyday objects for the majority of the population. This last feature is very
useful in order to enhance the children's motivation [1] and to reduce the stigma [14]. In a recent
meta-analysis [8], the authors found that the affordability and accessibility of iPods and iPads, in
conjunction with the fact that they are socially acceptable objects, where the cause of their
popularity. In other words, by using these devices, the person does not feel stigmatized or
uncomfortable. In particular it is very effective introducing iPad with adults, because they can
already use it and identify the object as an everyday item, unlike other AAC devices, that have to be
learned and mastered by the user, who needs to familiarize with menus, navigation tools and new
commands [12]. So, even though both PECS systems and iPad require learning and preparation
time, the advantages of the iPad are clear: this device needs the least work to be implemented, as
opposed to the time needed by the PECS to prepare, store, and move the cards depicting the objects
[5]. In light of what has been said, high-tech devices like iPads, and most generally tablets provide a
very good opportunity in improving and personalizing the approach to communication skills in
autistic subjects. It is for this reason that arises the necessity to start a systematic study on how to
design and implement a tablet app specifically created to meet the needs of the professionals
working in the field of rehabilitation of communication skills. The starting point has to be a
multidisciplinary approach that takes into account the contribution of every expert in the treatment
of autism that is part of the team. Recently, a lot of educational apps were designed to interact with
the environment, getting and visualizing informations in real time to the end user in order to adapt
dynamically. Augmented Reality (AR) apps are applications that combine digital information with
the real world. The most popular AR apps are the one that overlay virtual pictures on a feed from
the user’s camera. This technology uses device’s sensors to detect the environment and to arrange
the overlay visualization in real time. Using the experience of speech therapists, psychologist and
medical doctors working in our rehabilitation center with these instruments, we built the first
concept of LIAR.
2. Method
Our team started to design and build an app which aims to foster communication of children with
autism, by working on a multi step program. In the following lines we will describe the concept we
came up with.
The methodology of LIAR is the i-MAND which is a specific mand training teaching method
based on Verbal Behavior theory. According to Skinner (1957), a mand is “a verbal operant in which
the response is reinforced by a characteristic consequence and is therefore under the functional
control of relevant conditions of deprivation or aversive stimulation”. In other word, a mand is a
request for a specific preferred event and is the only verbal operant for which a response is directly
evoked by a motivating operation [MO; 9; 17]. Researchers have explored a large number of
behavior analytic procedures, that have been demonstrated to be effective in teaching children to
vocal mand[10], manual sign language [18], picture exchange communication system (PECS) [19],
and speech-generating devices (SGDs)[20]. About this last, however, there is no set of specific
procedures for teaching children with ASD how to operate and communicate with them. LIAR is
been designed with the purpose of create a clear and systematic procedure to implement it. In In
fact, it is composed by a 5-step teaching method that effectively allow children how to use the
system, starting from the use of touch-screen (warm-up) until to reach the request with a sentence
(mand phase).
LIAR is a software leveraging modern, cheap hardware commercially available. The software is
written in C#, using the Unity Engine (www.unity3d.com) to ensure broad devices support: in fact,
every post-2015 Windows, Android and iOS tablet can run effortlessly our app. The User Interface
design uses light colors and simple shapes in order to highlight the user-defined content. The
customization menu is hidden to the end user, as the therapist has to press a button on a Bluetooth
gamepad to show the content. This solution was adopted in order to allow the therapist to interact
with the tablet effortlessly and out of the sight of the child, avoiding the interruption of his
environment exploration and the loss of attention.
The hardware controller, a universal 4-button Bluetooth device built by BlueBeach (Fig. 1), it’s
tiny (6,2 x 5,6 x 1 cm) and light (41 g), offer a long-lasting built-in battery and can be associated with
every target Operating System.
Figure 1:Bluetooth Controller
The graphical interface is optimized for tablet with display sizes from 8 to 10 inches, because every
on-screen element can be dynamically adapted. In order to ensure the best user experience, edgeless
devices are recommended. LIAR is compatible with ARMv7 (Cortex) CPU or Atom CPU. Is also
required full support for OpenGL ES 2.0 or later. The device must have a rear camera to allow the
user to add new pictures as photos of his world. Near Field Communication (NFC) and BLE
(Bluetooth Low Energy) chip are required to let the device interact with special, RFID-tagged items
and the therapist’s controller.
The methodology used for the development is based on 4 steps:
1) Analysis of the needs: using the data from 4 focus groups conducted with therapists and
psychologists, it was aimed to find the best way to treat children with ASD, discussing the PECS
methodology, sharing videos of treatment sessions, gathering ideas and requests.
2) Systematic Review: the most recent tablet applications were tested and analyzed in the
Focus Group. Every app has been classified using a rubric from Boyd et al. (2015), outlining
advantages, usability, customizability and costs [3].
3) Development environment and target platform: target of this step was to find the best
platform and development environment to target while building the first prototype. Scalability,
programming languages, integration of peripherals and capabilities of cross-compilation are been
taken into account.
4) Design and development: flow charts were made to describe every functionality to the
developers. The team’s work was focused on usability and user interface, with the purpose of
smoothing the approach both to the children and the operators. Long and non useful actions were
redefined, distracting elements were removed. The whole user experience was optimized to reduce
cognitive load and excessive color stimulation. During the development of the application we
avoided accidental interactions to limit the frustration level. Through the design phase we also tried
to hide the inner technical details and setup settings to the user. All the on-screen objects are
designed to be interactive with a single tap, as the size of the items are adequate also for children
with difficulties in motor coordination. In the following paragraphs the single phases will be
described, from the user interface to the methodology used.
2.1. Main Menu
The main menu (Fig. 2) shows six squared buttons, grayed-out by default, that can be unlocked and
highlighted by the therapist, allowing the child to focus and interact exclusively with the designated
exercise.
Figure2:Main Menu
White, unique-looking icons were used to differentiate all the activities, giving less visual contrast
to locked phases but keeping them on screen, as a visual indicator of the route through the app.
2.2. Warm up
This phase is useful to familiarize children with the drag and drop mechanic we will use in the
whole app. It is also useful to increase sensomotor coordination: in fact, many children with autism
have issues with fine motor skills [12] that can be trained in this phase.
Figure 3:“Warm Up” boxes
Figure 4:“Warm Up” customization menu
On a black screen, two squares are presented: a full and a hollow one (Fig. 3). Our patients have to
drag the full one in the hollow, covering at least 80% of the blank space in order to complete the
task, triggering voice and visual reward. Both can be customized (Fig. 4) by the therapist, that can
choose to give a physical reinforcement too. The therapist can also schedule different reinforcement
ratios, change the reinforcement image and increase the size of the squares: this feature, in
particular, is useful for younger or movement-impaired patients. The operator can also use a
Bluetooth gamepad to autonomously trigger the rewarding sequence, as a way to shape the wanted
behavior. In this phase, some customization was needed to best help our target audience: multi
touch is disabled and touch sensitivity is toned down.
2.3. First Steps
This phase is about gradual exposing the user to the communication interface. The operator has to
press a gamepad button in order to spawn a picture. The child can drag this picture onto the white
square in order to let the tablet speak the object name and to receive the dragged item in real world
as a reinforcement. In the settings' menu it is possible to customize a despawn rate for the pictures
and to select multiple images for all the buttons on the gamepad.
Figure 5:“First Steps”
Figure 6:“First Steps” customization menu
2.4. I want…
In this phase, visually identical to the previous one, the operator has to trigger the visualization of a
picture by pressing a button on the gamepad. The child can drag and drop the showed pic on a
target square, but the tablet does not emit any sound unless the game pad is pressed another time.
The operator can define the stimulus to display and has to define with the child a “hand off”
behavior (i.e.: giving the tablet to the therapist, looking for eye contact, etc.).
2.5. Mand Discrimination
This phase, visually similar to the other ones, keeps the “I Want...” confirmation needs. The new
element is that the therapist can choose to show, along with the target, other images, by a
randomized list of reinforced and unreinforced items. The child has to discriminate the correct
stimulus and drag and drop it on the white box.
Figure 7: “Mand Discrimination” customization menu
2.6. Mand Phase
This phase extends the software UI, showing more information like picture’s categories, target
squares and a play button to start the text to speech function. The child has a library of pictures he
can improve and organize within the software, using images from the internet or from the tablet’s
camera. This section includes augmented reality technology, as localization and RFID. Localization
is useful to discriminate between settings by highlightings the objects used in that particular
environment. For example, when the child is at school, the software will show first items like books,
pencils, etc. The software can also read RFID tags placed on specific objects used in therapy room
for training. Bringing the tablet near a tag, the user can interact with the attached object, watching
a video on how to use it and can request it by using the drag and drop UI.
Figure 8: “Mand Phase”
2.7. Cloud Database
A cloud platform was also deployed to collect anonymized data of every interaction with our
software in milliseconds accuracy. Only the therapist, who got physical access to the tablet, can
search for personalized data. A growing number of statistics are generated on the fly to report
overall and specific trends in the everyday use. The therapist can also export statistics and raw data,
in order to give feedback to the family of ask for a supervision.
3. Discussion and Implications
LIAR is the result of years of work in the NeapoliSanit rehabilitation center. It is born to combine
proven and evidence-based methodologies with the latest frontiers of technology. The research and
development team has benefited from the contribution and experience of rehabilitation technicians
in order to meet the needs emerging in their daily activities.
Our goal is to develop a system capable of supporting people’s communicative exchanges and
enriching their environment with interactive objects through the use of Augmented Reality. Those
objects can be a powerful stimulus to formulate a request (Mand).
Making use of dynamics facilitators during the learning process should result in a very efficient
system. Therefore, our future researches will focus on:
• Evaluating the efficacy of a communication methodology based on interactive objects and
pictures and comparing it to the traditional PECS approach.
• Observing the number of self-learning occurrences in a system that encourage environment
exploration and communicative exchanges.
References
[1] Bhatt, S. K., De Leon, N. I., & Al-Jumaily, A. (2014). Augmented reality game therapy for
children with autism spectrum disorder. International Journal on Smart Sensing & Intelligent
Systems, 7(2).
[2] Bondy, A. S., & Frost, L. A. (1994). The picture exchange communication system. Focus on
Autistic Behavior, 9(3), 1–19.
[3] Boyd, T. K., Hart Barnett, J. E., & More, C. M. (2015). Evaluating iPad technology for enhancing
communication skills of children with Autism Spectrum Disorders. Intervention in School and
Clinic, 51(1), 19–27.
[4] Dawson, G., Osterling, J., Rinaldi, J., Carver, L., & McPartland, J. (2001). Brief report:
Recognition memory and stimulus-reward associations: Indirect support for the role of
ventromedial prefrontal dysfunction in autism. Journal of Autism and Developmental Disorders,
31(3), 337–341.
[5] Flores, M., Musgrove, K., Renner, S., Hinton, V., Strozier, S., Franklin, S., &Hil, D. (2012). A
comparison of communication using the Apple iPad and a picture-based system. Augmentative
and Alternative Communication, 28(2), 74–84.
[6] Frolli A, Ciotola S, Esposito C, Fraschetti S, Ricci MC, Cerciello F, Russo MG. AAC and Autism:
ManualSigns and Pecs, a Comparison. Behavioral Sciences. 2022; 12(10):359.
[7] Ganz, J. B., Earles-Vollrath, T. L., Heath, A. K., Parker, R. I., Rispoli, M. J., & Duran, J. B. (2012).
A meta-analysis of single case research studies on aided augmentative and alternative
communication systems with individuals with autism spectrum disorders. Journal of Autism
and Developmental Disorders, 42(1), 60–74.
[8] Kagohara, D. M., van der Meer, L., Ramdoss, S., O’Reilly, M. F., Lancioni, G. E., Davis, T. N., …
Sutherland, D. (2013). Using iPods® and iPads® in teaching programs for individuals with
developmental disabilities: A systematic review. Research in Developmental Disabilities, 34(1),
147–156.
[9] Laraway, S., Snycerski, S., Michael, J., & Poling, A. (2003). Motivating operations and terms to
describe them: Some further refinements. Journal of Applied Behavior Analysis, 36(3), 407–414.
[10] Lechago, S. A., Carr, J. E., Grow, L. L., Love, J. R., &Almason, S. M. (2010). Mands for
information generalize across establishing operations. Journal of Applied Behavior Analysis,
43(3), 381–395.
[11] Marans, W. D., Rubin, E., & Laurent, A. (2005). Addressing social communication skills in
individuals with high‐functioning autism and asperger syndrome: Critical priorities in
educational programming. Handbook of Autism and Pervasive Developmental Disorders, Volume
2, Third Edition, 977–1002.
[12] McNaughton, D., & Light, J. (2013). The iPad and mobile technology revolution: Benefits and
challenges for individuals who require augmentative and alternative communication. Taylor &
Francis.
[13] Mirenda, P. (2003). Toward functional augmentative and alternative communication for
students with autism: Manual signs, graphic symbols, and voice output communication aids.
Language, Speech, and Hearing Services in Schools, 34(3), 203–216.
[14] Mirenda, P., & Iacono, T. (2009). Autism spectrum disorders and AAC. ASHA.
[15] Prizant, B. M., & Wetherby, A. M. (2005). Critical issues in enhancing communication abilities
for persons with autism spectrum disorders. Handbook of Autism and Pervasive Developmental
Disorders, Volume 2, Third Edition, 925–945.
[16] Shane, H. C., Laubscher, E. H., Schlosser, R. W., Flynn, S., Sorce, J. F., & Abramson, J. (2012).
Applying technology to visually support language and communication in individuals with
autism spectrum disorders. Journal of Autism and Developmental Disorders, 42(6), 1228–1235.
[17] Skinner, B. F. (1957). Verbal Behavior, Englewood Cliffs, NJ: Prentice Hall.
[18] Tincani, M. (2004). Comparing the picture exchange communication system and sign language
training for children with autism. Focus on Autism and Other Developmental Disabilities, 19(3),
152–163.
[19] Tincani, M., Crozier, S., &Alazetta, L. (2006). The Picture Exchange Communication System:
Effects on manding and speech development for school-aged children with autism. Education
and Training in Developmental Disabilities, 177–184.
[20] van der Meer, L., Sutherland, D., O’Reilly, M. F., Lancioni, G. E., & Sigafoos, J. (2012). A further
comparison of manual signing, picture exchange, and speech-generating devices as
communication modes for children with autism spectrum disorders. Research in Autism
Spectrum Disorders, 6(4), 1247–1257.
[21] Frolli, A., Cerciello, F., Esposito, C., Rega, A., Ricci, C., & Moderato, P. (2022).A descriptive
survey on Applied Behavioral Analysis: a sample of Italian teachers.
[22] Piedimonte, A., Conson, M., Frolli, A., Bari, S., Della Gatta, F., Rabuffetti, M., Keller, R., Berti, A.,
& Garbarini, F. (2018). Dissociation between executed and imagined bimanual movements in
autism spectrum conditions. Autism research : official journal of the International Society for
Autism Research, 11(2), 376–384. https://doi.org/10.1002/aur.1902
[23] Caliendo, M., Di Sessa, A., D'Alterio, E., Frolli, A., Verde, D., Iacono, D., Romano, P., Vetri, L., &
Carotenuto, M. (2021). Efficacy of Neuro-Psychomotor Approach in Children Affected by
Autism Spectrum Disorders: A Multicenter Study in Italian Pediatric Population. Brain
sciences, 11(9), 1210. https://doi.org/10.3390/brainsci11091210