Communication skills are a major priority for the successful treatment of people with autism in developmental age [21]. According to Osterling, Dawson and McPartland (2001), 25% of children with autism got little to no speech [4]. This lack in the use of speech is often due to restrictive and repetitive repertories of interest, activity and limited interest to the others [22; 23]. Also many children with autism are trying to communicate even though their attempts do not have a typical outcome [15] resulting in misunderstandings and errors [11]. Training children with autism how to communicate can decrease maladaptive behavior (aggressive and auto-aggressive behavior, impulsivity) in frequency and severity, and enhance autoregulation and adaptation skills [13]. Some authors suggested that by improving communication in persons with autism it could be achieved a better social interaction and a decrease in problematic behavior [7]. The origins of augmentative and alternative communication are in strategies like hand gestures and manual signs, that gradually evolved in more advanced tools [6]. In particular, in this phase emerged low tech tools, that had in common the non-electronic nature, like communication boards and graphic symbols [16]. Using these tools, the subject could point to the desired object or exchange a symbol depicting an object or an action. This last strategy is better known as Picture Exchange Communication System (PECS) [16]. As a low-tech methodology, the Picture Exchange Communication System (PECS) is a communication training program system largely used to enhance communication skills in children with autism [2]. This program, divided into multiple individual training moments, can improve the children's communication by using a set of cards depicting everyday objects and actions. While PECS system grew in popularity, some more advanced technology began to emerge, leading to the development of high tech Augmentative and Alternative Communication (AAC) software and hardware. This technological advancement leads to the invention of speech generating devices (SGD), that offered to the user the opportunity to actively communicate what he wanted [16]. Speech Generating Devices (SGD) are currently used to help those children to express their needs by providing an artificial voice output. Hardware boards are very easy to use, but can be quite limited for number of words and interaction. The main drawback in the use of those devices is that they were specifically designed for people with language impairments, and partially adapted to autism only recently. The most important limitations in applying this systems in treating autistic subjects are their high cost, but above all the scarce possibility of personalization [16]. Autism is a spectrum disorder, meaning that every person has different need and for that reason it is unlikely that everyone could benefit in the same way from a static system like SGD devices [3]. On the other hand new technologies, constantly evolving in the last decade, are capable of bringing new potential advantages in terms of both hardware and software features. Specifically, modern tablets are very light and portable, come packed with advanced features (GPS, gyroscope, internet access, etc.) and are part of everyday objects for the majority of the population. This last feature is very useful in order to enhance the children's motivation [1] and to reduce the stigma [14]. In a recent meta-analysis [8], the authors found that the affordability and accessibility of iPods and iPads, in conjunction with the fact that they are socially acceptable objects, where the cause of their popularity. In other words, by using these devices, the person does not feel stigmatized or uncomfortable. In particular it is very effective introducing iPad with adults, because they can already use it and identify the object as an everyday item, unlike other AAC devices, that have to be learned and mastered by the user, who needs to familiarize with menus, navigation tools and new commands [12]. So, even though both PECS systems and iPad require learning and preparation time, the advantages of the iPad are clear: this device needs the least work to be implemented, as opposed to the time needed by the PECS to prepare, store, and move the cards depicting the objects [5]. In light of what has been said, high-tech devices like iPads, and most generally tablets provide a very good opportunity in improving and personalizing the approach to communication skills in autistic subjects. It is for this reason that arises the necessity to start a systematic study on how to design and implement a tablet app specifically created to meet the needs of the professionals working in the field of rehabilitation of communication skills. The starting point has to be a multidisciplinary approach that takes into account the contribution of every expert in the treatment of autism that is part of the team. Recently, a lot of educational apps were designed to interact with the environment, getting and visualizing informations in real time to the end user in order to adapt dynamically. Augmented Reality (AR) apps are applications that combine digital information with the real world. The most popular AR apps are the one that overlay virtual pictures on a feed from the user's camera. This technology uses device's sensors to detect the environment and to arrange the overlay visualization in real time. Using the experience of speech therapists, psychologist and medical doctors working in our rehabilitation center with these instruments, we built the first concept of LIAR.
Our team started to design and build an app which aims to foster communication of children with autism, by working on a multi step program. In the following lines we will describe the concept we came up with.
The methodology of LIAR is the i-MAND which is a specific mand training teaching method based on Verbal Behavior theory. According to Skinner (1957), a mand is "a verbal operant in which the response is reinforced by a characteristic consequence and is therefore under the functional control of relevant conditions of deprivation or aversive stimulation". In other word, a mand is a request for a specific preferred event and is the only verbal operant for which a response is directly evoked by a motivating operation [MO; 9; 17]. Researchers have explored a large number of behavior analytic procedures, that have been demonstrated to be effective in teaching children to vocal mand [10], manual sign language [18], picture exchange communication system (PECS) [19], and speech-generating devices (SGDs) [20]. About this last, however, there is no set of specific procedures for teaching children with ASD how to operate and communicate with them. LIAR is been designed with the purpose of create a clear and systematic procedure to implement it. In In fact, it is composed by a 5-step teaching method that effectively allow children how to use the system, starting from the use of touch-screen (warm-up) until to reach the request with a sentence (mand phase).
LIAR is a software leveraging modern, cheap hardware commercially available. The software is written in C#, using the Unity Engine (www.unity3d.com) to ensure broad devices support: in fact, every post-2015 Windows, Android and iOS tablet can run effortlessly our app. The User Interface design uses light colors and simple shapes in order to highlight the user-defined content. The customization menu is hidden to the end user, as the therapist has to press a button on a Bluetooth gamepad to show the content. This solution was adopted in order to allow the therapist to interact with the tablet effortlessly and out of the sight of the child, avoiding the interruption of his environment exploration and the loss of attention.
The hardware controller, a universal 4-button Bluetooth device built by BlueBeach (Fig. 1), it's tiny (6,2 x 5,6 x 1 cm) and light (41 g), offer a long-lasting built-in battery and can be associated with every target Operating System.
The graphical interface is optimized for tablet with display sizes from 8 to 10 inches, because every on-screen element can be dynamically adapted. In order to ensure the best user experience, edgeless devices are recommended. LIAR is compatible with ARMv7 (Cortex) CPU or Atom CPU. Is also required full support for OpenGL ES 2.0 or later. The device must have a rear camera to allow the user to add new pictures as photos of his world. Near Field Communication (NFC) and BLE (Bluetooth Low Energy) chip are required to let the device interact with special, RFID-tagged items and the therapist's controller. The methodology used for the development is based on 4 steps:
1) Analysis of the needs: using the data from 4 focus groups conducted with therapists and psychologists, it was aimed to find the best way to treat children with ASD, discussing the PECS methodology, sharing videos of treatment sessions, gathering ideas and requests.
2) Systematic Review: the most recent tablet applications were tested and analyzed in the Focus Group. Every app has been classified using a rubric from Boyd et al. (2015), outlining advantages, usability, customizability and costs [3].
3) Development environment and target platform: target of this step was to find the best platform and development environment to target while building the first prototype. Scalability, programming languages, integration of peripherals and capabilities of cross-compilation are been taken into account.
4) Design and development: flow charts were made to describe every functionality to the developers. The team's work was focused on usability and user interface, with the purpose of smoothing the approach both to the children and the operators. Long and non useful actions were redefined, distracting elements were removed. The whole user experience was optimized to reduce cognitive load and excessive color stimulation. During the development of the application we avoided accidental interactions to limit the frustration level. Through the design phase we also tried to hide the inner technical details and setup settings to the user. All the on-screen objects are designed to be interactive with a single tap, as the size of the items are adequate also for children with difficulties in motor coordination. In the following paragraphs the single phases will be described, from the user interface to the methodology used.
The main menu (Fig. 2) shows six squared buttons, grayed-out by default, that can be unlocked and highlighted by the therapist, allowing the child to focus and interact exclusively with the designated exercise.
White, unique-looking icons were used to differentiate all the activities, giving less visual contrast to locked phases but keeping them on screen, as a visual indicator of the route through the app.
This phase is useful to familiarize children with the drag and drop mechanic we will use in the whole app. It is also useful to increase sensomotor coordination: in fact, many children with autism have issues with fine motor skills [12] that can be trained in this phase. On a black screen, two squares are presented: a full and a hollow one (Fig. 3). Our patients have to drag the full one in the hollow, covering at least 80% of the blank space in order to complete the task, triggering voice and visual reward. Both can be customized (Fig. 4) by the therapist, that can choose to give a physical reinforcement too. The therapist can also schedule different reinforcement ratios, change the reinforcement image and increase the size of the squares: this feature, in particular, is useful for younger or movement-impaired patients. The operator can also use a Bluetooth gamepad to autonomously trigger the rewarding sequence, as a way to shape the wanted behavior. In this phase, some customization was needed to best help our target audience: multi touch is disabled and touch sensitivity is toned down.
This phase is about gradual exposing the user to the communication interface. The operator has to press a gamepad button in order to spawn a picture. The child can drag this picture onto the white square in order to let the tablet speak the object name and to receive the dragged item in real world as a reinforcement. In the settings' menu it is possible to customize a despawn rate for the pictures and to select multiple images for all the buttons on the gamepad.
In this phase, visually identical to the previous one, the operator has to trigger the visualization of a picture by pressing a button on the gamepad. The child can drag and drop the showed pic on a target square, but the tablet does not emit any sound unless the game pad is pressed another time. The operator can define the stimulus to display and has to define with the child a "hand off" behavior (i.e.: giving the tablet to the therapist, looking for eye contact, etc.).
This phase, visually similar to the other ones, keeps the "I Want..." confirmation needs. The new element is that the therapist can choose to show, along with the target, other images, by a randomized list of reinforced and unreinforced items. The child has to discriminate the correct stimulus and drag and drop it on the white box.
This phase extends the software UI, showing more information like picture's categories, target squares and a play button to start the text to speech function. The child has a library of pictures he can improve and organize within the software, using images from the internet or from the tablet's camera. This section includes augmented reality technology, as localization and RFID. Localization is useful to discriminate between settings by highlightings the objects used in that particular environment. For example, when the child is at school, the software will show first items like books, pencils, etc. The software can also read RFID tags placed on specific objects used in therapy room for training. Bringing the tablet near a tag, the user can interact with the attached object, watching a video on how to use it and can request it by using the drag and drop UI.
A cloud platform was also deployed to collect anonymized data of every interaction with our software in milliseconds accuracy. Only the therapist, who got physical access to the tablet, can search for personalized data. A growing number of statistics are generated on the fly to report overall and specific trends in the everyday use. The therapist can also export statistics and raw data, in order to give feedback to the family of ask for a supervision.
LIAR is the result of years of work in the NeapoliSanit rehabilitation center. It is born to combine proven and evidence-based methodologies with the latest frontiers of technology. The research and development team has benefited from the contribution and experience of rehabilitation technicians in order to meet the needs emerging in their daily activities.
Our goal is to develop a system capable of supporting people's communicative exchanges and enriching their environment with interactive objects through the use of Augmented Reality. Those objects can be a powerful stimulus to formulate a request (Mand).
Making use of dynamics facilitators during the learning process should result in a very efficient system. Therefore, our future researches will focus on:
• Evaluating the efficacy of a communication methodology based on interactive objects and pictures and comparing it to the traditional PECS approach. • Observing the number of self-learning occurrences in a system that encourage environment exploration and communicative exchanges.