AR Interface for Teaching Students with Special Needs

Abstract

The article is an overview of one of the practical realization of the augmented reality interface. The purpose of the study is to create an alpha version of the augmented reality application-interface to assist children with special needs in the social orientation. The objects of the study are methods and tools that can help in the education of this category of students. The interface is implemented to solve the educational problems of a certain category of students with the help of augmented reality and machine learning. The work is being completed by highlighting aspects that need further development and research.

Keywords

• Augmented reality Machine learning Special needs

Introduction

The education system must be relevant to the stage of social development and its needs and characteristics so that adolescents are prepared for full integration as members of the community. The role of ICT in education is very important as it creates the conditions for the prevalence and accessibility of education. New technologies, combined with advanced pedagogical tools and practices, create an innovative digital learning environment where collaboration and interaction between students are possible. Learning becomes more attractive and interesting for students and encourages them to participate actively in the learning process. As a result, the quality and effectiveness of the training are improved. But there are still situations where teachers do not want to take advantage of educational opportunities. For example, when teachers are encouraged to use new technology that can simplify student’s learning and save time for teachers, they do not want to use it because they would have to learn how to apply it before teaching students. That is why it is necessary to create an application that will be intuitive - it will be easy to learn for children, but also easy to use for teachers [ 1 ].

One of the common mistakes that can often be heard is the frequent identification of people with autism as immersed in their own inner world. This idea gives rise to the following misconceptions: 1) people with autism are mysterious, mystical; 2) should they be distant by others from their own world, where they are comfortable. In fact, people with autism spectrum disorders live in our world, but they perceive it differently. External (visual, auditory, tactile) or internal (pain, vibration) information they capture fragmentary, inconsistent and separate environmental impressions capture all their attention so that make them insensitive to the perception of the rest of the world, and especially to the ability to synthesize all into the full real picture. A person with an autistic spectrum disorder is not in his inner world, but in the world of his sensory impressions (images, sounds, tactile sensations, etc.) from the outside world [ 2, 3 ]. Very effective in correcting autism spectrum disorders are game techniques such as: 1. The “Son-Rise” method, also known as the “choice method”, widely known for correcting autism was developed by Barry and Samahria Kaufman to find effective ways to interact with their one-year-old son. The main idea of the method is to support the child’s own motivation. The main efforts are aimed at the gradual saturation of external stimuli in the life of an autistic child with sequential sessions, which will necessarily produce positive results. 2. The “Floortime” or “play time” method (literally, time spent on the floor) was developed by American child psychiatrist Stanley Greenspan. The task of the play time is to help the child go through the 6 stages described by Greenspan. An adult captures the interest and initiative of the child as if he or she was most interested in what the child was paying attention to. If the child runs around the room, the adult in the play style may catch up or interfere. 3. The Mifne method for autism therapy, developed by Israeli experts, is gaining popularity worldwide. The method is named after the Mifne Center located in Rosh Pinna, where autism spectrum disorders from an early age were first treated. Therapy at the Center is not clearly structured or didactic. The main goal is to develop social interaction based on the child's innate (potential) abilities [ 2 ].

The use of AR has become more affordable as it no longer requires specialized equipment and can be easily used on mobile devices [ 3, 4 ]. Researchers in this field confirm that virtual objects or avatars can reduce psychological stress during social contacts and improve communication processes between people, making it easier to control the environment and their social interaction.

The AR interface can help to increase the self-esteem of patients who have difficulties controlling social situations in situations such as [ 5 ] and [ 12 ]. In this case, interaction is slower, so children with special needs have more time to think about how to respond to a situation.

Students can change size, shape, position, and other properties of virtual objects allowed by the system and match them to real ones. Augmented reality allows students to experiment and study the properties and behavior of objects in a way that cannot be achieved through traditional approaches. These options activate students’ imagination, creativity, and demonstrate their cognitive and research skills.

Augmented reality can be applied to children using "magic books" as well as to university courses where students learn abstract concepts.

Many of apps are free, so they can be used not only by teachers to develop their own augmented reality programs, but also by students to solve their learning tasks.

Description of the Informational Service

With augmented reality, printed materials can be enriched with digital information audio, video, animation, and 3D objects. Learning becomes interactive, dynamic, context-dependent, more engaging for students, and easier to perceive and interact with. Augmented reality offers the opportunity to review the content of the subject from different perspectives, which is a prerequisite for a deeper insight into the concepts and theories and their understanding. The technology overcomes the disadvantages of static printed materials that do not involve interaction with readers, based on passive transfer of knowledge from the user and require full concentration and use imagination to not just read words but to think through pictures in their heads.

Augmented reality is a technology based on the two-way transfer of information and knowledge. In this case, the content is dynamic, so there are opportunities to add activity information and tasks for students. All these opportunities increase the interest of children in teaching materials, which is a desirable effect in an era when the modern generation lacks interest in reading and using printed materials.

The platform is designed to help children with special learning needs to improve their level of knowledge and socialization. AR has the unique ability to create immersive hybrid learning environments that combine real and virtual objects [ 1 ]. It is expected that the program will be able to support a child with autism in improving communication skills and provide effective practical value.

It is also worth mentioning that in the next version, the possibility of adding an auxiliary solution using machine learning is already included in the development. Using a certain algorithm, the program will guide the student in completing the lessons so as to reduce the maximum possible number of errors.

In Fig. 1 shows the tree of the system being created. IDEF0 methodology was chosen to perform the system analysis. It is the clearest and simplest way to describe the functions of the system. Ten arrows were used. Inputs: “The theme of a lesson”, “Markers”. Outputs: “Student's self-learning”, “Results of the lesson”. Controls: "Ministry of Education and Science's instructions for teaching students with special needs", "Information technology tools (glasses, controllers, tablets, etc.)", "Recommendations for the education of students with special needs". Mechanisms: "Student with special needs", "Teacher", "Hardware" [ 8-11 ]. After that, the main functional block was decomposed into three processes: "Adapt to a device", "Create a profile / Log in", "Work with interface". Two new arrows have been added: “Call the form of registration/log in”, “Data about profile”. Additionally, the breakdown of the “Work with interface” process is described in more detail. There are four new functional blocks here: “Save profile/enter profile”, “Recognize object”, “Display avatar”, “Transition to the lesson”. Also added are three new inputs and outputs: "Login confirmation", "Recognized desk, table, paper, etc.", "Marker".

Practical Realization

While analysing the capabilities of the various free software packages, it is decided which one will be used to implement the required interface functions. The packages Unity 3D and Unreal Engine were analysed. The interface is created using such free assets like Vuforia and Unity tools and plugins. Unity's cross-platform capability makes it easy to use on various devices (smartphones, tablets, laptops, AR/VR sets, etc.). Vuforia provides the ability to use C ++, Java, Objective-C ++ programming languages and .NET (API) programming interfaces with the Unity engine extension. Thus, the SDK supports embedded development for iOS, Android and UWP, while the latter also allows the development of Unity AR applications that are easily adaptable to both platforms. There are two types of augmented reality features, with or without a marker. In this practical implementation, we used AR without markers, that is, for correct use, you need to have a certain image, preferably in a large number of different colors, for realworld objects to appear. These types of markers are recognized if they meet the specified characteristics: large number of details, large number of colors, the size of the printed marker should be at least 12 cm wide and of acceptable height. Recognition algorithms use characteristic points, that is points where the color of the image changes, and the more such points, the more accurately the marker is recognized. In addition, AR will search for predefined templates to identify matches and positions, and then will be practically reproducing information (sounds, images, 3D models, etc.). The Vuforia plugin also calculates the position and orientation of the marker in order to properly display the contents in some cases [ 14-19 ]. The application also uses a neural network to evaluate the quality of the student's lesson and to produce results for the teacher and the child. Machine learning can be accomplished in two ways: with or without a teacher. Training with the teacher to function properly as an input takes some data that should help the system to learn properly. A striking example of the machine learning method with the teacher is the support vector machine. It is great for tasks such as classification and regression because it is very accurate and easy to understand and implement. But for this application, when using machine learning, the reference vector method is not appropriate, since the computing power of a mobile phone is not as large as, for example, a computer and the more dimensions a vector has, the more computational power it will use. This is why reinforcement training will be used.

It will not use much of the device's memory to calculate the quality of the lesson. A login screen was created for this application (Figure 4), so when students write down their name, the system remembers their profile and the results they achieved. These names will then be used to report on the output of each student in the final version of the product, according to all current guidelines and needs available [ 20-26 ]. The application has been designed to meet the requirements (accessibility of the Internet, software and universal design [ 24 ]), and such technologies must meet the specific learning needs of the student in general and the particular needs of the particular student, taking into account his / her psychophysical development.

Such adherence allows to combine hardware and software components of augmented reality information technologies into appropriate complexes, which will allow to achieve a specific educational goal formed by a parapedogue for teaching a student with special needs [ 31-32 ]. It must be remembered that this is an alpha version of the app, so the color combination has not been finalized yet, as there is a need to evaluate several options that may be appropriate for the app to be used by an experimental group of children [ 33-45 ]. The robot is chosen as an avatar because children with autism are very technocritical and love gadgets more than communicating with teachers. And the owl was chosen because this type of bird is known as a symbol of wisdom, so children easily associate the tips of this avatar as a path to proper behaviour [46-66]. Then the screen with the avatar selection appears (Fig. 6). Students can choose one avatar to help them pass the lesson. Currently available avatars are owls and robots. Figure 6 shows an additional menu where parents, a teacher or a student can view the outcome of each lesson (0 to 6 stars), which is evaluated by a neural network algorithm. The student should place his camera in front of the first marker. Then the avatar appears with a special animation. He greets the student and then the avatar recommends that the student should point his camera at the next marker (Fig. 8). As soon as the student points his camera at the marker, the lesson begins. In the next versions of the product, a part will be developed that will use machine learning tools to create specific tips to help students with disabilities learn more effectively. Machine learning input will use data from a student's previous attempts, such as the time the lesson was passed, the attempt to cross the street on a red light, whether the student was passing through an underground passage, and the number of interactions with manipulators by which the student moves and looks at the environment with aiming to give him a rough estimate for the passage of the lesson [ 27-28 ]. Education is constantly changing in order to meet the trends of social development. However, the main causes of the shift in education are not new technologies, but new students with their personal needs and requirements. Technology is a tool to create the necessary educational environment where the learning process can be implemented most effectively. Therefore, the augmented reality working interface helps children with special needs to learn easily and socialize more quickly, as shown by the results of this study. There are two avatars to choose from, with which you can visualize an owl and a robot. After selecting the appropriate avatar, the animation is played. They are designed to be better perceived by children as all of these details help to keep attention of a student with special needs and increase the likelihood of accelerated learning. Using machine learning to generate recommendations for lessons learned is a further goal for improving the system.

Machine learning is used to implement the application, a neural network to evaluate a student's practical knowledge. The child may receive a rating from 1 to 6, but may not receive it at all. That is, if he or she passes the lesson, he/she will eventually get a graphic result of his/her actions to further improve her skills. 53. Zdebskyi, P., Vysotska, V., Peleshchak, R., Peleshchak, I., Demchuk, A., Krylyshyn, M.: An Application Development for Recognizing of View in Order to Control the Mouse Pointer. In: CEUR Workshop Proceedings, Vol-2386, 55-74. (2019) 54. Rzheuskyi, A., Kutyuk, O., Vysotska, V., Burov, Y., Lytvyn, V., Chyrun, L.: The Architecture of Distant Competencies Analyzing System for IT Recruitment. In: Proceedings of the International Conference on Computer Sciences and Information Technologies, CSIT, 254-261. (2019) 55. Gozhyj, A., Kalinina, I., Gozhyj, V., Vysotska, V.: Web service interaction modeling with colored petri nets. In: International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS, 1, 319-323. (2019) 56. Berko, A.Y., Aliekseyeva, K.A.: Quality evaluation of information resources in web-projects. In: Actual Problems of Economics 136(10), 226-234. (2012) 57. Holoshchuk, R., Pasichnyk, V., Kunanets, N., Veretennikova, N.: Information Modeling of Dual Education in the Field of IT. In: Advances in Intelligent Systems and Computing IV, Springer Nature Switzerland AG, Springer, Cham, 1080, 637-646. (2020) 58. Emmerich, M., Lytvyn, V., Yevseyeva, I., Fernandes, V. B., Dosyn, D., Vysotska, V.: Preface: Modern Machine Learning Technologies and Data Science (MoMLeT&DS-2019). In: CEUR Workshop Proceedings, Vol-2386. (2019) 59. Babichev, S.: An Evaluation of the Information Technology of Gene Expression Profiles Processing Stability for Different Levels of Noise Components. In: Data, 3 (4), art. no. 48. (2018) 60. Babichev, S., Durnyak, B., Pikh, I., Senkivskyy, V.: An Evaluation of the Objective Clustering Inductive Technology Effectiveness Implemented Using Density-Based and Agglomerative Hierarchical Clustering Algorithms. In: Advances in Intelligent Systems and Computing, 1020, 532-553. (2020) 61. Berko, A., Alieksieiev, V., Lytvyn, V.: Knowledge-based Big Data Cleanup Method. In:

CEUR Workshop Proceedings, Vol-2386, 96-106. (2019) 62. Rzheuskyi, A., Gozhyj, A., Stefanchuk, A., Oborska, O., Chyrun, L., Lozynska, O., Mykich, K., Basyuk, T.: Development of Mobile Application for Choreographic Productions Creation and Visualization. In: CEUR Workshop Proceedings, Vol-2386, 340-358. (2019) 63. Pasichnyk, V., Shestakevych, T.: The model of data analysis of the psychophysiological survey results. In: Advances in Intelligent Systems and Computing, 512, 271-281. (2017) 64. Shestakevych, T., Pasichnyk, V., Kunanets, N.: Information and technology support of inclusive education in Ukraine. In: Advances in Intelligent Systems and Computing, 754, 746-758. (2019) 65. Shestakevych, T., Pasichnyk, V., Kunanets, N., Medykovskyy, M., Antonyuk, N.: The content web-accessibility of information and technology support in a complex system of educational and social inclusion. In: International Scientific and Technical Conference on Computer Sciences and Information Technologies, CSIT, 1, 27-31. (2018) 66. Andrunyk, V., Pasichnyk, V., Antonyuk, N., Shestakevych, T.: A Complex System for Teaching Students with Autism: The Concept of Analysis. Formation of IT Teaching Complex. In: Advances in Intelligent Systems and Computing IV, Springer Nature Switzerland AG, Springer, Cham, 1080, 721-733. (2020)

References

1. Rothbaum , B. , Hodges , L. , Kooper , R. , Opdyke , D. , Williford , J. , North, M. : Virtual reality graded exposure in the treatment of acrophobia: A case report . Behavior Therapy , 26 ( 3 ), pp. 547 - 554 . ( 1995 ).

2. Skrypnyk , T.V.: Fenomenolohiya autyzmu: Monohrafiya. Vydavnytstvo “Feniks” ( 2010 ).

3. Wu , H. , Lee , S. , Chang , H. , Liang , J.: Current status, opportunities and challenges of augmented reality in education . Computers & Education , 62, pp. 41 - 49 . ( 2013 ).

4. Parsons , S., Mitchell, P. , Leonard , A. : The Use and Understanding of Virtual Environments by Adolescents with Autistic Spectrum Disorders . Journal of Autism and Developmental Disorders , 34 ( 4 ), pp. 449 - 466 . ( 2004 ).

5. Yufang Cheng, Moore, D. , McGrath , P. , Yulei Fan : Collaborative virtual environment technology for people with autism . Fifth IEEE International Conference on Advanced Learning Technologies (ICALT'05) , pp. 231 - 243 . ( 2005 ).

6. Klinger , E. , Marié , R. , Fuchs , P. : Réalité virtuelle et sciences cognitives : Applications en Psychiatrie et Neuropsychologie . Cognito (in press), pp. 1 - 31 . ( 2006 ).

7. Themenheft 2 zur Inklusion - Grundlagen und Hinweise für die Förderung von SuS mit Autismus-Spektrum-Stoerungen (ASS) an allgemeinen Schulen .

8. Parsons , S., Mitchell, P. and Leonard , A. : Do adolescents with autistic spectrum disorders adhere to social conventions in virtual environments? . Autism , 9 ( 1 ), pp. 95 - 117 . ( 2005 ).

9. Herrera , G. , Alcantud , F. , Jordan , R. , et al.: Development of simboli play through the use of virtual reality tools in children with autistic spectrum disorders , Autism , 12 ( 2 ), pp. 143 - 157 . ( 2008 ).

10. Cobb , S. , Beardon , L. , Eastgate , R. , Glover , T. , Kerr , S. , Neale , H. , Parsons , S. , Benford , S. , Hopkins , E., Mitchell, P. , Reynard , G. , Wilson, J.: Applied virtual environments to support learning of social interaction skills in users with Asperger's Syndrome . Digital Creativity , 13 ( 1 ), pp. 11 - 22 . ( 2002 ).

11. Strickland , D. , McAllister , D. , Coles , C. and Osborne , S.: An Evolution of Virtual Reality Training Designs for Children With Autism and Fetal Alcohol Spectrum Disorders . Topics in Language Disorders , 27 ( 3 ), pp. 226 - 241 . ( 2007 ).

12. Ogawa , H. , Hasegawa , S. , Tsukada , S. , Matsubara , M.: A Pilot Study of Augmented Reality Technology Applied to the Acetabular Cup Placement During Total Hip Arthroplasty . The Journal of Arthroplasty , 33 ( 6 ), pp. 1833 - 1837 . ( 2018 ).

13. Mota , J. , Ruiz-Rube , I. , Dodero , J. , Arnedillo-Sánchez , I. : Augmented reality mobile app development for all . Computers & Electrical Engineering , 65 , pp. 250 - 260 . ( 2018 ).

14. Sin , A. , Zaman , H.: Live Solar System (LSS): Evaluation of an Augmented Reality bookbased educational tool . 2010 International Symposium on Information Technology . ( 2010 ).

15. Techakosit , S. , Wannapiroon , P. : Connectivism Learning Environment in Augmented Reality Science Laboratory to Enhance Scientific Literacy . Procedia - Social and Behavioral Sciences , 174 , pp. 2108 - 2115 . ( 2015 ).

16. Meža , S. , Turk, Ž., Dolenc , M. : Measuring the potential of augmented reality in civil engineering . Advances in Engineering Software , 90 , pp. 1 - 10 . ( 2015 ).

17. Akçayır , M. , Akçayır , G. , Pektaş , H. , Ocak , M. : Augmented reality in science laboratories: The effects of augmented reality on university students' laboratory skills and attitudes toward science laboratories . Computers in Human Behavior , 57 , pp. 334 - 342 . ( 2016 ).

18. Mineo , B. , Ziegler , W. , Gill , S. , Salkin , D. : Engagement with Electronic Screen Media Among Students with Autism Spectrum Disorders . Journal of Autism and Developmental Disorders , 39 ( 1 ), pp. 172 - 187 . ( 2008 ).

19. Sirakaya , M. , Alsancak Sirakaya , D. : Trends in Educational Augmented Reality Studies: A Systematic Review . Malaysian Online Journal of Educational Technology , 6 ( 2 ), pp. 60 - 74 . ( 2018 ).

20. Martín-Gutiérrez , J. , Fabiani , P. , Benesova , W. , Meneses , M. , Mora , C. : Augmented reality to promote collaborative and autonomous learning in higher education . Computers in Human Behavior , 51 , pp. 752 - 761 . ( 2015 ).

21. Di Serio , Á., Ibáñez , M. , Kloos , C. : Impact of an augmented reality system on students' motivation for a visual art course . Computers & Education , 68, pp. 586 - 596 . ( 2013 ).

22. Lin , T. , Duh , H. , Li , N. , Wang , H. , Tsai , C. : An investigation of learners' collaborative knowledge construction performances and behavior patterns in an augmented reality simulation system . Computers & Education , 68, pp. 314 - 321 . ( 2013 ).

23. Gopalan , V. , Zulkifli , A. , Bakar , J.: A study of students' motivation using the augmented reality science textbook . ( 2016 ).

24. The UDL Guidelines . (2018, August 31) . Retrieved from http://udlguidelines.cast.org/

25. Web Content Accessibility Guidelines (WCAG) 2.0, www .w3.org/TR/WCAG20/.

26. Sommerauer , P. , Müller , O. : Augmented reality in informal learning environments: A field experiment in a mathematics exhibition . Computers & Education , 79, pp. 59 - 68 . ( 2014 ).

27. Santos , M. , Chen , A. , Taketomi , T. , Yamamoto , G. , Miyazaki , J. , Kato , H.: Augmented Reality Learning Experiences: Survey of Prototype Design and Evaluation . IEEE Transactions on Learning Technologies , 7 ( 1 ), pp. 38 - 56 . ( 2014 ).

28. Kugelmann , D. , Stratmann , L. , Nühlen , N. , Bork , F. , Hoffmann , S. , Samarbarksh , G. , Pferschy , A. , von der Heide , A. , Eimannsberger , A. , Fallavollita , P. , Navab , N. , Waschke , J.: An Augmented Reality magic mirror as additive teaching device for gross anatomy . Annals of Anatomy - Anatomischer Anzeiger , 215 , pp. 71 - 77 . ( 2018 ).

29. Ke , F. , Hsu , Y. : Mobile augmented-reality artifact creation as a component of mobile computer-supported collaborative learning . The Internet and Higher Education , 26 , pp. 33 - 41 . ( 2015 ).

30. Joo-Nagata , J. , Martinez Abad , F. , García-Bermejo Giner , J. , García-Peñalvo , F. : Augmented reality and pedestrian navigation through its implementation in m-learning and elearning: Evaluation of an educational program in Chile . Computers & Education , 111, pp. 1 - 17 . ( 2017 ).

31. Pasichnyk , V. , Shestakevych , T. : The application of multivariate data analysis technology to support inclusive education . In: Proceedings of the International Conference on Computer Sciences and Information Technologies , CSIT, 88 - 90 . ( 2015 )

32. Shestakevych , T. , Pasichnyk , V. , Nazaruk , M. , Medykovskiy , M. , Antonyuk , N.: WebProducts, Actual for Inclusive School Graduates: Evaluating the Accessibility . In: Advances in Intelligent Systems and Computing , 871 , 350 - 363 . ( 2019 )

33. Aziz N , Aziz K , Paul A , Yusof A. : Providing augmented reality based education for students with attention deficit hyperactive disorder via cloud computing: Its advantages , in Proceeding 14th International Conference on Advanced Communication Technology (ICACT) , pp. 577 - 581 . ( 2012 )

34. Reardon , C. : An Intelligent Robot and Augmented Reality Instruction System / Reardon . - Knoxville: The University of Tennessee,( 2016 )

35. Quintero , J. , Baldiris , S. , Rubira , R. , Cerón , J. , Velez , G.: Augmented Reality in Educational Inclusion. A Systematic Review on the Last Decade . Frontiers in Psychology, 10 . ( 2019 ).

36. Tang , T. , Xu , J. , Winoto , P. : Automatic Object Recognition in a Light-Weight Augmented Reality-based Vocabulary Learning Application for Children with Autism . Proceedings of the 3rd International Conference on Innovation in Artificial Intelligence - ICIAI . ( 2019 ).

37. Lee , I. , Lin , L. , Chen , C. , Chung , C. : How To Create Suitable Augmented Reality Application To Teach Social Skills For Children With ASD . ( 2020 ).