Design of Mobile Augmented Reality System for Early Literacy Marlen Ablyaev, Afife Abliakimova, Zarema Seidametova Crimean Engineering-Pedagogical University, Simferopol {ablyaev.marlen, ablyakimova.afife, z.seydametova}@gmail.com Abstract. Mobile applications allow connecting the world around us with digital information. Augmented Reality (AR) applications are commonly used in museums, education and for research. AR applications can attract users with contextual and locally specific information. We present a concept, development of augmented reality (AR) application “Tilsimli arifler” in which we create literate environment for motivating of children in practicing reading and writing. In order to enhance the kids reading ability we propose to develop a learning mobile application that allows using modern mobile technologies, in particular, using the technology of augmented reality, to study the alphabet and learn how to make words. In this case, a prerequisite was the use of special (drawn) cards and a camera of a mobile device. When scanning a card with a camera on a user's smartphone, a three-dimensional image of the object deposited on the card should appear and which can be rotated in different directions (“spinning”) in real time. Keywords: augmented reality (AR), virtual reality (VR), software development kit (SDK) software development, augmented marks, vuforia, quick response (QR) code. 1 Introduction The rapid growth in information and communication technologies and the large-scale process of informatization society are the leading factors that identify the needs and necessities for changes in education. Mobile applications allow connecting the world around us and digital information. Therefore such applications are widely used in museums, research and educational institutions. Augmented Reality (AR) applications offer intuitive ways to provide users with contextual and locally specific information. One of the widely used approaches in the development of augmented reality mobile applications is the use of visual markers. Visual markers used in AR applications allow users to “animate” information associated with markers. Applications that use markers allow user to read information about a specific object or its location using mobile devices. The modern period of the society evolution is characterized by a strong influence on it of computer technologies that penetrate into all spheres of human activity, ensure the dissemination of information flows in society, forming a global information space. An integral and important part of these processes is the computerization of education. At present, a new education system is being formed, oriented towards entering the world information and educational space. This process is accompanied by significant changes in the pedagogical theory and practice of the educational process associated with making adjustments to the content of learning technologies that should be adequate to modern technical capabilities, and contribute to the harmonious entry of the child into the information society. One of the first stages of the child’s entry into the educational environment is the study of the alphabet and the acquisition of word-constructing skills. Considering that modern technologies surround us in everyday life, children also periodically encounter them. But, to make the process of interaction between the child and computer technologies more productive and expedient, it is necessary to prepare appropriate software that will allow users to provide educational content in a game form. Thus, there is a need to create a mobile application for learning the alphabet and constructing words using special cards, which depict a specific letter and subject, starting with this letter. To attract the attention of the child uses augmented reality. In order to enhance the kids reading ability we propose to develop a learning mobile application that allows using modern mobile technologies, in particular, using the technology of augmented reality, to study the alphabet and learn how to make words. In this case, a prerequisite was the use of special (drawn) cards and a camera of a mobile device. When scanning a card with a camera on a user's smartphone, a three- dimensional image of the object deposited on the card should appear and which can be rotated in different directions (“spinning”) in real time. In the paper [1] authors provided an overview of Augmented Paper Systems use in education. Studies related to the effect of augmented reality on learning efficiency presented in papers [2], [3], [4], [5], in which also discussed questions on how to combine the capabilities of augmented reality and students’ learning. An experience of the successful use of AR mobile applications in learning and teaching process described in [6]. In [7] – [14] authors presented the results of their research on the development and implementation of AR apps for children. In [15] – [20] researchers discussed the effects of marketing, usability, and design on augmented reality technologies and ways of the development of AR apps. Comparative analysis of augmented reality frameworks that are used to develop educational applications presented in [21]. 2 Tools for Augmented Reality Development One of the most common environments for visualizing and interacting of the three- dimensional graphics as development environment is Unity [22]. Unity is a tool for developing two- and three-dimensional applications and games. Applications created in Unity are supported by following operating systems: Windows, OS X, Windows Phone, Android, Apple iOS, Linux. It is also possible to use the Unity Web Player plug- in, which enables to launch and use applications in different web browsers. Applications created using Unity also support DirectX [23] and OpenGL [24], [25]. One of the main advantages of Unity is cross-platform that means the ability to use the same code with minimal changes to create applications for multiple platforms at once. It is possible to expand the functionality with the help of Asset Store. There are a large numbers of different plug-ins and extensions in Asset Store. Another popular engine is the Unreal Engine [26]. The main difference from Unity Unreal Engine is to use a different programming language. Unreal Engine uses C ++, and Unity uses C # and Java. Unreal Engine has a modular system of components that are dependent on each other, which simplifies the transfer of the project from one system to another. Supported systems are: Microsoft Windows, Linux, Mac OS and Mac OS X; Xbox, Xbox 360, PlayStation 2, PlayStation 3, PSP, PS Vita, Wii, Dreamcast, GameCube, etc., as well as on various portable devices, such as Apple (iPad, iPhone) devices controlled by iOS [27] and others. To create augmented reality, developers use systems such as Vuforia [28] and Kudan [29]. Vuforia is a platform that allows an application installed on a mobile device with a camera to “see” and recognize various objects, images, texts and markers located on them. Object recognition is used for the so-called “revitalization” of them in the form of 3D models on the device screen. For example, with the help of this platform, it is possible to “revive” some mechanisms, having previously read them with the help of the special utility Vuforia Object Scanner. Using it, a special file that loads markers into the manager is created, and then these markers are used to recognize the original object. The same principle applies when you create markers from the images and other texts. In particular, there is an opportunity to create a variety of effects, such as video playback when you move the marker, the use of night vision effects or creating controls on the markers in the real world, read the camera device. Vuforia is supported by mobile devices running Android 4.0.3 and iOS 7 and above. It is also supported by augmented reality glasses, for example, such as Epson Moverio [30]. Kudan is another augmented reality developing system. Among the key features of this technology are the enhanced tracking capabilities of the markers, such as steeper viewing angles, accelerated guidance to the marker, long distance to the marker, the ability to work in difficult lighting conditions. Another feature is the provision of various methods of augmented reality, such as marker and marker-less. Marker technology provides reliable tracking of the object, the ability to create custom markers and their simultaneous tracking. Marker-less technology allows you to track objects without installing special labels on them. The advantage of this technology is that real- world objects can serve as markers themselves and the creation of special visual identifiers for them are not required [29]. Both of these technologies work using the OpenCV [25] specialized computer vision library. This library is able to work with virtually any operating system, has C ++, C, Java and Python interfaces. The library has more than 2500 optimized algorithms, which includes a full range of both classical and computer vision and machine learning algorithms. EasyAR [31] is a free and easy-to-use alternative to Vuforia. EasyAR supported platforms are Android, iOS, UWP, Windows, OS X and Unity. It is known that the applied development augmented reality technology tools depend on the type of tasks and the available equipment for implementation. Development tools such as Daqri, MixAR and ZooBrust are fairly simple and do not require high programming skills. Other tools that include SDKs, such as ARToolKit, Unifeye Mobile SDK, and Wikitude, were created for advanced developers [32]. Wikitude is a set of software for developers that allow you to create your own AR- applications from scratch, and integrate AR-functionality into ready-made applications. You can see a comparison of most popular tools for the development of augmented reality applications in the table 1. Table 1. Comparison of the most popular AR tools Type Cross-platform Programming Documentatio languages n / tutorials Unity Boost Windows, OS X, Windows С#, JavaScript + Software Phone, Android, Apple iOS, License Linux Unreal Boost Microsoft Windows, Linux, C++ + Engine Software Mac OS и Mac OS X; License консолей Xbox, Xbox 360, PlayStation 2, PlayStation 3, PSP, PS Vita, Wii, Dreamcast, GameCube Vuforia Free + iOS, Android, Unity C + +, Java, + Commercial Objective-C, SDK option .Net Kudan Free + Microsoft Windows, Linux, C++, C, Java и + Commercial Mac OS, Mac OS X; Python SDK option PlayStation 2, PlayStation 3, PSP, PS Vita, Wii, Dreamcast, GameCube OpenCV Boost iOS, PC, Android, Linux С/C + +, + Software Python, Java, Ruby, Matlab, Lua Wikitud Free + Android, iOS, smart glasses С#, Java – e Commercial SDK option Because of the accessibility, availability of the necessary functionality, the possibility of connecting additional sets of development tools and the availability of peripheral (additional) services the Unity platform and Vuforia SDK were selected for development of the “Tilsimli arifler” application. 3 Project Design with Vuforia Vuforia has several recognition methods (data entry) for visual information ([33], [34], [35], [36]): 1) Markers (Image Targets) - these are real flat images that applied to any favorable flat surface. Markers can be represented as QR or matrix codes. The marker does not require special black and white areas in order for it to be tracked in the Vuforia Engine. It is convenient to use markers anywhere, but in the near future they will most likely be used for prompts, information references, for creating interactive books and manuals, visualizing any concepts, and so on. 2) Multi-Targets are analogous to ordinary markers, but with their use, more than one marker is tracked on one real object, for example, on a cube. This is especially useful for extra-large physical objects that are overlaid with augmented reality and need to be captured from all sides, for example: houses, monuments, museums, etc. 3) Cylinder Targets are real flat images applied to cylindrical surfaces. They allow you to track markers in the form of twisted cones and cylinders. Such markers are used on cups, bottles, wide footsteps, etc. 4) Virtual buttons work thanks to the obstacle detection technology, which allows the user to perform any action or function visually resembling a real button when it overlaps a certain predefined zone. 5) Recognition of visual text (Text Recognition) is recognition of images of a predetermined list of text characters. The default SDK includes 100,000 Latin characters of various formats that can be recognized by the application. It is also possible to create personal character sets (maximum 10,000 additional characters). 6) Recognition of physical objects in 3D (Object Scanner) recognizes 3D objects and makes of them simplified geometric skeletons-primitives (label-objects). In the future this allows identifying a specific physical object in the physical space. To create a new object marker first program scans an existing object (Object Scanner), after which the Object Data File is automatically generated. The generated file is uploaded to the Vuforia Target Manager (it is possible to load up to a maximum of 20 Object Data File). For the application being developed, targets were used Image Targets, therefore Single Image objects were loaded into the database. It should be noted that not every picture is suitable for creating a target. Good targets are those in which there are many contrasting details. It is on these details that the reference matrix is built for their subsequent recognition. In Figure 1 in the table of loaded targets the Rating field shows how much the picture is suitable for recognition. Fig. 1. Downloaded objects into the database. In the “Tilsimli arifler” application, targets were created for loading into the database, the design of which took into account the peculiarities of Vuforia recognition. Samples of targets are presented in Figure 2. Fig. 2. Samples of targets for recognition. You can add Vuforia targets to your scene by selecting the corresponding game objects in the GameObject–>Vuforia menu. The target game object will be added to the scene hierarchy and appear on the scene. Each target object can be configured in the Inspector. The configuration window indicates the database and the name of the target to be used. Now you can add digital content to complement the target. To do this, add objects that will complement the target as child elements of the target in the scene hierarchy (Figure 3). Fig. 3. Screen view of the object and targets. Displaying the contents of augmented reality is done by customizing the rendering and physical behavior. Vuforia provides a simulator in game mode, which can be activated by pressing the play button. You can use this feature to evaluate and quickly prototype your scene (s) without having to deploy it on the device. Playback mode is configured in the Vuforia WebCam section. 4 Implementation of “Tilsimli Arifler” The implementation of this application was made in the Unity development environment, which by its means and capabilities allows processing three-dimensional objects and images obtained from the camera of a mobile device using the Vuforia Augmented Reality SDK [30]. Vuforia Augmented Reality SDK allows you to track one or more images with the camera, and Unity allows you to place a three-dimensional object on the user's screen in a certain position. Thus, the Vuforia Augmented Reality SDK determines the image coordinates in which Unity should place the corresponding object. Thus, in the application there is a process of finding letters and displaying the necessary models. To enable the user to compose words from letters, a special algorithm was developed, a fragment of which is given below (Listing 1): Listing 1. Camera image processing and word recognition algorithm. public class SetRay : MonoBehaviour { public int NumberWord = -1; [SerializeField] private GameObject cameraGO; private int tmp = 200; private RaycastHit hit; private RaycastHit hit1; private Ray rayRight; private Ray rayLeft; private bool firstChar = false; void Update(){ rayRight = new Ray(transform.position, transform.right); rayLeft = new Ray(transform.position, -transform.right); if (Physics.Raycast (rayRight, out hit)) { if (hit.transform.tag == "Char" && NumberWord !=0) { if (!Physics.Raycast (rayLeft, out hit1)) { firstChar = true; cameraGO.GetComponent ().Word = ""; NumberWord = 0; cameraGO.GetComponent ().Word = "" + transform.name; tmp = NumberWord; } } } if (Physics.Raycast (rayRight, out hit)) { if (hit.transform.tag == "Char" && SetRay().NumberWord != NumberWord + 1) { int tmpInt = NumberWord + 1; hit.transform.GetComponent ().NumberWord = tmpInt; MainWord().Word = MainWord().Word+ hit.transform.name; } } } } This algorithm processes the image from the camera and recognizes the word composed with the help of appropriate cards (Fig. 4). Fig. 4. Word recognition algorithm. The Left () function in this algorithm is necessary to determine the first element, that is the first letter of the word. If there is no other card (letter) to the left of the card (letter), then this element is the first, therefore it will be first element written into the variable that defines the word. Then the Right () function is executed defining the next element. If during the execution of the Right () function it turns out that another card (letter) is on the right, a new element is added to the variable, thereby forming a word. And if there is no other element to the right, then this element is the last one, it starts the Search () function. The Search () function searches a local database of words, comparing the resulting word with those already in the program. If the constructed word is found in a local database, a three-dimensional model that corresponds to the received word will be displayed on the user's screen (Fig. 5). If the constructed word is not found in the local database, the application will notify the user about a possibly incorrectly composed word or the absence of the corresponding model. To visualize all the letters of the alphabet and the words existing in the local database, it was necessary to use three-dimensional models that correspond to one or another word or image on the card. Source objects were obtained from the official Unity public store. The resulting objects were imported into Unity with their subsequent configuration. Setting up each model meant changing the size of the object, which should be proportional to the size of the card; this allows avoiding displaying an object beyond the visibility of the camera. The important step in setting up objects was the stage of texture optimization, which should be clearly visible under different light conditions and spend as little as possible on the memory of the mobile device. Next, animations were added to each model for a smoother appearance of the object on the scene and improvement of the visual perception of the final image on the screen of the mobile device. In addition, all three-dimensional models were optimized in order to reduce the size of each object; this reduced the size of the source file of the application and increased the overall performance. Fig. 5. A running example of the “Tilsimli arifler” application: recognized word image appearance. The application development process is shown in video tutorials and published on YouTube [37], [38]. The videos describe step by step the entire development process from creating a project to compiling it. The course is designed for students and novice developers who want to use the technology of augmented reality (Augmented Reality). The development of the project “Tilsimli arifler” was used as a demonstration example in the videos. You can see screenshot of videos playlist in Fig. 6. Fig. 6. YouTube Playlist of the “Unity AR for beginners” videos created in Applied Informatics Department of CEPU [ ]. 5 Conclusions Nowadays the augmented reality is one of the most innovative technologies in the developing mobile applications that create a platform for the emergence of new types, genres and activities of similar applications. The AR technology opens a new horizon for the development of mobile applications, including in the educational activities. The AR project “Tilsimli arifler” developed by us is given the opportunity to study English and Crimean Tatar alphabets, and learn how to compose words in the appropriate languages. In this case, the process of learning is accompanied by three- dimensional visualization and animation of each letter and each word. This contributes to the quick memorization, retention of the learner's attention, visibility, dynamism and susceptibility of the new information. As a result, learning becomes exciting and more productive. In addition, the experience gained from the development of the “Tilsimli arifler” application opens up a new look at the development of mobile learning applications. In the future we plan to introduce a larger number of three-dimensional objects for letters and words, expand the local database of words, audio explanations to accompany each word, expand the application in order to add new databases for learning as many languages as possible. References 1. Prieto, L.P., Wen, Y., Caballero, D. and Dillenbourg, P.: Review of Augmented Paper Systems in Education: An Orchestration Perspective. Journal of Educational Technology & Society 17(4), 169-185 (2014). 2. Lee, K.: Augmented reality in education and training. TechTrends 56(2), 13-21 (2012). 3. Liarokapis, F. and Anderson, E.F.: Using augmented reality as a medium to assist teaching in higher education. In: Proceedings of the 31st annual conference of the european association for computer graphics (Eurographics 2010), pp. 9-16. Norrkoping, Sweden (2010). 4. Bower, M., Howe, C., McCredie, N., Robinson, A. and Grover, D.: Augmented Reality in education–cases, places and potentials. Educational Media International 51(1), 1-15 (2014). 5. Akçayır, M. and Akçayır, G.: Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review 20, 1- 11(2017). 6. Santos, M.E.C., Chen, A., Taketomi, T., Yamamoto, G., Miyazaki, J. and Kato, H.: Augmented reality learning experiences: Survey of prototype design and evaluation. IEEE Transactions on learning technologies 7(1), 38-56 (2014). 7. Hornecker, E. and Dünser, A.: Supporting Early Literacy with Augmented Books- Experiences with an Exploratory Study. GI Jahrestagung (1), 555-559 (2007). 8. Bhadra, A., Brown, J., Ke, H., Liu, C., Shin, E.J., Wang, X. and Kobsa, A.: March. ABC3D—Using an augmented reality mobile game to enhance literacy in early childhood. In: Pervasive Computing and Communication Workshops (PerCom Workshops) IEEE International Conference on, pp. 1-4(2016). 9. Alakärppä, I., Jaakkola, E., Väyrynen, J. and Häkkilä, J.: Using nature elements in mobile AR for education with children. In: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, pp 41 (2017). 10. Barkhaya, N.M.M., Halim, N.D.A. and Yahaya, N.: The Importance of Augmented Reality Application for Children’s Development During Preschool Years. Advanced Science Letters 24(11), 7935-7938 (2018). 11. ChanLin, L.: Bridging Children’s Reading with an Augmented Reality Story Library. Libri 68(3), 219-229 (2018). 12. Baloch, S., Qadeer, S. and Memon, K.: Augmented Reality, a Tool to Enhance Conceptual Understanding for Engineering Students. In: International Journal of Electrical Engineering & Emerging Technology, Vol. 01, No. 01, DEC 2018, pp 41-48 (2018). 13. Huang, Y., Li, H. and Fong, R.: Using Augmented Reality in early art education: a case study in Hong Kong kindergarten. Early Child Development and Care 186(6), 879-894 (2016). 14. With Help from Augmented Reality, ELL's are Making Strides in Literacy, https://bit.ly/2TGKYQr, last accessed 2019/02/05. 15. Liao, T.: Augmented or admented reality? The influence of marketing on augmented reality technologies. Information, Communication & Society 18(3), 310-326 (2015). 16. Santos, M.E.C., Taketomi, T., Sandor, C., Polvi, J., Yamamoto, G. and Kato, H.: A usability scale for handheld augmented reality. In: Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology, pp. 167-176 (2014). 17. Sánchez, A., Redondo, E. and Fonseca, D.: Developing an augmented reality application in the framework of architecture degree. In: Proceedings of the 2012 ACM workshop on User experience in e-learning and augmented technologies in education, pp. 37-42. (2012). 18. Santos, M.E.C., Taketomi, T., Yamamoto, G., Rodrigo, M.M.T., Sandor, C. and Kato, H.: Augmented reality as multimedia: the case for situated vocabulary learning. Research and Practice in Technology Enhanced Learning 11(1), 1-23 (2016). 19. Chenga, L.K., Selamatb, A., Putehc, F. and Mohamedb, F.: AReview Of recent methodologies, technologies and usability in English language content delivery. Jurnal Teknologi 78(12-3), 1-11(2016). 20. Huang, Z., Hui, P., Peylo, C. and Chatzopoulos, D.: Mobile augmented reality survey: a bottom-up approach. arXiv preprint arXiv:1309.4413, 112-126 (2013). 21. Herpich, F., Guarese, R.L.M. and Tarouco, L.M.R.: A comparative analysis of augmented reality frameworks aimed at the development of educational applications. Creative Education 8(09), 1433-1451(2017). 22. nity for Mobile AR, https://unity.com/solutions/mobile-ar, last accessed 2019/02/08. 23. DirectX at Curlie, https://curlie.org/Computers/Programming/Libraries/DirectX/, last accessed 2019/02/06. 24. OpenGL – The Industry Standard for High Performance Graphics, https://www.opengl.org/, last accessed 2019/02/06. 25. OpenCV library, https://opencv.org/, last accessed 2019/02/06. 26. Unreal Engine, https://www.unrealengine.com/en-US/what-is-unreal-engine-4, last accessed 2019/02/06. 27. Xcode - Apple developer, https://developer.apple.com/xcode/, last accessed 2019/02/06. 28. Vuforia. Augmented Reality for the Industrial Enterprise, https://www.vuforia.com, , last accessed 2019/02/09. 29. Kudan. Artificial Perception technologies, https://www.kudan.eu/, last accessed 2019/02/06. 30. Epson Moverio / Smart Glasses, https://moverio.epson.com/, last accessed 2019/02/06. 31. EasyAR-Best engine for developing Augmented Reality, https://www.easyar.com/, last accessed 2019/02/06. 32. Amin, D. and Govilkar, S.: Comparative study of augmented reality SDKs. International Journal on Computational Science & Applications 5(1), 11-26 (2015). 33. Wang, J.T., Shyi, C.N., Hou, T.W. and Fong, C.P.: December. Design and implementation of augmented reality system collaborating with QR code. In: Computer Symposium (ICS), 2010 International, pp. 414-418. IEEE (2010). 34. Alcantara, M.D., Hounsell, M.S. and Silva, A.G.: Alternative position, orientation and data recognition algorithms for augmented reality markers. In: International Conference Applied Computing (https://bit.ly/2DugAli). Rio de Janeiro:[sn](2011). 35. AR tracking with different markers, http://it-jim.com/2016/09/19/augmented-reality- tracking-with-different-markers/, last accessed 2019/02/08. 36. Theory and applications of marker-based augmented reality, https://www.vtt.fi/inf/pdf/science/2012/S3.pdf, last accessed 2019/02/08. 37. Seidametova, Z.: Design and Development of MOOCs. CEUR Workshop Proceedings (CEUR-WS.org). Proceedings of the 14th International Conference on ICT in Education, Research and Industrial Applications. Integration, Harmonization and Knowledge Transfer. Volume 2104, issue 2, pp. 462-471 (2018). 38. CP4B. Applied Informatics of CEPU. YouTube Playlist, https://bit.ly/2th4jvP, last accessed 2019/02/11.